Epitopological learning and Cannistraci-Hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning
Authors: Yingtao Zhang
Jialin Zhao
Wenjing Wu
Alessandro Muscoloni
Carlo Vittorio Cannistraci
Venue: ICLR 2024 poster (2024)
Keywords: Sparse Training
Network Science
Epitopological Learning
Network Automata
Network Percolation
Hyperbolic Network
Primary area: general machine learning (i.e., none of the above)
TL;DR: The study presents ESML and CHT, methods that combine dynamic sparse training and epitopological learning to efficiently structure artificial neural networks, resulting in a significant reduction in node size while maintaining performance

Abstract: Sparse training (ST) aims to ameliorate deep learning by replacing fully connected artificial neural networks (ANNs) with sparse or ultra-sparse ones, such as brain networks are, therefore it might benefit to borrow brain-inspired learning paradigms from complex network intelligence theory. Here, we launch the ultra-sparse advantage challenge, whose goal is to offer evidence on the extent to which ultra-sparse (around 1\% connection retained) topologies can achieve any leaning advantage against fully connected. Epitopological learning is a field of network science and complex network intelligence that studies how to implement learning on complex networks by changing the shape of their connectivity structure (epitopological plasticity). One way to implement Epitopological (epi- means new) Learning is via link prediction: predicting the likelihood of non-observed links to appear in the network. Cannistraci-Hebb learning theory inspired the CH3-L3 network automata rule for link prediction which is effective for general-purpose link prediction. Here, starting from CH3-L3 we propose Epitopological Sparse Meta-deep Learning (ESML) to apply Epitopological Learning to sparse training. In empirical experiments, we find that ESML learns ANNs with ultra-sparse hyperbolic (epi-)topology in which emerges a community layer organization that is meta-deep (meaning that each layer also has an internal depth due to power-law node hierarchy). Furthermore, we discover that ESML can in many cases automatically sparse the neurons during training (arriving even to 30\% neurons left in hidden layers), this process of node dynamic removal is called percolation. Starting from this network science evidence, we design Cannistraci-Hebb training (CHT), a 4-step training methodology that puts ESML at its heart. We conduct experiments on 7 datasets and 5 network structures comparing CHT to dynamic sparse training SOTA algorithms and the fully connected counterparts. The results indicate that, with a mere 1\% of links retained during training, CHT surpasses fully connected networks on VGG16, GoogLeNet, ResNet50, and ResNet152. This key finding is an evidence for ultra-sparse advantage and signs a milestone in deep learning. CHT acts akin to a gradient-free oracle that adopts CH3-L3-based epitopological learning to guide the placement of new links in the ultra-sparse network topology to facilitate sparse-weight gradient learning, and this in turn reduces the convergence time of ultra-sparse training. Finally, CHT offers the first examples of parsimony dynamic sparse training because, in many datasets, it can retain network performance by percolating and significantly reducing the node network size.
Our code is available at: https://github.com/biomedical-cybernetics/Cannistraci-Hebb-training

Published as a conference paper at ICLR 2024
EPITOPOLOGICAL LEARNING AND CANNISTRACI- HEBB NETWORK SHAPE INTELLIGENCE BRAIN- INSPIRED THEORY FOR ULTRA-SPARSE ADVANTAGE
IN DEEP LEARNING
Yingtao Zhang Jialin 22.0 Wenjing Wu Alessandro Muscoloni
& Carlo Vittorio Cannistraci .2,3,4
CCenter for Complex Network Intelligence (CCNI)
TTsinghua Laboratory of Brain and Intelligence (THBI) 3 Department of Computer Science, *Department Biomedical Engineering
Tsinghua University, Beijing, China.
ABSTRACT
Sparse training (ST) aims to ameliorate deep learning by replacing fully con- nected artificial neural networks (ANNs) with sparse or ultra-sparse ones, such as brain networks are, therefore it might benefit to borrow brain-inspired learning paradigms from complex network intelligence theory. Here, we launch the ultra- sparse advantage challenge, whose goal is to offer evidence on the extent to which sttrr-sparse (around 1% connection retained) topologies can achieve any leaning advantage against fully connected. Epitopological learning is a field of network science and complex network intelligence that studies how to implement learning complex networks by changing the shape of their connectivity structure (epi- topological plasticity). One way to implement Epitopological (epi- means new) Learning via link prediction: predicting the likelihood of non-observed links appear in the network. Cannistraci- work automata rule for link prediction which is effective for purrpoe link prediction. Here, starting from CH3-L3 we propose Epitopological Sparse Meta- deep Learning (ESML) apply Epitopological Learning to sparse training. In em- pirical experiments, we find that ESML learns ANNs with ultra-sparse hyperbolic (epi-)topology in which emerges community layer organization that is meta- deep (meaning that each layer also has an internal depth due to power-law node hierarchy). Furthermore, we discover that ESML can in many cases automatically sparse the neurons during training (arriving even to 30% neurons left in hidden layers), this process node dynamic removal is called percolation. Starting from this network science evidence, design Cannistraci-Hebb training (CHT), 4- step training methodology that puts ESML at its heart. We conduct experiments on 7 datasets and 5 network structures comparing CHT to dynamic sparse training SOTA algorithms and the fully connected counterparts. The results indicate that, with mere 1 1% of links retained during training, CHT surpasses fully connected networks on VGG16, GoogLeNet, ResNet50, and ResNet152. This key finding is an evidence for ultra-sparse advantage and signs a milestone in deep learning. CHT acts akin to gradient-frree oracle that adopts CH3-L3-based epitopological learning guide the placement of new links in the ultra-sparse network topology to facilitate Sparse-weigh gradient learning, and this in turn reduces the conver- gence time ultra-sparse training. Finally, CHT offers the first examples par- simony dynamic sparse training because, in many datasets, it can retain network performance by percolating and significantly reducing the node network size. Our code is available at:
learning theory inspired the CH3-L3 net-
training
"Corresooningg author, kalokagathos.agonee gmail..cmmmmmmmmmmmmmmmmmmm

---

Published as a conference paper at ICLR 2024
INTRODUCTION
The human brain can learn using sparse and ultra-sparse neural network topologies and spending few watts, while current artificial neural networks are mainly based on fully connected topologies. How to make profits from ultra-sparse brain-inspired principles to ameliorate deep learning is the question that we try to address in this study, where we launch the ultra-sparse advantage challenge: the goal is offer evidence on the extent to which ultra-sparse (around 1% connection retained) topologies can achieve any leaning ad advantage against fully connected. The ultra-sparse advantage challenge in line with the current research in AI, focusing on how to reduce the training burden of the models to increase its trade-off efficiency between training speed, performance, and model size.
There exist two popular methods known as pruning (LeCun et al.] 1989 Hassibi et al.
1993 Li &
Louri 202 : Sun et al., 2023] Frantar & Alistarh, 2023] Zhang et al. 2023a Liu et al.
2023 and
sparse training (Mocanu et al. 2018: Lee et al. 2019; Evei et al., 2020 Yuan et al., 2021] 2022
Zhang et al. 2022 2023b) which introduce sparsity into deep learning. However, there are fun- damental differences between them. Pruning begins with a fully connected network and gradually eliminates connections, aiming to strike a balance between post-training inference speed and final performance. Sparse training methods initiate with a sparsely connected network, resulting in inher- sparsity in both the forward and backward processes during training. Consequently, the primary goal of sparse training methods is to find the equilibrium balance between training acceleration and final performance. Dynamic Sparse Training (DST) (Mocanu et al. 2018 Evei et al. 2020) Yuan et al. (02T). variant of sparse training. is always considered the most efficient strategy due to its ability to evolve the network's topology dynamically. However, it's worth noting that many DST algorithms don't solely rely on the intrinsic network sparsity throughout the training process. For instance, approaches like RigL. (Evci et al.] 2020) leverage gradient information from missing links, making the network fully connected during the backpropagation phase. Similarly, OptG (Zhang iet al. 2022) introduces an extra super-mask, adding learnable parameters during training. These DST methods deviate from the core principle of sparse training, which aims to accelerate model training solely through the existing connections in the network. Integrating theories from topolog- ical network science, particularly those inspired by complex network intelligence theory designed for sparse topological scenarios, into DST could yield fascinating results.
Epitopological Learning (EL) (Daminelli et al. 2015 Durán et al. 2017 Cannistraci 2018) is field of network science inspired by the brain that explores how to implement learning on networks by changing the shape of their connectivity structure (epitopological plasticity) and forming epitopo- logical engrams (memory traces). One approach to implementing EL is via link prediction, which predicts the existence likelihood of each non-observed link in network. EL was developed along- side the Cannistraci-H Hebb (CH) network automata learning theory (Daminelli et al., 2015) Durán jet al., 2017] [annistraci] 2018) according to which: the sparse local-community organization of many complex networks (such as the brain ones) is coupled to dynamic local Hebbian learning process and contains already in its mere structure enough information to predict how the connectiv- ity will evolve during learning partially.
The inspiration behind this theory is that for many networks associated with complex systems such as the brain, the shape of their connectivity is learned during training process, which is the result of the dynamic evolution of the complex systems across the time (Narula 2017 Cannistraci 2018 Cannistraci al., 2013a Abdelhamid et al. 2023) This means that the evolution of the complex system carves the network structure, forming typical features of network complexity such as cluster- ing, sorll-wwrlddesss, power-lawness, hyperbolic topology, and community organization (Muscoloni & Cannistraci] 2018). In turn, given a network with a shape characterized by these recognizable fea- tures of complexity. a significant (better than random) part of its future connectivity can be predicted by local network automata, which interpret the information learned in the engrams (memory traces)
of the complex network topology (Narula 2017 Cannistraci 2018 Cannistraci et al. 2013a). Based on these notions, network shape intelligence is the intelligence displayed by any topological network automata to perform valid (significantly more than random) connectivity predictions without train- ing by only processing the input knowledge associated with the local topological network organiza- tion (Abdelhamid et al., 2023). The network automaton training is unnecessary because it performs predictions extracting information from the network topology. which can be regarded as associa- tive memory trained directly from the connectivity dynamics of the complex system (Abdelhamid et al. 2023)
2

---

Published as a conference paper at ICLR 2024
CH3-L3 (Muscoloni al. 2020), one of the network automata rules under CH theory, is effec- tive for general-purposed link prediction in bipartite networks. Building upon CH3-L3, we propose Epitopological Sparse Meta-deep Learning (ESML) implement EL in evolving the topology of ANNS based on dynamic sparse training procedures. Finally, we design Cannistraci-Hehh train- ing (CHT), a 4-step training methodology that puts ESML at its heart, with the aim enhance prediction performance. Empirical experiments are conducted across 7 datasets (MNIST, Fash- ion MNIST, EMNIST, CIFARIO, COFARI00, TinylmageNet, and ImageNet2012) and 5 network structures (MLPs, VGG16, GoogLeNet, ResNet50, and ResNet152), comparing CHT to dynamic sparse training SOTA algorithms and fully connected counterparts. The results indicate that with mere 1 1% of links retained during training. CHT surpasses fully connected networks on VGGI6, GoogLeNet, and ResNer50. This key finding is evidence of ultra-sparse advantage and signs a milestone in deep learning. To help the reader get familiar with the concepts of network science introduced in this article, we provide a glossary that summarizes their definitions in Appendix C
2
RELATED WORK
2.1
DYNAMIC SPARSE TRAINING
The basic theoretical and mathematical notions of dynamic sparse training together with the several baseline and SOTA algorithms including SET (Mocanu et al.] 2018), MEST (Yuan et al., 2021) and Rigl. (Evci et al.) 2020) which we adopt for comparison in this study are described in the Appendix D We encourage readers who are not experts on this topic to take a glance at this section in Appendix D to get familiar with the basic concepts before going forward and reading this article.
2.2 EPITOPOLOGICAL LEARNING AND CANNISTRACI- NETWORK AUTOMATA THEORY FOR LINK PREDICTION
Drawn from neurobiology, Hebbian learning was introduced in 1949 Hebb 1949) and can be sum- marized in the axiom: "neurons that fire together wire together". This could be interpreted in two ways: changing the synaptic weights (weight plasticity) and changing the shape of synaptic con-
nectivity (Cannistraci et al., 2013a; Daminelli et al.) 2015 Durán et al. 2017 Cannistraci 2018 Narula 2017). The latter is also called epitopological plasticity (Cannistraci et 2013a), because plasticity means "to change shape" and epitopological means "via new topology". Epitopolog- ical Learning (EL.) (Daminelli et al. 2015 Durán et al. 2017, Cannistraci] (018) derived from this second interpretation of Hebbian learning, and studies how to implement learning on networks by changing the shape of their connectivity structure. One way to implement EL is via link pre- diction: predicting the existence likelihood of each nonobserved link in network. In this study, we adopt CH3-L33 (Muscoloni et al., 2020) as link predictor to regrow the new links during the DST process. CH3-L3 is one of the best and most robust performing network automata which is inside Cannistraci-Hebb (CH) theory (Muscoloni et al. 2020) that can automatically evolve the network topology with the given structure. The rationale is that, in any complex network with local- community organization, the cohort of nodes tends to be co-activated (fire together) and to learn by forming new connections between them (wire together) because they are topologically isolated in the same local community (Muscoloni et al. 2020). This minimization of the external links induces a topological isolation of the local community, which is equivalent to forming barrier around the local community. The external barrier is fundamental to keeping and reinforcing the signaling in the local community, inducing the formation of new links that participate epitopological learning and plasticity. As illustrated in Figure 4 CH3-L3 accomplishes this task by incorporating external local community links (eLCLs) at the denominator of formula that ensures common neighbors minimize their interactions outside the local community. The mathematical formula of CH3-L3 and its explanation are reported in Appendix A
3. EPITOPOLOOIIAL SPARSE META-DEEP LEARNING AND CANNISTRACI-HEBB TRAINING
3.1 EPITOPOL.OGICAL SPARSE META-DEEP LEARNING (ESML)
In this section, we propose epitopological sparse meta-deep Learning (ESML), which implements Epitopological Learning in the sparse deep learning ANN. In the standard dynamic sparse training
3

---

Published as a conference paper at ICLR 2024
(DST) process, shown in Figure 6 (left side), there three main steps: initialization, weight update, and network evolution. Network evolution consists of both link removal and link regrowth that in standard DST is random (Erdös-Rényi model like initialization) 6 (left side). ESML initially follows the same steps of standard DST link removal (Figure 6 left side); but then it does not implement the standard random or gradient-based link growth. Indeed, ESML progresses with substantial improvements (Figure 6 right side): new additional node and link removal part (network percolation) and new regrowth part (EL via link prediction). ESML changes the perspective of training ANNs from weights to topology. As described in Section 2.2, Epitopological Learning is general concept detailing how the learning process works by evolving the topology of the network. This topological evolution can take various forms, such as link removal and link regrowth. In this article, we explore the benefits that link regeneration within Epitopological Learning offers to dynamic sparse training. The topological update formula in this article is:
gradient-based Figure
T = T + LPr opp
(1)
Where T represents the topology of the system at state n and T represents the topology of the system at state n + 1. LP can be any link predictor, such as SPM (Lü et al. 2015), CN (Newman 2001), or CH3-L3 (Muscoloni et al.) (2020), to predict the likelihood and select the Topk missing links between pair of nodes in the network. Through an internal examination, we observed that CH3- L33 outperforms SPM and CN, Once the network percolation (Figure 6 right side), which removes inactive neurons and links, is finished, Epitopological Learning via link prediction is implemented to rank the missing links in the network. This is done by computing likelihood scores with the link predictor (CH3-L3 in this article) and then regrowing the same number of links as removed with higher scores. For the necessity to contain the article's length, the details of the two removal steps are offered in Appendix G ESML can be repeated after any backpropagation. The most used update interval in dynamic sparse training can be in the middle of an epoch, after an epoch, or after any certain number of epochs. Thus, ESML is independent of the loss function, and after ESML predicts and adds the new links to the network topology. the backpropagation only transfers the gradient information through the existing links of topology T in an epoch. Our networks are ultra- sparse (1% connectivity) hence the gradient information navigates to find a minimum in a smaller dimensional space with respect to the fully connected. Compared to dense, fully connected network backpropagation, in the link prediction-based dynamic sparse training, the gradient is computed considering only the existing links in the ultra-sparse topology. This means that if we preserve only 1% of the links, we only need to compute the gradients for those existing links and backpropagate them to the previous layers, which in principle could imply a 100 times training speed up.
3.2 CANNISTRACI-HEBD TRAINING
ESML is a methodology that can work better when the network has already formed some relevant structural features. To address these requirements, we propose novel 4-step training procedure named Cannistraci-Hebb training (CHT). The first step is the sparse topological initialization. For instance, the fact that in each layer, the nodes display hicrarchical position is associated with their degree. Nodes with higher degrees are network hubs in the layer, and therefore, they have a higher hierarchy. This hierarchy is facilitated by a hyperbolic network geometry with power-law- like node degree distribution (Cannistraci & Muscoloni] 2022). We propose to adopt a Correlated sparse topological initialization (CSTI), which is obtained by considering links with the strongest Pearson correlations between node features in the input layer. However, in case CHT is applied to intermediate layers of network model such as CNN, the input information is randomized by the convolutional operation applied in the previous layer and this affects the ability of the Pearson cor- relation to extract patterns of associations between the nodes features. Hence, the network structure can be initialized by using generative network models such as Erdōs- Watts-Strogatz, and Barabási-Alle Albert, as we discuss in the Appendix M
The dimension of the hidden layers is determined by a scaling factor denoted as *x"_ A scaling fac- tor of x implies that the hidden layer's s dimension is equal to the input dimension, while a scaling factor of 2x indicates that the hidden layer's dimension is twice the input dimension, which allows the dimension of the hidden layer to be variable. In fact, since ESML can efficiently reduce the dimension of each layer, the hidden dimension can automatically reduce to the inferred size. The second step is sparse weight initialization (SWI), which addresses the importance of weight initial- ization in the sparse network. The standard initialization methods such as Kaiming (He et al. 2015)
4

---

Published as a conference paper at ICLR 2024
HRBST
Tachbe MART7
C 19X03
CPAR330
10.79
7.0%
(991)
47.
+05
28.1%
4,8%
422
435-
20 30.4%
16
154
00043 3MMC
006291001 09.3 XMY
nacor 00-43 KOMM
000-300
HMOH 609
00043 30RU
Figure 1: The comparison of CHH-LL L3 (ESML) and random (SET). In the first row, we report the accuracy curves of CH3-L3 (ESML) and random (SET) on datasets and mark the increment at the 250th epoch on the upper right corner of each panel. The second row reports the value of area across the epochs (AAF) which indicates the learning speed of different algorithms corresponding the upper accuracy plot. Each panel reports the percentage increase.
or Xavier (Glorot & Bengio 2010) are designed to keep the variance of the values consistent across layers. However, these methods not suitable for sparse network initialization since the variance is not consistent with the previous layer. To address this issue, we propose method that can assign initial weights in any sparsity cases. In Appendix [E] we provide the mathematical formula for SWI and the rationale that brought us to its definition. The third step is the epitopological prediction, which corresponds to ESML (introduced in Section
and at the 'heart' of CHT. The fourth step is early stop and weight refinement, during the process of epitopological prediction, it is com- mon to observe an overlap between the links that are removed and added, as shown in the last plot in Figure 8 After several rounds of topological evolution, once the network topology has already sta- bilized, ESML may continuously remove and add mostly the same links, which can slow down the training process. To solve this problem, introduce early stop mechanism for each sandwich layer. When the overlap rate between the removal and regrown links reaches a certain threshold (we significant level of 90%), stop the epitopological prediction for that layer. Once all the sandwich layers have reached the early stopping condition, the model starts to focus on learning and refining the weights using the obtained network structure. Description and ablation tests for each of these steps are in Figure Figure 5 and 8 and we detail the steps of CSTI in Appendix B
4 RESULTS
Because of the page limitation, we detail our experimental setup in Appendix H
4.1 RESULTS FOR ESML AND NETWORK ANALYSIS OF EPITOPOI. LINK REGROWN
4.1.1 ESML PREDICTION PERFORMANCE
In this subsection, we aim to investigate the effectiveness of ESML from a network science per- spective. We utilize the framework of ESML and compare CH3-L3 (the adopted topological link predictor) with randomly assigning new links (which is the regrown method of SET (Mocanu et al., 2018 Figure compares the performance of CH3-L33 (ESML) and random (SET). The first row shows the accuracy curves of each algorithm and on the upper right corner of each panel, we mark the accuracy improvement at the 250th epoch. The second row of Figure [ shows the area across the epochs (AAB), which reflects the learning speed of each algorithm. The computation and ex- planation of AAE can be found the Appendix L Based on the results of the accuracy curve and AAE values, ESML (CH3-L3) outperforms SET (random) all the datasets, demonstrating that the proposed epitopological learning method is effective in identifying lottery tickets (Frankle & Carbin 2019) and the CH theory is boosting deep learning.
5

---

Published as a conference paper at ICLR 2024
Initialization
Hyperbolic emoedcing
CH3-L3 (ESML)
Ieao
2004
2003
204
usi: 0,625 10
Plain presentatio
Random (SET)
aGR 1p p::
y:
Eaye layen
AN
Levare
Epecte
poowar- excement
consistency
moculanity
cheracter path length
00-3 mose
Epocts
Spects
Spachs
Ezodh
Figure 2: Network science analysis of the sparse ANNs topology. Various states of the network in A (a 1% density MLP trained on EMNIST) depicted through both plain representation and 2D hyperbolic em- bedding. This includes the network randomly initialized, the network its final epoch after Random (SET), the network
final epoch after CH3-L3 (ESML) training. The blocks within the network also report two key metrics: the angular separability index (ASI, which signifies the capacity to distinguish communities) and the power law exponent (reflecting degree distribution). The ESML panel also displays the active neuron post-pend post-percolation rate (ANP) trend the epochs. B shows the results of four network topological measures drawn from the average five datasets by CH3-L3 (ESML) random (SET). The shadow area represents the standard error across these datasets.
4.1.2 NETWORK ANALYSIS
Figure 2 provides the analysis from the network science perspective. Figure 2 provides evidence that the sparse network trained with ESML can automatically percolate the network to a signif- icantly smaller size and form a hyperbolic network with community organization. The example network structure of the EMNIST dataset on MLP is presented, where each block shows a plain representation and a hyperbolic representation of each network at the initial (that is common) and the final epoch status (that is random (SET) CH3-L3 (ESML)).
Network percolation. The plain representation of the ESML block in Figure 2 shows that each layer of the network has been percolated small size, especially the middle two hidden layers where the active neuron post-percolation rate (ANP) has been reduced to less than 20%. This in- dicates that the network trained by ESML can achieve better performance with significantly fewer active neurons and that the size of the hidden layers can be automatically learned by the topological evolution of ESML. ANP particularly significant have a large number of redundant neurons that do not contribute significantly to the model's perfor- mance. Moreover, we present the analysis of the ANP reduction throughout the epochs within the ESML blocks/sandwiches. From results, it is evident that the ANP diminishes to a significantly low level after approximately 50 epochs. This finding highlights the efficient network percolation capability of ESML.. At the time, it suggests that prolonged network evolution may be necessary for ESMI,, as the network stabilizes within a few epochs, indicating its rapid convergence.
deep learning because neural networks often
Hyperbolic Hierarchical Organization. Furthermore, we perform coalescent embedding (Cacci- ola et al. 2017) of each network in Figure into the hyperbolic space, where the radial coordinates associated with node degree hicrarchical power-law-like distribution and the angular coordinates with geometrical proximity the nodes in the latent space. This means that the typical tree-like structure of the hyperbolic network emerges if the node degree distribution is power law (gamma exponent the power law distribution S 3 (Cannistraci & Muscoloni 2022)). Indeed, the more
A commented video that shows how FSML shapes and percolates the network structure across the epochs provided at this link https: wwwp://wwww///////////////ow//w wbe wwwp:/twwwwwwww.w..otetot.t..ttt 2v- 11p0bb311

---

Published as a conference paper at ICLR 2024
power law in the network, the more the nodes migrated towards the center of the hyperbolic 2D representation, and the more the network displays a hierarchical hyperbolic structure.
Hyperbolic Community Organization. Surprisingly, we found that the network formed by CH3- L3 finally (at the 250th epoch) becomes a hyperbolic power law (>=2.03) network with the hyper- bolic community organization (angular separability index, ASI = 1, indicates a perfect angular separability of the community formed by each layer (Muscoloni & Cannistraci 2019)), while the others (initial and random at the 250th epoch) do not display either power law (>=10) topology with latent hyperbolic geometry crystal-clear community organization (ASI around 0.6 denotes substantial presence of aberration in the community organization (Muscoloni & Cannistraci] (019))
Community organization is fundamental mesoscale structure of real complex networks (Muscoloni
& Cannistraci 2018 Alessandro & Vittorio 2018), such as biological and socio-cconomical (Xu al. 2020p. For instance, brain networks (Cacciola et al. 2017) and maritime networks (Xu et al.
2020 display distinctive community structures that are fundamental to facilitate diversified func-
tional processing each community separately and global sharing to integrate these functionalities between communities.
The Emergency of Meta-depth. ESML approach not only efficiently identifies important neurons but also learns a more complex and meta-deep network structure. With the prefix meta- (meaning after or beyond) in the expression meta-depth, we intend that there is a second intra-layer depth beyond the first well-known inter-layer depth. This mesoscale structure leverages a topologically separated layer-community to implement diversified and specialized functional processing in each of them. The result of this 'regiona'' layer-community processing is then globally integrated together via the hubs (nodes with higher degrees) that each layer-community owns.
Thanks to the power-law distributions, regional hubs that are part of each regional layer-community emerge as meta-deep nodes in the global network structure and promote hierarchical organization. Indeed, in Figure 2A right we show that the community layer organization of the ESML-learned network is meta-deep, meaning that also each layer has internal hierarchical depth due power- law node degree hierarchy. Nodes with higher degrees in each layer community are also more central and cross-connected in the entire network topology, playing a central role (their radial coordinate is smaller) in the latent hyperbolic space, which underlies the network topology. It is as the ANN has a meta-depth that orthogonal to the regular plan layer depth. The regular plan depth is given by the organization in layers, the meta-depth is given by the topological centrality of different nodes from different layers in the hyperbolic geometry that underlies the ANN trained by ESML.
Network Measure Analysis. To evaluate the structure properties of the sparse network, we uti- lize measures commonly used in network science (Figure 23). We consider the entire ANN network topology and compare the average values in datasets of CH3-L3 (ESML) with random (SET) using 4 network topological measures. The plot of power-law yy indicates that CH3-L3 (ESML) produces networks with power-law degree distribution (7 S 3), whereas random (SET) does not. We want to clarify that SET (Mocanu et al. (2018) also reported that SET could achieve a power-law distribution with MLPs, but in their case, they used the learning rate of 0.01 and extended the learning epochs to 1000. We acknowledge that maybe with higher learning rate and much longer waiting time, SET can achieve a power-law distribution. However, we emphasize that in Figure 2B (first panel) ESML achieves power-law degree distribution regardless of conditions in 5 different datasets and 2 types (MLPs, VGG16) of network architecture. This is because ESML regrows connectivity based on the existing network topology according to a brain-network automaton rule, instead SET regrows ran- dom. Furthermore, structure consistency (Lü et al. 2015) implies that CH3-L3 (ESML) makes the network more predictable, while modularity shows that it can form distinct obvious communities. The characteristic path length is computed as the average node-pairs length in the network, it is a measure associated with network worldness and also message passing and navigability effi- ciency of a network (Cannistraci & Muscoloni] 2022). Based on the above analysis, ESML-trained network topology is ultra-small-worile because it is small-world with a power-law degree exponent lower than 3 (more precisely closer to 2 than 3) (Cannistraci & Muscoloni] 2022), and this means that the transfer of information or energy in the network is even more efficient than a regular small- world network. All of these measures give strong evidence that ESML is capable of transforming the original random network into a complex structured topology that is more suitable for training and potentially valid lottery-ticket network. This transformation can lead to significant improve-
7

---

Published as a conference paper at ICLR 2024
Ulta-spaase advantagg
AAE (Tixning Sperd)
004
Fully-oomeeted oxmmespents
Reis Imstenaeee
0.22
maggetent
Crgg102 Sogtee
Acnuny
Fragelet
Coion
CHI
MARLD0
Ryf.
Ultra-sparse
disadvariage
0.06
Shops
ViaaB
CPAALOD
leeleco
Figure 3: Percentage change in Accuracy-AAE for DST Methods retaining 1% of links compared to fully connected counterparts. Each seatter indicates the best performance of that dataset in Ix or 2x cases. In every lask, CHT (red) surpasses its fully connected counterparts in both performance and training speed. This indicates that it has gained ultra-sparsc advantage.
ments in network efficiency, scalability, and performance. This, in turn, supports our rationale for introducing the novel training process, CHT.
Credit Assigned Paths. Epitopological sparse meta-deep learning percolates the network, trigger- ing a hyperbolic geometry with community and hierarchical depth, which is fundamental to increase the number of credit assigned paths (CAP, which is the chain of the transformation from input to output) in the final trained structure. detailed discussion of the emergence of CAPs in the ESMI. of Figure A is provided in Appendix
4.2 RESULTS OF CHT
Main Results. In Figure we present the general performance of CHT, Rigl., and MEST rel- ative to the fully connected network (FC). For this comparison, the x-axis and y-axis of the left figure represent the change rate (increase or decrease) in accuracy and area across epochs (AAE, indicating learning speed) compared to the fully connected configuration. The value computed as (v fc) fc, where stands for the measurement of the algorithms and represents the corre- sponding fully connected counterparts' performance. In this scenario, algorithms positioned further to the upper right of the figure both more accurate and faster. Table provides the specific per- formance details corresponding to Figure 3 Observing the figure and the table, CHT (depicted in red) consistently surpasses the other dynamic sparse training methods and even the fully connected network in performance and also exhibits the fastest learning speed. Noteworthily. our method con- sistently retains only 1% of the links throughout the training. The preliminary results that we gained on 3 datasets are consistent and show that, in general, the static sparse training via ER, BA, or WS outperformed by the respective dynamic sparse training that, in turn, outperformed by CHT. In addition, in some cases, CHT initialized with BA and WS displays better performance than CHT ini- tialized by ER or CSTI. detailed discussion on different initialization can found in Appendix M
Sparsity. Additionally, delved into sensitivity analysis of sparsity range from 0.999 to 0.9, depicted in Figure C,. which reveals astonishing results. CHT exhibits good performance even at an ultra-sparse level of 0.999, whereas other dynamic sparse training methods lag behind. This strongly suggests that under the guidance of topological link prediction, dynamic sparse training can rapidly and accurately identify the optimal subnetworks, thereby improving training efficiency with minimal information.
Running Time Analysis. We evaluate the running times of CHT in comparison to other dynamic sparse training methods from two aspects. Firstly, we consider the area across the epochs (AAE), which reflects the learning pace of the algorithms. As emphasized in Table [ method con- sistently surpasses other dynamic sparse training algorithms in learning speed, even outpacing the dense configuration. Secondly, assess from the standpoint of actual running time. The acceler-
8

---

Published as a conference paper at ICLR 2024
Table I: Results CHT comparing to DST methods and fully connected network (FC) on 1x and 2x cases. In this table. we report the accuracy (ACC) and area across the epochs (AAE) of each case. We executed all the DST methods in an ultra-sparse scenario (1%). The best performance among the sparse methods is highlighted in bold, and values marked with "*" indicate they surpass those of the fully connected counterparts.
S represents the total number of training samples within the datasct, and C denotes the number of classes present
that datasel.
Googl 5:106
1:30
S:00K
C3100
C:200
N:0RR C:190
5.90K C-I0
ACC
AAR
ACC
AAE
ACC
AAE
ACC
AAE
ACC
AAE
ACC
AAF
PCpp
7E383003
11:221835
13
16/2:00
15.9-11555
51.3 0.12
12 .2.591022
73 1350
245-001
18.09 2016
5151:000
MEST,
S48811.0
51.6 0.15
13:2410.15
18 18:3110147
2006.0.11
61 613511117
76/711031
56.77 18.7718000 on
CHT
5192=9110
44280-0-4
64.0:8004
53.28-0084
4632=8011
07.95-0034
63.90=000-
19:2:1.16
63.51:011
Bidk.
$1.500111
43.35am:55
14,87 44
33.12044
32.125600
43.52e9.01
11 11.72a0.10 10
66:1:2210
34,9229.07
61.769.00
73.359.14
64.5-000
MISES.
11.75 0187
15:91.001
03:91100
41.25:00:5
10
6-61-2011
14.2611000
00.0
T8.11
5444
ation benefits are derived from quicker convergence and the early-stop mechanism integrated into topologial evolution. Notably, the time complexity for the link regrowth process is O(N x dº), where represents the maximum degree of any node within the network. When the network is ultra-sparse, the time required for evolution remains relatively low. Through practical testing on a 1024-1024 network with a density of 1%, the link prediction duration is roughly 1-2 seconds. Although the runtime for the link predictor is slower than the others, we utilize an early-stop ap- proach that truncates topological adjustments within a few epochs. As presented in Figure I03. which exhibits the actual running times of various algorithms from initiation to convergence, CHT is consistently faster than the others. Given CHT's oracle-like capacity to intuitively suggest the placement of new links that boost gradient learning, it greatly abbreviates the convergence period for dynamic sparse training.
Adaptive Percolation. As depicted in Fig,re CHT adeptly percolates the network to preserve approximately 30-40% of the initial node size for relatively simpler tasks such as MNIST when applied with MLPs. In contrast, for more complicated tasks like ImageNet on ResNet50, the ANP remains high. This behavior underscores that CHT isn't just a myopic, greedy algorithm. It seems that it intelligently adjusts the number of active neurons based on the complexity of the task at hand, ensuring no compromise on performance, even in ultra-sparse scenarios. In contrast, other sparse training methods like Rigl. do show lower ANP rate on ImageNet when using ResNet50. However, this reduction is accompanied by a performance drop, possibly because such methods lack task awareness.
This adaptive percolation ability of CHT to provide a minimalistic network modeling that preserves performance with a smaller architecture is typical solution in line with Occam's 's razor also named in science as the principle of parsimony, which is the problem-solving principle ad advocating to search for explanations constructed with the smallest possible set of elements (Gauch et al.) 2003). On this basis, we can claim that CHT represents the first example of an algorithm for "parsimony ultra- sparse training". We stress that the parsimony obtained as an epiphenomenon of the epitopological learning, indeed neither random nor gradient link regrowth is able to trigger the adaptive percolation of ESML or CHT.
5 CONCLUSION
In this paper, we investigate the design of ultra-sparse (1% links) topology for deep learning in comparison to fully connected, introducing a novel approach termed Epitopological Sparse Meta- deep Learning (ESML), which leverages insights from network science and brain-inspired learn- ing to evolve deep artificial neural network topologies in dynamic ultra-sparse (1% 1 links) training. Furthermore, we present the Cannistraci-Heds Training (CHT) process. comprising four essential components to augment ESML's performance. The performance and network analysis indicate that ESML can convert the initial random network into a complex, structured topology, which is more conducive to training and potentially represents a valid lottery ticket network. Further empirical findings demonstrate that anchored by the 4-step methodology. CHT not only learns more rapidly and accurately but also adaptively percolates the network size based on task complexity. This perfor- mance surpasses other dynamic sparse training methods and even outperforms the fully connected counterparts in 6 tests by maintaining just 1% of its connections throughout the training process. This achievement marks a significant milestone in the field of deep learning. The discussion of limitations and future challenges provided in Appendix F

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGE
C.V.C thanks his master student professors: Prof. Giancarlo Ferrigno for inspiring his interest for sparsity in neural networks and Prof. Franco Maria Montevecchi for the support and encouragement to continue this research over the years. We thank Yuchi Liu, Yue Wu, Yuan Yuan Song, Yining Xin, Giada Zhou, Shao Qian Yi, Lixia Huang, and Weijie Guan for the administrative support at THBI; Hao Pang for the IT support at THBI. This Work is supported by the Zhou Yahui Chair professorship of Tsinghua University, the starting funding of the Tsinghua Laboratory of Brain and Intelligence, and the National High-level Talent Program of the Ministry of Science and Technology of China.
REFERENCES
Ilyes Abdelhamid, Alessandro Muscoloni, Danny Mare Rotscher, Matthias Lieber, Ulf Markwardt, and Carlo Vittorio Cannistraci. Network shape intelligence outperforms alphafold2 intelligence in vanilla protein interaction prediction. bioRxiv, pp. 2023-08, 2023.
Muscoloni Alessandro and Cannistraci Carlo Vittorio. Leveraging the nonuniform pso network model as a benchmark for performance evaluation in community detection and link prediction. New Journal of Physics, 20(6):06302,, 2018.
Mukund Balasubramanian and Eric L Schwartz. The isomap algorithm and topological stability. Science, 295(5552):7-7, 2002.
Albert-László Barabási. Network science. Philosophical Transactions of the Royal Society A: Math- ematical, Physical and Engineering Sciences, 371(1987:20120337 2013.
Albert-LászIó Barabási and Réka Albert. Emergence of scaling in random networks. science, 286 (5439):509-512, 1999.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1333-336, 2003.
Zoe Budrikis. Networks with exactly six degrees of separation. Nature Reviews Physics, 5(8):
441-441, 2023.
Alberto Cacciola, Alessandro Muscoloni, Vaibhav Narula, Alessandro Calamuneri, Salvatore Nigro, Emeran A Mayer. Jennifer S Labus, Giuseppe Anastasi, Aldo Quattrone, Angelo Quartarone, et al. Coalescent embedding in the hyperbolic space unsupervisedly discloses the hidden geometry of the brain. arXiv preprint vrXivv/705... /922 2017.
Carlo Vittorio Cannistraci. Modelling seff-organization in complex networks via a brain-inspired network automata theory improves link reliability in protein interactomes. Sci Rep, 8(1):2045- 2322, 10 2018. doi: 10. (038//41198-018-3833
Carlo Vittorio Cannistraci and Alessandro Muscoloni. Geometrical congruence, greedy navigability and myopic transfer in complex networks and brain connectomes. Nature Communications, 13 (1):7308, 2022.
Carlo Vittorio Cannistraci, Gregorio Alanis-Lobato, and Timothy Ravasi. From link-prediction in brain connectomes and protein interactomes to the local-community-paraaa in complex net- works. Scientific reports, 3(1):1613, 2013a.
Carlo Vittorio Cannistraci, Gregorio Alanis-Lobato, and Timothy Ravasi. Minimum curvilinearity to enhance topological prediction of protein interactions by network embedding. Bioinformatics, 29(13):1199-i209 2013b.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 international joint conference on neural networks (ICCN), pp-
2921-2926. IEEE, 2017.
Simone Daminelli, Josephine Maria Thomas, Claudio Durán, and Carlo Vittorio Cannistraci. Com- mon neighbours and the waaallmmmmmttt--rrrattta for topological link prediction in bipartite networks. New Journal of Physics, 17(13)1 nov 2015. doi: 10. 10.18881337-263000 2633//117//////11/
113037.
/ooi http:///e.......oooo..0.0000////
10

---

Published as a conference paper at ICLR 2024
Claudio Durán, Simone Daminelli, Josephine M Thomas, V Joachim Haupt, Michael Schroeder, and Carlo Vittorio Cannistraci. Pioneering topological methods for betwork-baser drug-target prediction by exploiting a brain-network self-organization theory. Briefings in Bioinformatics, 19(6):1133-1200 04 2017. doi: 10. 1090////b//bb0041. URL htttps://ae http:///doo.ogg/////// 1009///
bib/bbx041
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lot- tery: Making all tickets winners. In Proceedings of the 37th International Conference on Ma- chine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 2943-2952. PMLR, 2020. URUhttp::/procediing: .mlr. press/ v1 www19//vci22...i.
Fan Feng, Lu Hou, Qi She, Rosa HM Chan, and James T Kwok. Power law in deep neural net- works: Sparse network generation and continual learning with preferential attachment. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
met/forue d= b3RcF7
Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2001.00744 2023.
Hugh G Gauch Jr, Hugh G Gauch, and Hugh G Gauch Jr. Scientific method in practice. Cambridge University Press, 2003.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Yee Whye Teh and D. Mike Titterington (eds.), Proceedings of the Thirteenth Inter- national Conference Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp. 249-256. JMLR.org, 2010. URL http::/ppocceeiinne
mwr. press v9 wttp://tt1loo......// html
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Donald Hebb. The organization of behavior. emphnew york, 1949.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- crated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 34:21099-11111, 2021.
Alex Krizhevsky, Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL https:// wwtp://wwww//. cs www.ttorooooooooo.. wttp://tto..dd/kkieeeeeeeeeaeaee featurrreeeeeeueeeeeeeee -attuuuress-00----- 2009 TR.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky (ed.), Advances Neural Information Processing Systems [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pp. 598-605 Morgan Kaufmann, 1989. Uttp:tt:://papen ni nips. wtt.scccpppeeeeeee 250 10::000pptii imal brain brain-damage
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognniion.... Proc. IEEE, 86(11):2278 23244, 1998. doi: 1299799977..... URL
https://doi www.oiooorgg/o11 12697 726791

---

Published as a conference paper at ICLR 2024
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based on connection sensitivity. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreview.
d=B V
Jiajun Li and Ahmed Louri. Adaprune: An accelerator-aware pruning technique for sustainable cnn accelerators. IEEE Transactions Sustainable Computing, 7(1):47-60, 2021.
Ming Li, Run-Ran Liu, Linyuan Lüi, Mao-Bin Hu, Shuqi Xu, and Yi-Cheng Zhang. Percolation on complex networks: Theory and application. Physics Reports, 907:1-68, 2021.
Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, 26(1):41, 2023.
Linyuan Lüi, Liming Pan, Tao Zhou, Yi-Cheng Zhang, and H. Eugene Stanley, Toward link pre- dictability of complex networks. Proceedings of the National Academy of Sciences, 112(8): 2325 2330, 2015, doi: 10. 10.1073/pnas. 00073//ma..1444441111 1424644112.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint w0Xiv:2/04.08838, 2021.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications. 91):1112, 2018.
A. Muscoloni, U. Michieli, and C.V. Cannistraci. netorrk network networks. preprints, 2020. URL http:::00.2204///pprrrrrt0222220
v2 automaat autommaa modelling of complex
Alessandro Muscoloni and Carlo Vittorio Cannistraci. A nonuniform popularity-similari opti- mization (mpso) model to efficiently generate realistic complex networks with communities. New Journal of Physics, 20(5):05000, 2018.
Alessandro Muscoloni and Carlo Vittorio Cannistraci. Angular separability of data clusters net- work communities in geometrical space and its relevance to hyperbolic embedding. arXiv preprint arXiv: 2019.
Alessandro Muscoloni, Josephine Maria Thomas, Sara Ciucci, Ginestra Bianconi, and Carlo Vit- torio Cannistraci. Machine learning meets complex networks via coalescent embedding in the hyperbolic space. Nature communications, 8(1):1615, 2017.
Vaibhav et al Narula. Can local pocal-community-prrrdiiinn and epitopological learning enhance our un- derstanding of how local brain connectivity able to process, learn and memorize chronic pain? Applied network science, 2(1), 2017. doi: 10.1007/s4 -17-00004
Mark EJ Newman. Clustering and preferential attachment in growing networks. Physical review E, 64(2)x025102, 2001.
Mark EJ Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences, 103(23):8577-86)) 85882, 2006.
Mark EJ Newman and Duncan Watts. Scaling and percolation in the small-world network model. Physical review E, 6036), 1999.
Christopher L Rees, Keivan Moradi, and Giorgio A Ascoli. Weighing the evidence in peters' rule: does neuronal morphology predict connectivity? Trends in neurosciences, 40(2):63-71, 2017,
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34.8883-8595. 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 15(3):211255) 2015. doi: 10. 1007/1 1263-015-0816-y.
12

---

Published as a conference paper at ICLR 2024
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Duncan Watts and Steven H Strogatz. Collective dynamics of 'small-world networks. nature, 393 (6684):440 442, 1998.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: (0Xi::7708,07747, 2017.
Mengqiao Xu, Qian Pan, Alessandro Muscoloni, Haoxiang Xia, and Carlo Vittorio Cannistraci. Modular gateway-ness connectivity and structural core organization maritime network science. Nature Communications, 1 11(1):2849, 2020..
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-econome sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838 14:20838-20850, 2021.
Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang, Yanzhi Wang, and Jian Ren. Layer freezing & data sieving: Missing pieces of generic framework for sparse training, arXiv preprint arXiv:2209. 11204, 2022.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. An efficient plug-and-play post-training pruning strategy in large language models. 2023a.
Yuxin Zhang, Minghao Lin, Mengzhao Chen, Fei Chao, and Rongrong Ji. Optg: Optimizing gradient-driven criteria in network sparsity. arXiv preprint arXiv:2201. 128866 2022.
Yuxin Zhang, Yiting Luo, Mingbao Lin, Yunshan Zhong, Jingjing Xie, Fei Chao, and Rongrong Ji. Bi-directional masks for efficient n: M sparse training. arXiv preprint arXiv:2302.0660 2023b.
13

---

Published as a conference paper at ICLR 2024
iLCL
eLCL
CNs
seed nodes
?
u
V
An explanatory example of topological link prediction performed using CH theory bipartite net- works. The two yellow nodes the seed nodes whose non-observed interaction should be scored with likelihood. The green nodes represent the common neighbors (CNs) of the seed nodes in the bipartite network. The red links denote the interactions between CNs, which are referred to as internal local community links (iLCL). The blue links represent the interaction between CNs and external nodes of the community that are not shown in this example which termed external local community links (eI.CL.).
Figure 4:
A MATHEMATICAL FORMULA AND EXPLANATION OF CH3-L3
CH3-L3 is the link predictor employ in the epitopological sparse meta-deep learning phrase of ESML and CHT. In this section, we delve into both the formula and underlying principles of CH3-L3 enhance understanding of this concept.
CH3-L3 can be formalized as follows:
CH3 L)(,
E
21,32E))( v(1 de (1 + de
where: u and e the two seed nodes of the candidate interaction; 21, 22 the intermediate nodes on the considered path of length three, which are seen as the common neighbors of and on the bipartite network; dēzı. ze,, are the respective number of external local community links of each common neighbor; and the summation is executed over all the paths length three (13) between the seed nodes and v. The formula of CH3 Ln is defined the path of any length, which adapts to any type of network organization (Muscoloni et al.) 2020). Here, we consider the length 3 paths of each ANN sandwich layer separately, which has bipartite network topology organization. And, bipartite topology is based on the path of length 3, which displays a quadratic closure (Daminelli al. 2015). During the link prediction process, all the missing links in the network will be ranked based on scores generated by the CH3 L3 formula. higher score suggests a higher probability of the link being present the next stage of the network.
CH3(C H3 I.3) because the topological regrowth process is implemented
B CORRELATED SPARSE TOPOLOGICAL INITIALIZATION (CSTI)
To enhance the performance of the link predictor, we proposed the Correlated Sparse Topological Initialization (CSTI) to initialize the topology of the layers that can directly interact with the input features. We explain here the detailed steps of CSTI.
As shown in Figure AA CSTI consists of steps. 1) Vectorization: During the vectorization phase, we follow a specific procedure to construct matrix of size x M, where represents the number of samples selected from the training set. M denotes the number of valid features obtained by excluding features with zero variance (V aro) among the selected samples. 2) Feature selection: Once we have this n X M matrix, we proceed with feature selection by calculating the Pearson Correlation for each feature. This step allows to construct correlation matrix. 3) Connectivity selection: Subsequently, we construct a sparse adjacency matrix, where the positions marked with "I" (represented as white in the heatmap plot of Figure (AA) correspond the top-k% values from
14

---

Published as a conference paper at ICLR 2024
Correlated Sperse Tapological Iniiaaizzzion (CSTI)
Vedastoster
Postare corsiaiion
Cornecivity adlecion
Assenteing keolopioaly
Aduanny
f
Top-t'
5
Af
2rAdaay Muze
Fastura:: Vars
B
Sparse Weight taation
Iouu Luyer Degran Distt bution
SWW(N]
poin
Figure 5: Main innovations introduced in CHT (CSTI and SWI). A shows an example of how to construct the CSTI on the MNIST dataset, which involves four steps: vectorization, feature correlation, connectivity selection, and assembling topological bubbed network blocks. presents real network example of how SWI assigns weights the sparse network. C displays the input layer degrce distribution along with the associated heatmap of 2x case on the MNIST figure template and the node degree distribution.
the correlation matrix. The specific value of depends on the desired sparsity level. This adjacency matrix plays a crucial role in defining the topology each sandwich layer. The dimension of the hidden layer is determined by scaling factor denoted 'x'. A scaling factor of 1x implies that the hidden layer's dimension is equal to the input dimension, while a scaling factor of 2x indicates that the hidden layer's dimension twice the input dimension, which allows the dimension of the hidden layer to be variable. In fact, since ESML can efficiently reduce the dimension of each layer, the hidden dimension can automatically reduce the inferred size. 4) Assembling topologically hubbed network blocks: Finally, we implement this selected adjacency matrix to form our initialized topology for each sandwich layer.
C GLOSSARY OF NETWORK SCIENCE
In this section, we introduce some basic notions and concepts of network science involved this article. Given simple undirected network G(V, F) where V is the of nodes and E is the set of links. N is the number of nodes in the article (N [V]) and M is the total number of edges in the network (M [E). A is the adjacency matrix of the network, Aij 1 if nodes and are connected and A. otherwise.
Scale-free Network. A scale-free network (Barabási & Albert 1999) a type of node degree het- erogeneous network characterized by highly uneven distribution of connections among its nodes, where a small number of nodes (hubs) have a very high number of connections, and a large number of nodes have few connections. The degree of scale-free networks always follows power-law dis- tribution P(k) ~ k-,, where is constant in the range of 2 to 3. In contrast, the degrees of a random network always follow a Binomial distribution (Barabási 2013)
Characteristic Path Length. The Characteristic Path Length is fundamental network measure to quantify the average shortest distance between pairs of nodes within network. It provides insight into how efficiently the information signal is transferred across the network average. The formula can be expressed as:
L=
N(N 1) Σ d(i,j)
15

---

Published as a conference paper at ICLR 2024
Where d(i.j) is the shortest path length between node i and node j.
Watts-Strogatz Model and Small- World Network. A Watts-Strogatz model of small-world net- work (Watts & Strogatz 1998) is characterized by (1) high clustering coefficient and (2) short average path length among nodes. This short average path lengths, on the other hand, mean that there short chain of connections between any two nodes (in social networks, this effect is known as six degrees of separation (Budrikis (023)). Therefore, the Watts Strogatz model (Watts & Stro- gatz 1998) well-known for its 'small wooldd' properties, characterized by high clustering and short path lengths. This model has parameter / and displays transitions from regular high clustered lattice (.3 0) to random small-world graph (3 = 1). Intermediate values of B can generate net- works that retain clustering while displaying small-world connectivity, striking a balance between regularity and randomness.
More in general, the necessary condition to be small-world network (Newman & Watts 1999) is to have (2) a short average path length among nodes, which mathematically is defined by network where the path length L between two randomly chosen nodes grows proportionally to the logarithm of in the network, that is:
I. log N
Structural Consistency. Structural consistency (Lü et al. 2015) is an index that is based on the first-order matrix perturbation of the adjacency matrix, which can reflect the inherent link pre- dictability of a network and does not require any prior knowledge of the network's 's organization. perturbation set A randomly selected from the original links set E, where E is the set of |A The structural consistency is
top-I, ranked links given by SPM (Lü et al. 2015 L computed as:
0 EELNAF
A
Modularity In network science, modularity (Newman 2006) quantifies the ability divide a net- work into modules (also known as communities or clusters). The modularity of a network ranges from -1 to 1. A high modularity score (close 1) suggests that the network exhibits dense nections between nodes within modules but sparse connections between nodes in different modules. A modularity score of 0 indicates that the network structure lacks any discernible community ganization. Nodes within the network are randomly connected, and there are no clear groupings or clusters. Imagine network where interactions between nodes are essentially uniform: no specific communities emerge. Modularity 0 implies a homogeneous network without distinct communi- ties. When the modularity score approaches -1, it signifies an anti-community structure. Nodes are strongly interconnected across the entire network, and there is little differentiation into separate groups. Think of it as network where everyone is closely linked to everyone else, regardless of any community boundaries. In practice, achieving modularity of exactly -1 is rare, but values close to it indicate a highly cohesive network. However, such extreme anti-community behavior is not common in real-world networks. The formula for modularity Q is:
1 2m
kg 2m
Q =
Au
8(c,,;)
Where: Ai represents the adjacency matrix of the network, h; and ky are the degrees of nodes and j. respectively. 5(c,, ej) is the Kronecker delta function, which equals if nodes and are in the same community and otherwise.
Coalescent Embedding. Coalescent embedding (Muscoloni al. 2017) a class of topological- based machine learning algorithms for nonlinear unsupervised dimensionality reduction and embed- ding networks in a geometric space such as hyperbolic one. In our case, coalescent embedding is used to map networks that have latent hyperbolic geometry onto the two-dimensional hyperbolic space. The approach always involves 4 steps: 1) pre-weighting links with topological rules that approximate the underlying network geometry (Cannistraci & Muscoloni] 2022); 2) non-linear di- mension reduction including three (but not limited to these) manifold based methods: Isomap (Bal- asubramanian & Schwartz. 2002), non-centered Isomap (Muscoloni al. 2017), Laplacian eigen-
16

---

Published as a conference paper at ICLR 2024
Dynamic Sparse Training
Epitopological Sparse Meta-deep Learning
(Mocanu 2018 Evet 2020)
Network percolation
EL (Link prediction)
Inttialization
Link Removal
Inactive Neurons Removal
hrkking irtenval
Bindwent
Sendwerd
two rounds
Swnshwicht
SandwictO
CH3-L3
hearte
New tnks
Incomplete Adjustment
Figure 6: Illustration the standard DST progress and ESML. ESML incorporates two additional steps for network evolution, namely network percolation and EL(link prediction). Network percolation involves Inactive Neurons Removal (INR) and the Incomplete Path Adjustment (IPA). In the epitopological learning (EL) part, which is based link prediction, after dividing the entire network into several sandwich layers, the chosen link predictor (CH3-L3) separately calculates link prediction score each non-existing link and then regrows links with top-k scores equal number to those that were removed.
maps (Belkin & Niyogi 2003) and two minimum corirummmtuuviiinrrtttyee -ased methods: minimum curvilin- ear embedding (Cannistraci et al., 2013b) and non-centered minimum curvilinear embedding (Can-) nistraci et al. 2013b); 3) generate angular coordinates; 4) generate radial coordinates. In this way, we can effectively embed the original network into 2D hyperbolic space, reconstructing latent manifold structure for visualization.
D DYNAMIC SPARSE TRAINING
In this section, we introduce some basic concepts and relevant algorithms of dynamic sparse training. Let's first define how the network will learn through gradients with the sparse mask created by dynamic sparse training. Consider / E RR as the weights matrix of the entire network, M E 1) the adjacency matrix of the current existing links inside the model. The general optimization function can be formulated as:
min L(O & M; s.t. || kk
0
where the L(-) is the loss function and D is the training dataset. is the remaining weights in the entire number N of weights regarding the preset sparsity S, k N (1 1 S). The main difference between dynamic sparse training and static sparse training is whether it allows the mask M to change dynamically. A standard dynamic sparse training method (Mocanu al., 2018 Evei al., 2020 Yuan et al. (2021). as illustrated in Figure [A, begins with sparse initialized network and undergoes the prune-and-gron regime with multiple iterations after each update interval of weights. In one update iteration, the gradient of the weights can be formulated as:
Oc 00, a(0, O M,)
M,
á indicates the layer index the model. The main differences between the current standard dynamic sparse training methods are how remove links and how to regrow new links. SET (Mocanu et al., 2018), as the first dynamic sparse training method, removes links with weight magnitude and regrows new links randomly. Different from the removal process of SET, MEST (Yuan al.
17

---

Published as a conference paper at ICLR 2024
2021) also regrows new links randomly while removing weights with the combination of weight magnitude and gradient magnitude of the existing links. RigL (Evci et al. 2020 utilizes the gradient magnitude of the missing links to regrow the new links. Some other innovations in dynamic sparse training are adding some tricks or making it more hardware-friendll. (Zhang et al. (023b) introduces method called Bi-Mask that accelerates the backpropagation process of sparse training with N:M constraint (Mishra et al. 2021). SpFDE (Yaan et al. (0222) introduces layer freezing and data sieving strategy to further accelerate the training speed of DST. We let you notice that Mixture-of-Expert (MoEs) (Riquelme et al.) 2021) is also type of DST method that is widely used in training large language models since the experts are always sparse during training.
However, most methods for training sparse neural networks are not actually sparse during training. For example, Rigl. (Evci et al., 2020) needs to collect gradients for missing connections to regrow new ones. OptG (Zhang et al., (002) trains an extra weight mask to learn the network topology. Only, methods like SET and MEST that randomly regrow connections actually use just the sparse model during training. However, they don't work well as methods that use gradient information. So in this paper, we introduce an efficient, gradient-free, and brain-inspired method from network science called epitopological learning to evolve the sparse network topology.
E SPARSE WEIGHT INITIALIZATION
Our derivation mainly follows kaiming(He et al. 2013). Similar to SNIP et al. 2019) we Then, for densely connected layer, we
introduce auxiliary indicator variables C {0,1) mxn have:
y1 = C O Wix bi.
(5)
where ① denotes the Hadamard product, Wmxn is the weight matrix, xx1 is the input vector, m is a vector of biases and y is the output vector. We use / to index layer.
Then, for i-th element of the pre-activation layer yı, by ignoring the influence of bias, can get its variance:
Var Var
Σ
of W/l z/1.
As we initialize elements in C to be independent and identically distributed random variables, also same for elements in W. We also assume elements in are mutually independent and share the same distribution. C, W and x are independent of each other. Variance of y will be:
Varlyil n aarl w 2+1 = n((Var(c) (V arlw] + (V + (²)) ()))) (7)
where y is mean. We assume e follows Bernoulli distribution, then the probability mass function is:
SS c=0
S()) - 1- c=1'
(8)
where S denotes the sparsity. We could infer that Varld] S(1 - S) and 2² = (1 S)- Same
as previous work (He et al. 2015). we define follow zero-mean Gaussian distribution, therefore 2. 0. And when activation layer is ReLU, Var[at] + m²; 2 could be changed to:
1 Var Warly ar Then equation
1
Varlyil n(aalld + (V (V + +
n(( - [eVarru]Varlvt-... (9)
2
We expect the variance of signal to remain the same across layers Varlyd = Var (arlwin) then:
2
Var[w]
(10)
n(1 ()*
We denote indim as n, which represents the dimension of the input layer of each sandwich layer. The constant gain varies depending on the activation function used. For example, when using ReL.U,
18

---

Published as a conference paper at ICLR 2024
set gain to 22. The resulting values of w are sampled from normal distribution, resulting in the following expression:
gain
(WI ~ ~N((,0)), /Var[w]
(11)
v(1 S) indim
F LIMITATIONS AND FUTURE CHALLENGES
With respect to the dynamic sparse training SOTA (RigL and MEST), the computational evidence in Figure 3 on 6 empirical tests obtained in different combinations of network architecturesdlatssets (VGG16 on TinylmageNet; GoogLeNet on CIFAR100 and TinylmageNet; ResNet50 on CIFARI00 and ImageNet; 155 on CIFAR 100), demonstrate that CHT offers a remarkable ultra-sparse (1% connectivity) advantage on the fully connected baseline, whereas the current SOTA cannot. For the sake of clarity, we acknowledge that in this study, we do not compare with the specific SOTA in the respective data architecture types because they often include methodologies with tailored technical features that are network and data-dependent. Including these specific SOTA, would make it difficult fair comparison across data and networks because it would largely depend on the way we adapt the ultra-sparse training strategy to the specific case. The attempt of this study is instead to evaluate whether, and in which cases, CHT can help achieve ultra-sparse advantage in comparison to the fully connected baseline, regardless of the specific network or data considered.
A limitation of not only our proposed method CHT but of all current sparse training methods is the current difficulty in deploying such unstructured sparse training methods efficiently in hardware like GPUs or specialized AI accelerators. Some studies (Hubara al., 2021 Zhang et al. 2023b) attempt to integrate semi-structured sparsity patterns to speed up the network, but this often comes the cost of reduced performance. Most hardware is optimized for dense matrix multiplications, while sparse networks require more dedicated data access patterns. But this limitation could also represent a challenge to address in future studies because the gain in computational speed could be relevant considering that the backpropagation and the gradient search would be implemented with respect to an ultra-sparse structure. Another important limitation to address in the next studies is to better understand the process of node percolation and to design new strategies of node removal that allow setting priory a wished value of node sparsity lower-bound.
The future challenges of dynamic sparse training should be in opinion to identify novel adaptive mechanisms of epitopological parsimony learning that, for different datasets and network architec- tures, can adaptively learn any aspect of the network structure such node layers, node size, and link sparsity.
G LINK REMOVAL AND NETWORK PERCOLATION IN ESML
We detail our link removal methods and how we implement network percolation in this section.
Firstly, in the removal phase, F.SLI adopts the commonly used DST removal method that prunes links with lower weights magnitude (Figure [ left side), which are considered less useful for the model's performance. We also try an alternative removal method, which combines weight and gradient magnitude as adopted by MEST. However, this approach did not yield any performance improvement us, at least in the tested datasets.
After the process of standard DST link removal, ESML checks and corrects the topology for the presence of inactive neurons that either do not have a connection on one side of the layer or do not have any connections on both sides of the layer. The forward and backward processes can get stuck when information passes through these inactive neurons. Moreover, if a neuron becomes inactive, no new links will be assigned to it in that network according to the formula of CH3-L3. Thus, ESML implements the Inactive Neurons Removal (INR) to eliminate those inactive neurons from the topology. In particular, in the case that inactive nodes possess connections on one layer side, we need to conduct Incomplete Path Adjustment (IPA) to remove those links. Since IPA may produce some extra inactive neurons, process of INR and IPA as network percolation because it is reminiscent of the percolation theory in network science, which studies the connectivity behavior of networks when a fraction of the nodes
execute INR and IPA for two rounds. We termed the coupled
19

---

Published as a conference paper at ICLR 2024
or edges are removed (Li et al. 2021). If there are still some inactive neurons, we leave them in the network and clean up them during the next evolution.
H EXPERIMENTAL SETUP
In Section we present experimental results and network measurement analyses for ESML and SET Mocanu et al. 2018) across three datasets (MNIST (LeCun et al.) 1998). Fash- ionMMIIST (Xiao et al. 701 EMNIST (Cohen et al. 2017)) using MLPs, and two datasets (CI- FARIO Krizhhvsky| 2014). In addition, present comparative analysis of CHT, the 4-step training methodology, on VGG16, GoogLeNet (Szegedy et al. 2015), ResNet50 (He et al. 2016). and ResNet152 across CIFAR100, ImayeemageNet, and MmtaeNet([Russakoon representative link predictor for ESML and CHT.
CIFAR 100 (Krizhevsky 2009) with VGG16 (Simonyan & Zisserman,
al. 2015). CH3-L3 is chosen the
The models employing dynamic sparse training in this section adopt the final three fully connected layers (excluding the last layer) of all the convolution neural networks. It's noteworthy that the default configuration for GoogLeNet and ResNet comprises just one fully connected layer. We ex- panded this to three layers to allow the implementation of dynamic sparse training. All experiments are conducted using three random seeds, and a consistent set of hyperparameters is applied across all methods.
Hidden Dimension. It's important to highlight that the hidden dimension settings differ between the evaluations of ESML and CHT in this article. When assessing ESML and comparing it with SET, we adopt the same network structure hidden dimension set to 1000 to ensure a fair comparison with SET. For the CHT evaluation, we utilize two distinct hidden dimension settings: ' which corresponds to the same size as the input dimension, and '2x', which double the size of the input dimension. All the algorithms are tested considering these two dimensions.
outlined in SET (Mocanu et al. 2018). where the
Baseline Methods. To highlight the potential of link prediction in guiding the formation of new links, in this article, we compare ESML. with SET (Mocanu al. 2018) which serves as foun- dational method in dynamic sparse training that relies highlight the advantages of CHT, we compare it with two state-of-the-an dynamic sparse training algorithms: RigL (Evci et al. 2020h and MEST (Yuan et al. 202I). Both of these algorithms follow the prune-and-regrow approach. RigL prunes links by weight magnitude and regrows them based on the gradient magnitude of missing links. Additionally, it features gradually reduces the proportion links pruned and regrown over the training epochs. Conversely, MEST employs combination of weight and gradient magnitudes of the existing links for link moval and subsequently regrows new links at random. Notably, MEST's "EM&S" tactic, which enables training from denser network to sparse one, is also implemented in this article.
the random regrowth of new links. To
adaptive Ç strategy that
Sparsity. In Section 4 maintain sparsity level of 99% (equivalent to 1% density) for all evaluations except for the sparsity sensitivity test. The sparsity pattern adopted this study is unstructured. This is because both pattern-based and block-based patterns impose constraints on the network's 's topology, potentially detrimentally affecting model performance. Nonetheless, even with unstructured sparsity, CHT facilitates a quicker training acceleration due its enhanced learning speed and rapid convergence.
I SENSITIVITY TEST OF ESML AND CHT
We investigate the influence of hyperparameters which also significant for the training of ESML and CHT.
Learning rate. In particular, the learning rate (Ir) is a crucial factor that controls the size of the steps taken in each mini- during training. In dynamic sparse training, the learning rate is even more dominant. As an example, consider a scenario where weights are assigned to new links with value of zero. If the learning rate is set to a very small value, these weights may not be able to
20

---

Published as a conference paper at ICLR 2024
A
cifferent
dffferent
dfferent MMP
07-
64-
8
d fftrret di Accuracy
di dffirent di AAE
dfffrent AMP
01.3
01-2
MART DMVET
HART DMMET
07.
MUBT pexi, Vress
MABT) koJes
$2.5
04-
ENess Ineit syect
10
10
different sparsiy: WWST
diflrentt spantly: EMWIGT
sparsily
apensily
Figure 7: Sensitivity test of learning rate, update interval, and sparsity ESML and CHT. The evaluation measures involve Accuracy, area across the epoch (AAE). and ANP (active neuron post-percolation rate). A shows the impact of learning rate on the MNIST and EMNIST datascts when applied to MLPs. B investigates how varying the update interval of topology affects performance. C illustrates the performance of dynamic sparse training across different sparsity levels on MNIST and EMNIST, with 1x hidden dimension ranging from 0.999 to 0.9. It's noteworthy that even 99.9% sparsity, CHT can still learn some information, whereas the other methods cannot. All the experiments are with random seeds.
increase decrease beyond the positive or negative threshold. Consequently, the weights may get directly pruned in the next round. In Figure we fix ui=1, sparsity=0.99, and vary Ir from [0.01, 0.005, 0.001, 0.0005, 0.0001]. The conclusion drawn from A and B is that low learning rate could lead to more percolated network with lower ANP, indicating the effectiveness of ESML in reducing the network size. However, if the learning rate is too low, the network may not be able learn well due to the slow weight update.
Update interval. The update interval (ui) encounters a similar challenge. If the interval is not appropriately set, it can lead to suboptimal outcomes. The choice of update interval impacts how frequently the network structure is adjusted, and an unsuitable value may hinder the network from effectively adapting to new information. The results show that larger ui could make the network learn slower and result in a less percolated network.
Sparsity. In our evaluation, we also place significant emphasis on the role of sparsity when as- sessing the performance of dynamic sparse training methods. Our objective is to delve into the capabilities of these methods in scenarios of extreme sparsity. In Figure 7 we keep ui=l and Ir=0.01 constants while manipulate the sparsity levels within the range of [0.999, 0.99, 0.95, 0.9]. In this setup, we present a comparative analysis of CHT against SET, MEST, and RigL. The pur- pose is emphasize the differences among these dynamic sparse training methods as they operate under varying levels of sparsity. Analyzing the results on MNIST and EMNIST, we observed a remarkable trend: even in scenarios with an extreme sparsity level of 99.9% (density=0.1% where the connectivity is highly sparse, CHT manages to extract and transfer meaningful information. In contrast, both Rigl. and SET failed to learn anything significant. This disparity in performance can be attributed to the capabilities of link prediction methods in handling sparse information. CHT's ability to utilize topology for predicting new connections becomes especially advantageous in sit- uations of extreme sparsity. On the other hand, SET and MEST, which randomly regenerate new connections, face a vast search space, making it exceedingly challenging to identify the correct subnetwork. Meanwhile, RigL struggles in such scenarios due to the limited availability of credit assignment paths (CAPs) in the extremely sparse connectivity, rendering gradient information less useful.
21

---

Published as a conference paper at ICLR 2024
Number of Hidden Layers In the experiments detailed in Section 4, we consistently uti- lize three sparse layers fol- lowed by single fully con- nected layer in each model con- figuration. Additionally, we ex- plore configurations with just one sparse layer and one fully connected layer, presented in Table 2 This is com- pared to the original architec- ture of ResNet152 (only one fully connected layer). The re- sults demonstrate that the CHT outperforms both fully con- nected alternatives and other DST methods. Moreover, CHT also exceeds the performance benchmarks set by the original ResNet152 configuration.
Table 2: Results of CHT comparing to DST methods and fully connected network (FC) on 1x and 2x cases with only sparse layer. In this ta- ble, we report the accuracy (ACC) and area across the epochs (AAE) of 5esNet152 on CIFARIO0. We executed all the DST methods in an ultra- sparse scenario (1%). The best performance among the sparse methods is highlighted in bold, and values marked with "g" indicate they surpass those of the fully connected counterparts.
ResNet152-CIFARE
S:50K C:100
ACC
AAE
Original
79.76+00.19
68.26-F0.11
FCIx
79.2811016
67.33+0.21
RigLix
79,81
68.07:00.09
MEST 1x
79.97+0.11
68.42+0.00*
CHT (WSS, 1)
80.01+0.02**
68.91+0009*
FC2x
79.22:10.12
67.69
RigL2x
79.59++0.03
67.95+t0.1
MEST
79.86+00.0**
68.28+00.04
2WS, (S,
80.15+0015*
68.9+..13*
J CHT RESULTS ON THE MLP TASKS
In the main text, we showcase the results of CHT on larger datasets and models, while here, we detail CHT's performance on basic datasets and models. From the results Table 3 CHT sur- passes other dynamic sparse training methods in accuracy on MNIST (MLP) and EMNIST (MLP) while achieving similar performance as others on Fashion MNIST. It's noteworthy that while certain dynamic sparse training methods might slightly surpass CHT in some instances, CHT consistently exhibits higher AAES across tasks, indicating faster learning pace.
K ABLATION TEST OF EACH COMPONENT IN CHT
In this section, we evaluate each component introduced in CHT (Sparse Topological Initialization, Sparse Weight Initialization, and early stop).
Sparse Topological Initialization. In this article, we discuss 4 types of initializations: Correlated Sparse Topological Initialization (CSTI), Erdōs-Rényi (ER), Barabási-Albert, and Wttts-Strogatzz (WS). Note that the WS model is equivalent to the ER model if 33 equals 1. For the layers that directly receive information from the imputs, we adopt Correlated Sparse Topological Initialization (CSTI). We first report the initialization structure in Figure 8C which is a CSTI example with a 2x version of the hidden dimension MNIST. We construct the heatmap of the input layer degrees and node distribution density curve. It can be observed that with CSTI, the links are assigned to nodes mostly associated with input pixels in the center of figures, and indeed, the area at the center of the figures is more informative. We also report the CSTI-formed network has an initialized power law exponent ? 5.4, which indicates topology with more hub nodes than In Figure 8 we demonstrate that with CSTI, ESML can achieve faster learning speed compared to ER initialization, and in Table we compare CSTI with 3 types of random initialization: BA, WS, and ER (WS, B = 1). The results that we gained on 3 datasets are consistent and show that, in general, the static sparse training via ER, BA, or WS is outperformed by the respective dynamic sparse training that, in turn, is outperformed by CHT. In addition, on the Fashion MNIST dataset, CHT initialized with WS and varying 3 values display better performance than CHT initialized by CSTI.
random network with > 10.
For the layers in the mediate positions of the models, we compare distinct models of random initialization: ER, BA, and WS in Table We conduct the ablation test on these different differenttppologico. initializations on CNNs. The findings indicate that the CHT achieves optimal performance on the WS network when / varies between 0.25 and 1.0 1.0, depending the specific case. Nevertheless,
22

---

Published as a conference paper at ICLR 2024
CSTI us ER anzaainn
5WI Keiming
warty stop no early stop
Dverlap acroes spachs
80-
Threshoke: 90%
eartyy stop opochh
eeatly stop spoch
20
10 30 40
20
50
30
10
60
30
50
2o0cha
Spacdta
Epoddo
Epeedts
Figure 8: Ablation test of each component in CHT on the MLP architecture for MNIST dataset with 1x hidden dimension. The first three ligures demonstrate the efficiency of different strategies while keeping the other components fixed and only varying the one being compared. The last figure illustrates the overlap rate of the new links and removed links, providing insight into the necessity of utilizing an early stop of the evolutionary process.
selecting the most suitable initialization for each scenario remains a topic for further exploration. For the sake of simplicity, ER (WS, B = 1) initialization is utilized as the default approach in the experiments presented in the main text.
Sparse Weight Initialization (SWI. On the second panel of Figure assess SWI against the commonly used weight initialization method in dynamic sparse training, Kaiming. The results indicate that SWI facilitates faster training from the initial phases.
Early stop. The third panel of Figure g presents evidence that early stop can prevent the over- evolution of the network topology. Once the network structure becomes stable, running ESML is unnecessary, and it is more optimal to conduct an early stop and focus on refining the weights to achieve convergence. As an example. at the 12th epoch, the overlap rate (the overlap between re- moved links and new links) exceeds 90%, which can be seen as a topological stability threshold, indicating that the network should no longer evolve. Based on this observation, we implement an early stop for the evolutionary process. With early stop, we not only achieve performance improve- ments but also accelerate training speed since there's no need for further topological exploration.
L AREA ACROSS THE EPOCHS
The Area Across the Epochs (AAE) is computed as the cumulative sum of the performance indicator of an algorithm till certain epoch is divided by the number of epochs. Therefore, AAE is the average performance that an algorithm displays till certain epoch during training, it is bounded [0,1] and it is an indicator of the learning speed of algorithm. This means that algorithms whose training curves grow faster have larger AAE and can be considered: "faster learners". ". For instance, considering the same algorithm and fixing the same number of epochs for each trial, AAE can be used to measure and compare the impact that the tuning of a hyperparameter has on learning speed and to quantify the extent which this hyperparameter governs learning speed.
f dm
AAE(A)
(12)
E-1
Where A denotes the algorithms that need to be evaluated, E is the number of epochs. m denotes the accuracy at each epoch.
Pay attention, when we come to comparing different algorithms, there are two important remarks about the way to apply AAE for a fair comparison of learning speed.
First, AAE should be applied considering for all different algorithms the same number of epochs and the same hyperparameter settings that can influence learning speed. Suppose an algorithm stops at a number of epochs that is smaller than others. In that case, all the algorithms' AAEs will be computed till this smallest epoch value to make sure that the algorithms are fairly evaluated considering the same interval of epochs. Suppose two algorithms are run using different update intervals of epochs during training. In that case, the comparison is not fair, and both algorithms should be run using the same update intervals. This is valid for any setting that influence the learning speed. All
23

---

Published as a conference paper at ICLR 2024
learning speed settings should be fairly fixed to the same value across methods to ensure that the learning speed difference between algorithms is intrinsic to the sparse training mechanisms and is not biased by confounding factors.
Second, AAE can only be used to compare the learning speed of algorithms inside the same com- putational experiment and not across different datasets and network architectures because different datasets or network architectures can generate different learning speeds. However, if the mission of the comparison is to investigate the different learning speeds of the same algorithm with the same hyperparameter settings and network structure but using different datasets, then using AAE is still valid and can help determine how the same algorithm might react differently in learning speed be- cause of some variations in the data. An interesting test to try is, for instance, investigating the impact of different increasing noise levels on the same type of data considering the same algorithm and setting. This would help answer the question of how noise impacts an algorithm's 's learning speed.
M SELECTING NETWORK INITIALIZATION METHODS: A NETWORK SCIENCE PERSPECTIVE
In our study, we initialize our network employing Correlated Sparse Topological Initialization (CSTI) on layers that directly interact with input features from the dataset and apply Erdös-Rényi (ER) to intermediate layers. Additionally, to facilitate a thorough comparison, we also introduce two well-known network models: the Watts Stroggat (WS) model and the Barabási-Albert (BA) model.
The Watts Strogat model (Watts & Strogatz 1998) is well-known for its small-worldd' proper- ties, characterized by high clustering and short path lengths. This model displays transitions from a regular high-clustered lattice (3 = 0) to a random small-world graph (3 1). Intermediate values of 3 can generate networks that retain clustering while displaying small-world connectiv- ity, striking a balance between regularity and randomness. On the other hand, the Barabási-Albert model (Barabási & Albert 1999) is noted for its scale-free nature, featuring power-law degree distribution. This means a few nodes (hubs) have a very high degree, while most nodes have a low degree, mimicking the structure of many real-world networks. A significant limitation of both the WS and BA models is their lack of a bipartite structure, which is crucial in many real-world appli- cations. To address this, we provide detailed method for initializing networks using the WS and BA models in a bipartite network context. This involves adapting these models to suit the bipartite framework, thus expanding their applicability to more diverse and complex network scenarios.
Modelling Barabási-Albes Bipartite Networks. 1. Generate Barabási-Albert (BA) monopar- tite model consisting of n + m nodes, where n represents the size of the first layer and m the size of the second layer. 2. Randomly select n and nodes in the network and assign them to their respective layers. 3. Identify and count the 'frustrations" in both layers. A frustration in layer (FI) is defined as a link between nodes within the same layer. Similarly, count the frustrations in the second layer, F2. If the number of frustrations in Fl equals that in F2 (#F1 = #2)), proceed to match each frustration in Fl with one in F2 randomly. For each matched pair, apply a single rewiring step according to the Maslov-Sneppar adjust the process so that # F1 equals the size of a randomly sampled subset of F2 (F2s). For the remaining frustrations in F2 (F2r), where # F2r # F2 #2 F1, sequentially break each frustration in the layer and rewire the links of each of the two nodes to the opposite layer using the preferential attachment formula from step 1. If # F11 > #F2, apply similar procedure but adjusted to this specific case. The resulting BA network from this algorithm will be bipartite and exhibit a power- law distribution with a y value of 2.76. This is because the steps in this algorithm retain the same distributions as the original monopartite BA model generated in step 1.
randomization (MS) algorithm. If #11 < #F2,
Modelling Watts-Strogatz Bipartite Networks. 1) Construct regular ring lattice with N #L1 + # L2 nodes, where L1 and L2 represent the nodes in the first and second layers, respec- tively. Label the N nodes proportionally such that if L1 # L2, then L2 nodes placed every round Hund(#LL, LL1## #L2) nodes. For example, if # L1 = 100 and # = 10, then the L2 nodes will be positioned in the ring every 10 7.1 nodes. 2) For every L1 node, establish connections only with the
24

---

Published as a conference paper at ICLR 2024
K/2 nearest L2 neighbors in the regular ring lattice. 3) For every node .N...... - take every edge connecting to its K/2 rightmost neighbors, that is every edge (i.j) such that 0 < (j - i) mod N < K/2, and rewire it with probability 3. Rewiring is done by replacing (i.j) with (i, k), where k is chosen uniformly at random from all possible nodes while avoiding self-loops (k: o i) and link duplication (there is edge (i, k') with k' k at this point in the algorithm). Note that when b the generated WS network is equal to ER.
The Dynamic Version of BA and WS Network. Following the same network evolutionary prune- and-regrow regime, introduce the different link regrown methods based on the different network properties BA and WS.
1) BA network: After removing certain number of links, shooting from random nodes the same number of links using preferential attachment probability. Note that Feng et al. (022) also adopt the preference attachment to regrow new links in the continual learning field.
2) WS network: After removing a certain number of links, shooting from random nodes the same number of links using uniformly at random probability.
This means that the BA network will always keep a scale-free network while the node degree distri- bution in the WS network more uniform all the time.
Keeping The Sparsity Fixed For All The Methods. Since the initialization of BA and WS rely on the average degree which may not be integer. Therefore, the sparsity of the generated network might not be the same as we preset. We propose the below strategy to solve this problem. Given a certain m, approximate its value to the first decimal, then take the first decimal of m and call it FD. Then, a sequence of 10 nodes will be created, for which 10-FD nodes have degree [m] and FD nodes have degree [m] +1.
N SUBRANKING STRATEGY OF CH3-L3
Here, we describe the sub-ranking strategy adopted by rank all the node pairs that have the same CH score. Although we didn't apply this strategy in the current article due the ultra-sparse scenario, it can be useful for lower-sparsiyy DST applications in the future. The sub-ranking strategy aims to provide a more precise ranking for links with identical link prediction scores, particularly those that are tied-ranked and located just above the regrown threshold links.
et al. 2020) to internally
To implement the sub-ranking strategy, the following algorithmic steps are followed:
Assign weight to each link (i.j) in the network, denoted as wx, = 1+CH
Compute the shortest paths (SP) between all pairs of nodes in the weighted network.
For each node pair (i,j), compute the Spearman's is rank correlation (SPcorr) between the vectors of all shortest paths from node i and node j.
Generate final ranking where node pairs initially ranked based the CH score (CH,,)), and ties are sub-ranked based on the SPoorr,,j value. Random selection used to rank the node pairs if there are still tied scores.
While the SPoorr can be replaced with any other link predictor, we adopted this strategy because it aligns with the neurobiological principles underlying the CH model(Muscoloom et al.) (020). Ac- cording to the interpretation fluenced by the spatial proximity of their respective axonal and dendritic arbors(Rees et al. 2017) In other words, connectivity depends on the geometric proximity of neurons. This neurobiological concept is consistent with the SPcorr score. It fits within the framework of CH modeling, high correlation suggests that two nodes have similar shortest paths to other nodes in the network, indicating spatial proximity due their close geometric positioning.
Peters' rule, the probability of two neurons being connected is in-
25

---

Published as a conference paper at ICLR 2024
Table 3: The results of sparse training on MLP models. We explored different initialization methods CHT among Correlated Sparse Topological Initialziaiton (CSTI), Barabási- Albert (BA) network, and Watts- Stroogtt (WS) network with 8 = 0, 0.25, 0.5, 0.75, 1.0 (Note that when & = 1. WS network is equivalent to Erdós- Rényi model). We also compare CHT with the fixed BA network and WS network with different 3 and their dynamic version. In the meantime, we compare method with the SOTA dynamic sparse training methods (RigI., MEST) and the dense network. The best performance among the sparse training methods highlighted in bold.
MNIST
Fashion MNIST
EMNIST
S:60K
C:10
S:60K C:10
S:131K C:47
ACC
AAE
ACC
AAE
ACC
AAE
FC
08.69=00002
97.0120.12
90.43=0.09
88.5420.02
85.58:0.006
83.16=0.099
RigLix
97.40.0.077
94.7110.07
88.0210.10
86.1310.12
82.96.0.04
79.88
MEST
97.31++0.05
94.41+0.055
88.13+0.10
86.02+40007
83.05+0.04
80.02-0.07
CHT (CSTI)
98.05:10.04
96.9:10.66
88.07=0.11
86.12:00.55
83.82-1..44
81.05=0017
CHT
96.36.1.04
95.79110.03
87.11110.06
86.28.10.11
80.510.11
79.1411.007
CHT xWSS,
96.08=0.05
95.56+0.06
88.27=0.14
87.29:-0111
78.8=0.16
77.74=0.07
CHT (WS, 0.25)
97.18110003
96.59110055
88.5110.22
87.38:10.06
81.1210.04
80.09-0.04
CH 8 0.5)
97.25=:0.2
96.6:t0.03
88.1920.04
87.14::0.06
81.33+0.066
80.15=0.05
CHT (WWS, B 0.75)
97.06.10.05
96.49
80.84
86.95.10.06
80.97.10.07
79.77.00005
CHT (WWS, / 1)
96.92+0.02
96.37=0.03
87.72+0.12
86.83++0.12
81.03+0.113
79.71=0.15
BA1x static
06.1910009
94.310.16
87.04=0.007
85.01+0.122
80,93-10008
75.75
BA dynamic
97.11
95.47-10.09
87.98
85.78:10.09
82.610008
78. 0.00022
WS ix static(B 0)
95.4111.011
92.0110.110
86.7410.03
81.7110.11
77.9110.07
71.06.1021
WS static( 0.25)
96.4+0.03
92.6+0.10
87.27+0.12
83++0.06
81.3+0.07
73.18+0.11
WS static[8 U.5)
96.33.0.04
92.13:20.09
87.18=0.07
82.49:0.01
81.37
72.45=0.03
WS static(3 0.75)
%..0710.088
91.55.10.07
86.78
82.410006
81.39
72.00
WS static(3 1)
1..18.0.044
91.43=+0.08
86,59=0.02
82.53+0.07
81.1+0.11
71.75=0.12
WS dynamic($ = ()
97.0410.04
94.15-10.14
88.09
84.320.10
82.7910.09
75.89-0.11
WS dynamic(3 = 0.25)
96.95=00.05
93.87=20.06
87.85=0.114
84.420.04
82.42=0.008
75.1
WS dynamic(3 (3- 0.5)
96.8..00.55
93.02
10.09
83.87.10.04
80.45
74.28 10.12
WS dynamic(3 0.75)
96.44+0.66
92.74+0.12
87.57+0.07
83.7240.12
82.55+0.06
74.11=0.03
WSIx dynamic(3
96.81110.44
92.56.10007
87.4620.17
83.9110.08
82.52-1.007
73.92-0008
FC2x
98.7310.02
97.1410.033
10.74
58.37
15.85
84.1710.11
RigLax
97.91
98.1710.03
88 68.6610.007
86.1410.07
83.44
10.41
MEST
97.66++0.03
95.63+0.09
88.33+0.10
85.01+00077
83.50+00.99
80.77=0.03
(CST)
98.3410.08
97.51=0.07
88.6610.07
87.63.0.119
85.43110.10
83.91
cHT,.(BA)
97.08110.05
96.67-10.03
86.8+10.05
80.38
79.46:100
SHT,,.WS 8= 0)
96.3110.06
95.89
18.
87.83.10.07
79.15.10.03
78.2810.08
CHT,; $ 0.25)
97.64+0.02
97.29+00.06
88.57+0.08
88.21+0.13
81.91+0.01
81.02+0.01
CHT 8- 0.5)
97.75+10.04
97.36+10.04
88.98-0.07
87.73+-0.08
81.77+0.09
80.77
CHT,x B= 0.75)
97.47.0.04
97.0110.05
88.61 10009
87.42.10.04
81.13:10.07
10
(WSS, =)
97.39+0.05
97.03-+0.03
18.17-0.11
87.4+0.06
81=0.18
80,22-10033
BAax static
96.7910.05
95.67-10.04
80.69
86.1910.05
82.02-10.20
79.18-10.2
BA2x dynamic
97.49=0.02
96.6720.06
88.320.13
86.720014
83.18:10.04
80.5+0.26
WS static(3
9..03.0.10
93.4110.144
87.57
83.72.10.28
78.72.10.15
10.29
WS static( = 0.25)
96.5++0.10
93.98++0.06
88.26+0.12
84.15+0.22
82.42+0.09
76.03-00088
WS.. static(3 -U.5)
97.2710.07
93.48-10.09
88.26-00.08
83.8710.06
82.6610.08
70.38
WS2x static(8 = -0.75)
96.77+0.07
92.85
88.13
83.47+0.110
82.39+0.07
74.64
WS2x static(B 1)
96.67.10.06
92.52.10.09
87.68 10.04
83.46.10.07
82.2910.188
74.41
WS2x
dynamic(/3 0)
97.57+0.06
95.46+0.19
88.42+0.19
85.39+0.25
83.41+0.06
78.6+0.20
WS.x
dynamic(3 0.25)
97.3610.02
94.72-10.06
88.4920.17
85.35:1..14
83.4110.05
77.46:20.08
WS2x dynamic(3
97.15.10.03
93.82.10.11
10.08
84.71.0.05
83.0910.01
76.26
WS2x
dynamic( =0.75)
97.17+0.06
93.77=0.19
88.14=0.06
84.56+00.99
82.94+0.04
75.94=0.17
WS2x
dynamic(3
97.1411.01
93.5410.005
88.1110.07
84.46.1.111
83.1410.07
75.5510.06
26

---

Published as a conference paper at ICLR 2024
Table 4: The results of sparse training on CNN models. We explored different initialization methods of CHT among Barabási-Albert (BA) network, and Watts- Strogatz (WS) network with 3 0, 0.25, 0.5, 0.75, 1.0 (Note that when B = 1. WS network is equivalent to Erdós-Rényi model). We also compare CHT with the fixed BA network and WS network with different 3 and their dynamic version. In the meantime, we compare our method with the SOTA dynamic sparse training methods (RigI, MEST) and the dense network. The best performance among the sparse training methods is highlighted in bold, and the bolded values marked with "4" indicate that they surpass those the fully connected counterparts.
SUIRK C:380
NIICK
KAIM CHEER
NINKK CHIB
TC,,
51342611
13.55.0.4 54
75:17.0.13
7531035
C5.55220
NaSI
31.47511
413055330
18:233000
51222000
45442033 32
51352408
4340..0.3
3172.4015
1104000
40002108
BA. Ornnns
0121-005
133560
1177-0111
11256005
0-9-00
5194
154600
1300 16
0355 C3
0115-005
533550
3022102
443181
Rigla,
513021011
31200400
48.52044
21784600
61364055 57
10000m
7036-001
114000
1100-00
1155-555
13
25-00
905-011
100000
51N-3
Mal-nol
eatic 0,20
51.412000
015265 50
54 54.505 8
51213000
7..0.000
03361054
5.210030
(513:1:3
S00.011
(285:134
Obeand
2123.001
3001000
WS. 0.251
59.56037 17
1.341559
(455:15
305.001
(248:339
4147400
3.0-0u5
WS.
115540332
13
11
12:04433
S.16.0m
5135-139
315-000
WS. Gbeade 1)
41346606
900
-00
07 0.90-:3
5006600
6114-3
9.0-011
6109-117
O CREDIT ASSIGNED PATH
The epitopological learning implemented via CH3-L3 link prediction produces topology that en- courages new inter-layer links between pair of non-interacting nodes that are topologically close to each other because they share many externally isolated (minimum number of external links with respect to the local community) common neighbors on paths of length 3 (see Figure 2 This means that during epitopological-bsse based dynamic sparse training, the new predicted lnter-layer interactions will connect with continuous path nodes with increasing higher degrees across layers, creating meta-deep structure inside each layer that connects across the layers and that is composed of the nodes that are at the center of the hyperbolic disk because they have a higher degree and therefore they are more central in the hyperbolic geometry. Hence, cpitopological sparse meta-deep lcarning percolates the network, triggering hyperbolic geometry with community and hierarchical depth, which is fundamental to increase the number of credit assigned paths (CAP, which is the chain of the transformation from input to output) in the final trained structure. The total number of CAPs in the ESML network is 9,095,753, which is much larger (around 10 times) than the CAPs of SET-random, which is 782.032. Even more interestingly. we introduce new measure for the evaluation of sparse network topology arrangement: the ratio of CAPs that pass by the hub nodes in intermediate layers. In ESML network of Figure A the hubs are the 80 nodes the center of hyperbolic disk in layer 2 and 3) and the hub-CAPs ratio is 35514600 1551446099555--00.3
whereas when taking the top 80 nodes with higher degree in the layer 2 and of SET-random network of Figure A the hub-CAPs ratio This explains how crucial epitopological learning via CH3- L3 link prediction to trigger a layer hierarchical community organized hyperbolic structure at the center of which emerges a cohort of hubs that create frame that connects across the layers of the network, forming a scale-fre free topology with plwer-law distribution 2=2.03. The meta-deep struc- ture of these hubs across the layers, which is associated with the hierarchy (because nodes in the center of the hyperbolic representation) in hyperbolic geometry and to the power-law distribution triggers ultra-small-woor worldd network with higher navigability (Cannistraci & Muscoloni 2022).
passing by them is 2596/782032-0.000
27

---

Published as a conference paper at ICLR 2024
Table 5: The detailed information of warmup and learning rate decay strategy applied in this article.
Model
Epochss
Starting Learning rale
Milestones
Warm-up
Warm-up Egoch
VGG16
Tiny Mny--engeNNett
2001
0.1
61,121,161
0.2
256
No
GoogLeNNet
CIFAR100
60,120,160
GoogLeNet
Tin--IageeNer
200
0.1
61,121,161
0.2
256
No
ResNer50
CIFARI00
60,120,160
ResNet152
CIFARI00
200
0,1
60.120.160
0.2
128
Yos
ResNet50
ImageNet
Method
Connection Removal
Connection Regrown
Gradlent Backpropagation
Fixed Sparsity
Welght Initialization
Weight Update
Topological Initialization
Topological Early Stop
SET, DSR
Weight Magnitude
Random
Sparse
Kaiming
zero
ER
X
RigL
Weight Magnitude
Gradient
Dense
Kaiming
zero
ER, ERK
X
MEST
Weight
Sparse
Kaiming
zero
X
Gradient
Magnitude
GraNet
Weight Magnitude
Gradient
Sparse
X
Kalmming
zero
Dense
X
OptG
Trainable super mask
Dense
Kaimingg
ER
X
CHT (Ours)
Weight
ESML
Sparse
SWI
SWI
STI
Magnitude
Figure 9: Comparing various components of different pruning and sparse training algorithms. The innovations
in this article are highlighted in bold. Innovations introduced in this study, including 'ESML" (Epitopological Sparse Meta-deep Learning), 'SWI' (Sparse Weight Initialization), and 'STT' (Sparse Topological Initializa- tion, highlighted in bold. We also introduce an early stop mechanism for topological exploration once the network topology stabilizes.
P HYPERPARAMETER SETTING AND IMPLEMENTATION DETAILS
To improve the reproducibility of this paper, we have detailed the hyperparameter setting for each section here.
P.I HYPERPARAMETER SETTING
For the overall hyperparameter setting, test), the fraction of removed links zeta=0.3, weight decay=1e-05, batch size=32, and momentum =0.9 for all the MLP and VGG16 tasks. For the learning rate, we use fixed rate for all tasks, and to highlight the distinction between ESML and SET, we set a lower learning rate of 0.0001 Section 4.1. The detailed hyperparameter settings for comparing CHT and the other algorithms are provided in Table 5) All the experiments in this article involve image augmentation, which includes random cropping and horizontal flipping, followed by a standard normalization process. The models VGGI6, GoogLeNet, ResNet50, and 5esNer152 adhere to their default hyperparameter configurations, with the sole modification being the expansion the fully connected section model's end to encompass 4 layers. The hidden dimension is set to either 1 X 2x of the input dimension.
default sparsity=0.99 (except for the sensitivity
P.2 ALGORITHM TABLES OF CHT
Please refer the Algorithm
28

---

Published as a conference paper at ICLR 2024
Initial node Size Activate Neuron Post-percolation
Reel running time
8 Das 24
Vi Noce Stee Fryoleed Sparse Tra Fang
Dacaseta-Modor
Figure 10: Active Neuron Post-percolation rate and real running time of the dynamic sparse training methods. A shows the incremental reduction in Active Neuron Post-percon pestooeccoltiion corresponds to the initial node size. For datasets: 'M', 'F', and 'E' stand for MNIST, MaSIi and EMNIST, respectively; 'C10' and 'C100' denote "CIFARIO" and 'CIFARI00'; while 'I' signifies ImageNer2012. Regarding the models, 'M', 'V', and 'R' correspond to MLP, VGG16, and ResNet50, respectively. B shows the real running time (log 10) of each 'Dataset-Model" task. This running time is computed from starting training model convergence of x case.
Algorithm Algorihtm CHT.
Input: Network fe. dataset D, Update Interval At, Early Stop Signal Learning rate a, Overlap Threshold a.
T Sparse Topological Initialization,
Sparse Weight Initialization,
+ False.
for each training step do Sample a batch 1, ~ DD - [(f(x; w, T).3).
w w a WWC
i mod Al = 0 and !A then allLayersPassed + True for each layer do
w', T(+ FSSM(ww. TD)
allLayersPassed allLayersPassed A (OverlapRate(T42..)) > o)
end for if allLayersPassed then
A -- True end if end if
end for
29


Reviews:
[{"review_id": "mIO1F88lI1", "reviewer": "Reviewer_M5bF", "summary": "The authors propose a new form of sparse network learning, which was brain-inspired. They compare to other methods in the field and evaluate on vision dataset.", "questions": "No questions", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: I am not very familiar with the literature in this community, but from what I can tell the authors cite existing literature well and distinguish their approach adequately from other methods in the field.\nQuality: the paper is well written, and the empirical results are well done: they are thorough, include meaningful baselines, and cover a nice variety of datasets.\nClarity: the paper was well written\nSignificance: I am not familiar enough with this sub-field to judge, but given the result, this appears to be a solid step in the right direction, and is likely relevant to the subcommunity.", "weaknesses": "While I understand the space constraints, having something like Table1 or Table2 in the main text (maybe in reduced form) would be nice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29lNDcnJz5", "reviewer": "Reviewer_cMYk", "summary": "This paper is about a new training methodology for deep learning called Cannistraci-Hebb training (CHT). CHT is a training methodology that uses epitopological sparse meta-deep learning (ESML) to learn artificial neural networks (ANNs) with ultra-sparse hyperbolic topology.", "questions": "1. What's the inspiration of current architecture? \n2. Did you do extra experience on other graph structure? \n3. In Table 2 on page15. Why some accuracy result is not the best but still highlighted?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and all the methodological details are described clearly.\n2. CHT is a new 4-step training methodology for deep learning that uses ESML to learn ANNs with ultra-sparse hyperbolic topology.\n3. CHT has been shown to surpass fully connected networks on VGG16 and ResNet50.\n4. CHT could be used in a wide range of real-world applications, such as image classification, natural language processing, speech recognition, recommender systems, and medical diagnosis.", "weaknesses": "1. For the node structure, this paper only compares CHT with fully connected graph, the author should take other graph structure into consideration, like ER, BA, WS graph. \n2. Compared with SOTA models, the performance of CHT is not competitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhWEJb7cXP", "reviewer": "Reviewer_nVAm", "summary": "The paper introduces sparse training in deep learning, aiming to replace fully connected neural networks with ultra-sparse ones inspired by brain networks. They propose Epitopological Sparse Meta-deep Learning (ESML) using Cannistraci-Hebb learning theory. ESML learns ultra-sparse hyperbolic topologies, showcasing meta-deep organization. Empirical experiments reveal ESML's ability to automatically sparse neurons through percolation. Cannistraci-Hebb training (CHT) is introduced and compared with fully connected networks on VGG16 and ResNet50. CHT acts as a gradient-free oracle, guiding link placement for sparse-weight gradient learning. It introduces parsimony dynamic sparse training by retaining performance through percolation and reducing node network size.", "questions": "The proposed method appears effective in scenarios where neural networks are overparameterized. Does it maintain its efficacy when applied to non-overparameterized neural networks characterized by limited depth and width? Particularly, how does it perform in relation to a large dataset?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is praised for its density and detail.\n- The methods employed in the paper bring unique perspectives to sparse training.\n- The paper is commended for conducting various experiments, with promising results.\n- The inclusion of detailed experiment descriptions is noted as a positive aspect, providing transparency and allowing readers to understand the methodology.\n- The inclusion of visual illustrations is a positive aspect, aiding in the comprehension of dense presentation.", "weaknesses": "- Simplifying the language could enhance accessibility to a wider audience.\n- Some results lack standard deviation reporting, raising questions about the reliability and robustness of those specific findings.\n- Addressing the legibility issue by enlarging text within figures would enhance the overall effectiveness of these visuals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)