Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
Authors: Yingtao Zhang
Diego Cerretti
Jialin Zhao
Wenjing Wu
Ziheng Liao
Umberto Michieli
Carlo Vittorio Cannistraci
Venue: NeurIPS 2025 poster ()
Keywords: dynamic sparse training
network science
epitopological Learning
efficient training
Primary area: deep_learning
TL;DR: The proposed brain-inspired CHT Soft Rule with Sigmoid Decay Density (CHTss) achieves comparable even better performance compared to fully connected models across various tasks, enabling high sparsity in Transformers and LLMs.

Abstract: This study aims to enlarge our current knowledge on the application of brain-inspired network science principles for training artificial neural networks (ANNs) with sparse connectivity. Dynamic sparse training (DST) emulates the synaptic turnover of real brain networks, reducing the computational demands of training and inference in ANNs. However, existing DST methods face difficulties in maintaining peak performance at high connectivity sparsity levels. The Cannistraci-Hebb training (CHT) is a brain-inspired method that is used in DST for growing synaptic connectivity in sparse neural networks. CHT leverages a gradient-free, topology-driven link regrowth mechanism, which has been shown to achieve ultra-sparse (1\% connectivity or lower) advantage across various tasks compared to fully connected networks. Yet, CHT suffers two main drawbacks: (i) its time complexity is $\mathcal{O}(N\cdot d^3)$- N node network size, d node degree - hence it can be efficiently applied only to ultra-sparse networks. (ii) it rigidly selects top link prediction scores, which is inappropriate for the early training epochs, when the network topology presents many unreliable connections. Here, we design the first brain-inspired network model - termed bipartite receptive field (BRF) - to initialize the connectivity of sparse artificial neural networks. Then, we propose a matrix multiplication GPU-friendly approximation of the CH link predictor, which reduces the computational complexity to $\mathcal{O}(N^3)$, enabling a fast implementation of link prediction in large-scale models. Moreover, we introduce the Cannistraci-Hebb training soft rule (CHTs), which adopts a flexible strategy for sampling connections in both link removal and regrowth, balancing the exploration and exploitation of network topology. Additionally, we propose a sigmoid-based gradual density decay strategy, leading to an advanced framework referred to as CHTss. Empirical results show that BRF offers performance advantages over previous network science models. Using 1\% of connections, CHTs outperforms fully connected networks in MLP architectures on visual classification tasks, compressing some networks to less than 30\% of the nodes. Using 5\% of the connections, CHTss outperforms fully connected networks in two Transformer-based machine translation tasks. Finally, with only 30\% of the connections, both CHTs and CHTss achieve superior performance over other dynamic sparse training methods, and perform on par with—or even surpass—their fully connected counterparts in language modeling across various sparsity levels within the LLaMA model family. The code is available at: https://github.com/biomedical-cybernetics/Cannistraci-Hebb-Training-Soft-Rule-.

Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
Yingtao Zhang Diego Cerretti Jialin Zh Ziheng Liao Miaaol..Wenii Wenjing Wu
1,2 Umberto Michieli4.5 & Carlo Vittorio Cannistracf1.2.5
Center for Complex Network Intelligence (CCNI))
Dept. of Computer Science & Technology. "School of Biomedical Engineering, Tsinghua University Univeersity of Padova, CCanaa Research
Abstract
This study aims to enlarge our current knowledge on the application of brain- inspired network science principles for training artificial neural networks (ANNs) with sparse connectivity. Dynamic sparse training (DST) emulates the synaptic turnover of real brain networks, reducing the computational demands of train- ing and inference in ANNs. However, existing DST methods face difficulties in maintaining peak performance at high connectivity sparsity levels. The Cannistraci- Hebb training (CHT) is a brain-inspiree method that is used in DST for growing synaptic connectivity in sparse neural networks. CHT leverages a gradient-fres topology-drive link regrowth mechanism, which has been shown to achieve ultra- sparse (1% connectivity or lower) advantage across various tasks compared to fully connected networks. Yet, CHT suffers two main drawbacks: (i) its time complexity is O(N . dd N node network size, d node degree hence it can be efficiently applied only ultra-sparse networks. (ii) it rigidly selects top link prediction scores, which is inappropriate for the early training epochs, when the network topology presents many unreliable connections. Here, we design the first brain-insping network model termed bipartite receptive field (BRF) to initialize the connectivity of sparse artificial neural networks. Then, we propose matrix multiplication GPU-friendly approximation of the CH link predictor, which reduces the computational complexity to ((*),, enabling fast implementation of link prediction in saage-scale models. Moreover, we introduce the Cannistraci- Hebb training soft rule (CHTs), which adopts a flexible strategy for sampling connections in both link removal and regrowth, balancing the exploration and exploitation network topology. Additionally, we propose a sigmoid- based gradual density decay strategy, leading to advanced framework referred to as CHTss. Empiri- cal results show that BRF offers performance advantages over previous network science models. Using 1% of connections, CHTs outperforms fully connected networks in MLP architectures on visual classification tasks, compressing some networks to less than 30% of the nodes. Using 5% of the connections, CHTss outperforms fully connected networks in two Transformer- based machine trans- lation tasks. Finally, with only 30% of the connections, both CHTs and CHTss achieve superior performance over other dynamic sparse training methods, and perform on par with even surpass-their fully connected counterparts in lan- guage modeling across various sparsity levels within the L.aMMA model family.
Corresponding author: kal kalokagathos. saatshoss..gnngeee agon@gmail. com Research Center in Tsinghua Laboratory of Brain and Intelligence (THBI), Department of Psychological and Cognitive Sciences.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The code is available at: http:://gghh...ccoooooooddddddee Cannistraci-Hebb - Soft-Rule-
1 Introduction
Artificial neural networks (ANNs) have led to significant advances in various fields such as natural language processing, computer vision, and deep reinforcement learning. The most common ANNs consist of several fully connected (FC) layers, which account for a large portion of the total parameters in recent large language models M 2. This dense connectivity poses major challenges during the model training and deployment phases, In contrast, neural networks in the brain inherently exhibit sparse connectivity B 41. This natural design in the brain exploits sparsity, suggesting model in which the number of connections does not scale quadratically with the number of neurons. This could alleviate computational constraints, enabling more scalable network architectures.
Dynamic sparse training (DST) [ 6 7 g 9 has emerged as a promising approach to reduce computational and memory overhead of training deep neural networks while maintaining or even improving model performance. DST is also biologically inspired: it draws an analogy to synaptic turnover [10] in the brain, fundamental neurobiological process in which synapses continuously formed, strengthened, weakened, and eliminated over time. This dynamic rewiring enables the brain to adapt, learn. and store memories efficiently while preserving the overall stability of the network. Similarly, in DST, connections are dynamically pruned and regrown throughout training, allowing the network to adapt its connectivity structure in response learning signals while maintaining a fixed sparsity level.
Apart from some detailed distinctions, the primary innovation in this field centers the development of the regrowth criterion. A notable advancement is the gradient-free regrowth method introduced by Cannistraci- training (CHT) 9.. This method is inspired by epitopological learning literally
meaning 'new topology' -and -and is rooted in brain-inspired network science theory [1] 13 14] 15]
Epitopological learning explores how learning can be implemented on complex networks by changing the shape of their connectivity structure (epitopological plasticity). CHT has demonstrated remarkable advantages in training ultra-sparse ANNs with connectivity levels of 1% or lower, often outperforming fully connected networks in various tasks. However, despite these advances, CHT encounters two major challenges: 1) During dynamic sparse training, its rigid link selection mechanism can lead to epitopological local minima where the sets of removed links and regrown links exhibit significant overlap, severely impedes the exploration of optimal network topologies. 2) The time complexity of the CHT regrowth method, Cannistraci- -ebb 3 on Length 3 paths (CH3-L3p)) is O dd), where N represents the number of nodes in the network and dl is the average degree. A length 3 path is a walk of three consecutive links. As the network becomes denser, the time complexity approaches ONN)), rendering it impractical for large-scale and higher-density models.
In this article, we present the Cannistraci-Hobl Training soft rule (CHTs), which introduces several key innovations: To address the issue of epitopological local minima, CHTs employs multinomial distribution to sample link scores from removal and regrowth metrics, enabling more flexible and effective exploration of network topologies. 2) CHT's incorporates novel substitution nose- link prediction mechanisms, reducing the computational time complexity to ((33). This improvement makes CHTs scalable to large-scale models with higher network density. 3) CHTs initializes the sparse topologies for bipartite networks with the brain-like receptive field, demonstrating superior performance compared to the traditional Erdös-Rényi (5), bipartite small world, and bipartite scale- free model [9]. Additionally, we propose sigmoid gradual density decay strategy, which, when integrated with CHTs, forms an enhanced framework termed CHTss. This combined approach further optimizes the training process for sparse neural networks.
To evaluate the effectiveness of the Cannistraci- Hebb Training soft rule with sigmoid density decay (CHTss)), we conduct extensive experiments across multiple architectures and tasks. Firstly, to assess the basic concept of CHTs, we employ MLPs on benchmark datasets, including MNIST [16), EMNIST [7], and Fashion MNIST [18]. The results show that CHT's performs better than fully connected networks with only 1% of the connections (99% sparsity) in MLPs. Further, to evaluate the end-to-end approach CHTss, we utilize Transformers 19 on machine translation datasets such as Multi30k en-de 20), IWSLT14 en-de (T), and WMT17 en-de 22]. From the experimental results, CHTss surpasses fully connected Transformers while using only 5% of the connections
2

---

Dirnaioo
Iel
Ma
X
Netwoek Intieliaution
Link Remeval
Nenwork Percslation
CHIZ4L1 Salt Hegrowth
Finchh
Trikigg
burdey Nexso Beoane
Sadvica
faroeh webher
Fd
Two Booeds
960
Saldes Kloms 183
[2 10-0--0--1 WL.
Layed
Medtioomiat...p
Peoveial
Seawek Stucd
Bormnd D'ermata
Ircwedee Fabh Mesrcc
Figure I: Illustaaioon of the CHTs process. One training iteration follows the steps of (al) - (bl) + (cl) - (c2) + (d1) (e). (al) Network initialization with each of the sandwich layers (bipartite networks connecting layers' input nodes to their output nodes) being a bipartite receptive field (BRF) network. (a2) BRF network representation with O. (b1) Link removal process. (b2) Formula for determining which links to remove. (c1) Removal of inactive neurons caused by link removal. (c2) Adjust and remove incomplete links caused by inactive neuron removal. (d1) Regrowth of links according to the CH2-L3 node-based soft rule. (d2) Detailed illustration of the CH2-L3 node-based soft rule. (e) Finished state of the network after iteration. The next iteration repeats the steps (b1) (e) from this finished state. À indicates the removal set of the iteration and A is the regrown set.
on Multi30k and IWSLT, and achieves performance comparable to fully connected LLaMA-60N and LLaMA-130M models Open WebText language modeling tasks. Moreover, with 30% of the connections, CHTs even outperforms the fully connected LLaMA-IB counterpart. These findings underscore the potential of CHTs and CHTss in enabling highly efficient and effective large-scale sparse neural network training.
Related Work
2.1 Dynamic sparse training
Dynamic sparse training is a subset of sparse training methodologies. Unlike static sparse training methods (also known pruning at initialization) [23 24 25 26). dynamic sparse training allows for the evolution of network topology during the training process. The pioneering method this field is Sparse Evolutionary Training (SET) (5), which removes links based the magnitude of their weights and regrows new links randomly. Subsequent developments have sought to refine and expand upon this concept of dynamic topological evolution. One such advancement was proposed by DeepR (27) a method that adjusts network connections based on stochastic gradient updates combined with fayesian-inspiree update rule. Another significant contribution is Rigl. 77 which leverages the gradient information of non-existing links to guide the regrowth of new connections during training. MEST utilizes both gradient and weight magnitude information selectively remove and randomly regrow new links, analogously to SET. In addition, it introduces an EM&S strategy that allows the model to train higher density and gradually converge to the target sparsity, The Top-K KAST TASTR [6] method maintains constant sparsity throughout training by selecting the top K parameters based on parameter magnitude at each training step and applying gradients a broader subset B. where B 3 A. To avoid settling on suboptimal sparse subset, TASTKAST also introduces an auxiliary exploration loss that encourages ongoing adaptation of the mask. Additionally, sRigL 28 adapts the principles of RigL to semi-structured sparsity, facilitating the training of vision models from scratch with actual speed-ups during training phases. Despite these advancements, the state- of-the-art method remains RigL-based, yet is not fully sparse in backpropagation, necessitating
3

---

the computation of gradients for non-existing non-existing links. Addressing this limitation, Zhang et al. [9] propose CHT, a dynamic sparse training methodology that adopts a gradient-free regrowth strategy that relies solely on topological information (network shape intelligence), achieving an ultra-sparse configuration that surpasses fully connected networks in some tasks. Some extra related works of DST are provided in Appendix M
2.2 Cannistraci-Helle Theory and Network Shape Intelligence
As the SOTA gradient-free link regrown method, CHT [9] originates from brain-inspired network science theory. Drawn from neurobiology, Hebbian learning was introduced in 1949 [29] and can be summarized in the axiom: "neurons that fire together wire together." This could be interpreted in two ways: changing the synaptic weights (weight plasticity) and changing the shape of synaptic connectivity [ 12
[5]. The latter is also called epitopological plasticity [ because plasticity means "to change shape," and epitopological means "via new topology." Epitopological Learning (EL)
12 B 14 is derived from this second interpretation of Hebbian learning and studies how to implement learning on networks by changing the shape of their connectivity structure. One way to implement EL via link prediction, which predicts the existence and likelihood of each nonobserved link in network. CH3-L3 is one of the best-performing and most robust network automata, belonging to the Cannistraci- Hebb (CH) theory [0], which can automatically evolve the network topology starting from a given structure. The rationale is that, in any complex network with cocal-community organization, the cohort of nodes tends to be co-activated (fire together) and to learn by forming connections between them (wire together) because they topologically isolated in the same local community [30] This minimization of the external links induces a topological isolation of the local community, which is equivalent to forming barrier around it. The external barrier is fundamental to maintaining and reinforcing the signaling in the local community, inducing the formation of new links that participate in epitopological learning and plasticity.
3
Cannistraci-Hebh training soft rule with sigmoid gradual density decay
3.1 Soft removal and regrowth.
Definition 1. Epitopological local minima. In the context of dynamic sparse training methods, we define an epitopological local minima (ELM) as a state where the sets of removed links and regrown links exhibit a significant overlap.
Let A, be the set of existing links in the network at the training step t. Let A, be the set of removal links and Aü be the set of regrown links. The overlap set between removed and regrown links at step t can be quantified as O = A n At. An ELM occurs if the size of O, at step t is significantly large compared to the size of A₁. indicating high probability of the same links being removed and regrown repeatedly throughout the subsequent training steps. This can be formally represented as
10.1 2 0, where 0 is predefined threshold close to 1, indicating strong overlap. This definition is [ATT
essential for the understanding of CHT, evidenced by the article (9] indicating that the overlap rate between removed and regrown links becomes significantly high within just a few epochs, leading to rapid topological convergence towards the ELM. Previously, CHT implemented topological early stop strategy to avoid predicting the same links iteratively. However, it will stop the topological exploration very fast and potentially trap the model within the ELM.
In this article, we adopt a probabilistic approach where the regrowth process is modeled as sampling from {0, 1} multinomial distribution, with probabilities determined by link prediction scores, thereby introducing a "soft sampling" mechanism. Under this formulation, each mask value is not rigidly dictated by the link prediction score; instead, low-score links can still be selected with lower probability, facilitating escape from epitopological local minima (ELM).
To demonstrate that soft sampling effectively balances exploration and exploitation, we evaluate its behavior in Figure 3 which presents the impact of varying softness levels during training of LLaMA-60M for 5000 steps under 90% sparsity. We compare the in-time over-parameterization (ITOP) rate (II, which quantifies the cumulative proportion of links activated throughout training. As shown, deterministic regrowth leads to rapid topological convergence after approximately 1000

---

steps, indicating that it becomes trapped in an ELM without further exploration. Random regrowth, while capable of escaping ELMs by introducing new connections, fails to exploit the underlying topological structure effectively. In contrast, soft regrowth achieves balance by both exploiting the current topology and exploring new link combinations probabilistically. This balance enables more appropriate exploration of the connectivity space, ultimately leading to superior performance, as evidenced by the results.
Link removal alternating from weight magnitude and relative importance. We illustrate the link removal part of CHTs in Figure [bb)) and b2). We employ two methods, Weight Magnitude (WM) [W] and Relative Importance (RI) [32. to remove the connections during dynamic sparse training. Given an input node i and an output node j connected with weight Wij. we define the relative importance Rīg as:
Wayl RI, Σ W E
W
(1)
As illustrated in Equation I1 RI assesses connections by normalizing the absolute weight of links that share the same input or output neurons. This method does not require calibration data and can perform comparably to the baseline post-training pruning methods like sparsegpt (33 and wanda (3 Generally, WM and RI are straightforward, effective, and quick to implement in DST for link removal, but give different directions for network percolation. WM prioritizes links with higher weight magnitudes, leading to rapid network percolation, whereas RI inherently values links connected to lower-degree nodes, thus maintaining higher active neuron post-percolation (ANP) rate. The ANP rate is the ratio of the number of active neurons after training over the original number of neurons before training. These methods are equally valid but cater to different scenarios. For instance, using RI significantly improves results on the Fashion MNIST dataset compared to WM, whereas WM performs better on the MNIST and EMNIST datasets.
Soft link removal. In the early stages of training, both WM and RI are not reliable due to the model's underdevelopment. Therefore, rather than strictly selecting top values based on WM and RI, we also sample links from multinomial distribution using an importance score calculated by the removal metrics. The final formula for link removal is defined Equation 2
1/2 Sij S a (1-a))) a (1 - a) ΣIW,,
////2
(2)
Here, a determines the removal strategy, shifting from weight magnitude (a = 1) to rela- tive importance (a 0). In all experiments, justs the softness of the sampling process. As training progresses and weights become more reliable, we adaptively increase from 0.5 0.75 to refine the sampling strategy and im- prove model performance. These settings are constant for all the experiments in this article.
only evaluate these two a values. 5 ad-
Node-based link regrowth. Another signif- icant challenge for CHT lies in the time com-
(6 Surning time with difeeent deratley
(3) Runsing time with diferenttt networt
200
plexity of link prediction. In the original CHT framework (9) the path-based CH3-L3p metric is employed for link regrowth, as discussed in Appendix However, this method incurs a high computational cost due to the need to compute and store all rungth-three paths, resulting in time complexity of O(N dd), where N is the number of nodes and d is the network's 's average degree. This complexity is prohibitive for large models with numerous nodes and digherydensity layers.
12
C
127
Deraty
N4twort Ate
Figure 2: One-time Link Prediction Runtime Performance Evaluation of node-based and path- based methods across varying densities and net- work sizes. In (a), the network size is fixed at 1024 x 1024, while in (b), the density is fixed at 5%.
To address this issue, we introduce a more ef- ficient, node-based paradigm that eliminates the reliance on length-three paths between seed nodes. Instead, this approach focuses the
5

---

common neighbors of seed nodes. The mode-based version of CH3-L3p, denoted as CH2-L3n, also depends on the internal local community links (iLCL, the links between the common neighbors (30) to enhance the expressiveness of the formula. This variant is formulated as:
CH2-L3n(4,0)=
deg
SEL3
Here, u and v denote the seed nodes, while z is the common neighbor on the L3 path 30 walk of three consecutive links that connects u to e via two of those intermediate nodes. The terms di; de represent the number of internal local community links (iLCL) and external local community links (eLCL) of node i, with a default increment of to prevent division by zero. The detailed explanation of iLCL and eLCL can be found in Appendix C The pseudocode is provided in the supplementary material. The new node-based variant, CH2--Lnn, reduces the computational complexity to O(N3) and enables efficient matrix-based computations GPUs. We evaluate the one-time link prediction runtime performance of both CH3-L3p and CH2-L3n across different network sizes and sparsity levels, illustrated in Figure 2 The red and green lines depict the actual runtime for the path- path-based and node-based methods, respectively. The results reveal that the node-based version achieves significantly faster runtime performance compared to the path-based methods. Furthermore, the node-based method demonstrates consistently stable runtime across diverse network sizes and density levels, making it an ideal link prediction as the CH theory-based link regrowth component of CHTs in dynamic sparse training for large-scale models.
3.2 Bipartite receptive field network modeling.
In this article, we propose the Bipartite Receptive Field (BRF) network model, a novel sparse topological initialization method capable of generating brain-network- receptive field connectivity. The principal topological initialization approaches for dynamic sparse training are grounded in network science theory, where three basic generative models for monopartite sparse artificial complex networks are the Erdös-Rényi (ER) model [35] the Wttts-Stroggtz (WS) model (36), and the Barabási-
Albert (BA) model (37). which are not brain-inspired Since the standard WS and BA models are not directly designed for bipartite networks, they were recently extended [9] into their bipartite counterparts and term as Bipartite Small- World (BSW) and Bipartite Scale-Free (BSF), respectively. BSW generally outperforms BSF for dynamic sparse training (see Appendix D)
During BSW initialization, each output node is connected to its spatially nearest input nodes. This spatially local connectivity pattern aligns with the concept of receptive fields observed in biological neural systems, where neurons respond selectively to localized regions of input space. However, the rewiring process of BSW does not follow brain mechanisms: it simply deletes closer input nodes to rewire them uniformly at random anywher on the input layer. Conversely, the random allocation of connectivity in brain network topologies is guided by the spatial distance of the neurons [38] 39]. Unlike the BSW model that introduces random connectivity by rewiring process, which cannot control the extent of spatial-dependen randomness injected in the topology. the BRF model directly generates connectivity with a customized level of spatial-dependent randomness using a parameter r E [o, 1]. low value of r results in links that are densely clustered around the diagonal, while higher value of leads to less clustered connectivity patterns, which tend to be uniformly at random only for 1. Specifically, a bipartite adjacency matrix with links near the diagonal indicates that adjacent nodes from the two layers are linked, whereas links far from the diagonal correspond to more distant node pairs. The mathematical formula and implementation are detailed in Appendix D
of links from the
Furthermore, the degree distribution of the BSW model is fixed to the same value for all the nodes at the same layer before rewiring, whereas after rewiring, the degree distribution is not conserved, and the more links it rewires, the more it will be similar to the ER model. Instead, the BRF model has the important property to conserve the degree distribution of the output layer, which ensures that it maintains a designed receptive field connectivity. This means that an initialization setting of the BRF model is the output degree distribution, which in this study we consider fixed or uniformly at random, as shown in Appendix D We also conduct a sensitivity test of the influence of Figure 7p). should be noted that when r = O, the network is equivalent to the BSW with B = 0, and when r 1, the network becomes an ER network. The examples of the adjacency matrices of BSF, BSW, and BRF are shown in Figure 5

---

3.3 Sigmoid Gradual Density Decay
8 incorporating a density decrease strategy can significantly improve the performance of dynamic sparse training. In MESTEM&&s the density is reduced discretely at predefined epochs. In GraNet, the network evolution process consists of three steps: pruning, link removal, and link regrowth. The method first prunes the network to reduce the density, followed by removing and regrowing an equivalent number of links under the updated density. The density decrease in GraNet follows the same approach as Gradual Magnitude Pruning (GMP) 41, which adheres to a eubic function.
As demonstrated in GraNet 44) and MEST EMS
However, this density decay scheduler exhibits sharp decline in the initial stages of training, which risks pruning a substantial fraction of weights before the model has sufficiently learned. To enable a smoother decay during the pruning stage, we propose sigmoid -aseed gradual density decrease strategy, defined as:
8t 8; + (8: 88)
1 14e
where t E {to, to + At...,, starting epoch gradual pruning, l is the end epoch gradual pruning, and Al is the pruning frequency. & controls the curvature of the decrease. We set k=6 for all the experiments in this article. This strategy ensures a smoother initial pruning phase, allowing the model to warm up and stabilize before undergoing significant pruning, thereby enhancing training stability and performance. We explain how to adjust the training FL.OPs of sigmoid density decay to the same as cubic decay in Appendix[[ [[]
to + ndd), 8, is the initial sparsity, sy is the target sparsity, to is the
In addition to refining the decay function, we replace the weight magnitude criterion used in the original GMP and GraNet processes with relative importance (RI). This adjustment is motivated by prior work [32], which has shown that RI provides a significant performance advantage over weight magnitude, particularly when pruning models initialized with high density.
4 Experiments
4.1 Experimental Setup
We evaluate the performance of CHT's using MLPs for image classification tasks on the MNIST [16]. Fashion MNIST [18], EMNIST 17 and CIFARIO [42 datasets. To further validate our approach, we apply the sigmoid gradual density decay strategy to Transformers for machine translation tasks the Multi30k en-de (20 IWSLT14 en-de (2I), and WMT17 en-de 22 datasets. Additionally, we conduct language modeling experiments using the Open- WebText dataset (43) and evaluate zero-shot per- formance on the GLUE [44] and SuperGLUE [45 benchmark with LLaMA-60M, 130M, and 1B models [1. For MLP training, we sparsify all layers except the final layer, ultra-sparsity in the output layer may lead to disconnected neu- rons, and the connections in the final layer are relatively minor compared to the previous layers. For Transformers and LLaMA models, we ap- ply dynamic sparse training (DST) to all linear layers, excluding the embedding and final gen- erator layer. Detailed hyperparameter settings for each experiment are provided in Tables 4 and P sensitivity tests on the components proposed in this article for CHTs and CHTss in Appendix
Table 1: Performance comparison of different spar- sity dynamic sparse training methods on the CI- FARIO dataset trained on an MLP at 99% sparsity. The density decay of GMP, GraNet, and CHTss starts with sparsity of 50%. ACC represents ac- curacy, and ANP denotes the active neuron perco- lation rate, indicating the final size of the network. The lowest anp rate and the best dynamic sparse training method are highlighted in bold, and perfor- mances surpassing the fully connected model are marked with *** The results present standard error taken over three seeds of the experiments.
Method ACC (%)
Comparison to FC ANP
FC
62.85 + 0.16
CHTs
69.97 : 0.06*
+11.33%
54 %
CHT
59.10 + 0.06
-5.97%
96%
Rigl. SET
63.90 + 0.19*
+1.67%
59%
62.70 0.11
-0.24%
100%
CHTss
71.29 1 0.14*
+13.43%
63%
[ We also conduct series of ablation and
GraNet 69.31 - 0.17*
+10.28%
61% 1
GMP
65.11 = 0.11*
+3.60%
75%
H
7

---

Table 2: Performance comparison machine translation tasks of Multi30k, IWSLT, and WMT with varying final sparsity levels. The scores indicate BLEU scores, which the higher the better. CHTs (GMP) represents CHT's with GMP's density decay strategy. Bold values denote the best performance among fixed sparsity DST methods or density decay DST methods. The performances that surpass the fully connected model are marked with "*" The density decay of GMP, GraNet, and CHTss starts with a sparsity of 50%. The scores are averaged over three seeds # their standard error.
Multi30k
IWSLT
WMT
Method
95%
90%
95%
90%
90%
FC
31.38 t 0.38
24.48 + 0.30
25.22
SET
28,99 + 0.28
29.73 0.10
18.53 0.05
20.13
20.19 = 0.12
21.52 0.28
RigL
29.94 + 0.27
30.26 = 0.34
20.53 0.21
0.15
00.71
22.22 = 0.10
CHT
27.79
28.38
18.59
19.91
19.03
21.08 22.40 + 0.06
CHTs
28.94 + 0.57
29.81 = 0.37
21.15
21.92
2.69
MEST MESTEM&S
28.89 0.26
30.04
21.05 0.10
22.22 2.07
0.10
GMP
30.51 + 0.82
0.40
0.87
0.58
0.14
23.37 + 0.08
GraNet
31.31 + 0.31
0.12
23.46 0221 0.09
0.09
CHTss
32.03 + 0.29
32.86
0.16*
24.31 24.51 0.02
23.73 0.43
24.61 0.14
Baseline Methods. We compare our method with the baseline approaches in the literature. We divide the dynamic sparse training (DST) methods into two categories: fixed sparsity DST and density decay DST. For the fixed sparsity DST, we compare CHTs with the SET [5]. Rigl. (7]. and CHT (9) and for the density decay DST methods, we compare CHTss with MEST EM & [8, GMP [41], and GraNet [40]. We explain the detailed implementations and the reason split GMP type of DST method in Appendix G
4.2 MLP for image classification
Main results. In the MLP evaluation, we aim to assess the fundamental capacity of DST methods to train the fully connected module, which common across many ANNs. The sparse topological initialization of CHT and CHTs is CSTI [9] since the input bipartite layer can directly receive information from the input pixcls. Table 7 displays the performance of DST methods compared to their fully connected counterparts across three basic datasets of MNIST, Fashion MNIST, and EMNIST. The DST methods are tested at 99% sparsity. As shown in Table 77] both of the two regrowth methods of CHTs outperform the other fixed sparsity DST methods. Notably, the path-based CH3-L3p outperforms the fully connected one in all the datasets. The node-based CH2-L3n also achieves comparable performance these basic datasets. However, considering the running time of CH3-L33 is unacceptable, especially in large scale models, in the rest of the experiments of this article, we only use CH2-L.3n as the representative method to regrow new links. Table I presents comparison of fixed-sparsity dynamic sparse training (DST) methods against the fully connected (FC) baseline. Notably, CHTs outperforms all other DST methods and achieve 11% improvement in accuracy over the fully connected model. In addition, we present the active neuron post- post-percolation rate (ANP) for each method in Table and Table It is evident that CHTs adaptively percolates the network more effectively while retaining performance.
4.3 Transformer on Machine Translation
We assess the Transformer's performance on a classic machine translation task across three datasets. We take the best performance of the model on the validation set and report the BLEU on the set. Beam search, with a beam size of 2, is employed to optimize the evaluation process. In our evaluation, CHT' and CHTss configure the topology of each layer using the BRF model, employ weight magnitude soft link removal technique, and regrow new links using CH2-L3n-soff.. Additionally, we apply an adjusted network percolation technique to the Transformer, as detailed Appendix The findings, presented in Table demonstrate that 1) CHT's surpasses other fixed density DST methods on all the sparsity scenarios except for Multi30K. 2) Incorporating the sigmoid density decrease, CHTss outperforms even the fully connected ones with only 5% density Multi30K and IWSLT.
8

---

Table 3: Validation perplexity of different dynamic sparse training (DST) methods on Open- WebText using LLaMA-60M, LLaMA-130M, and LLaMA-1B across varying sparsity levels. Bold values denote the best performance among fixed sparsity DST methods or density decay DST methods. Lower perplexity corresponds to better model performance. GMP, GraNet, and CHTss are run with an initial sparsity of 8; = 0.5. The test of CHT over LLLMM-1BB missing due to its excessive runtime. The performances that surpass the fully connected model are marked with "*"
LLaMA-60M
LLMMA-13M
LLaMA-IB
Method
70%
80%
90%
95%
70%
80%
90%
95%
70%
FC
26.56
19.27
14.62
SET
31.77
30.69
35.26
39.70
20.82
22.02
24.73
28.37
16.37
RigL
39.96
41.33
45.34
51.49
25.85
66.35
37.18
49.39
149.17
CHT
31.02
32.99
35.01
41.87
21.02
22.82
26.27
30.01
CHTs
28.12
29.84
33.03
36.47
20.10
21.33
23.71
26.45
14,53*
MEST
28.26
29.94
33.60
37.87
21.32
22.21
24.98
27.96
60.36
GMP
29.22
30.59
33.68
39.00
20.49
22.28
23.61
27.16
31.76
GraNet
30.55
31.51
33.76
39.98
22.84
29.03
26.81
61.31
79.44
CHTss
27.62
29.00
31.42
35.10
19.85
20.70
22.51
25.07
15,41
4.4 Natural Language Generation
Language modeling. We utilize the LLaMA model family [ across 60M, 130M, and IB architec- ture as the baseline for our language generation experiments. We follow the experiment setup from (46) detailed in Table To ensure that the FLOPs are the same for all the density decrease methods, for CHTss, we implement a half-step strategy for the density decay steps to make sure the sparsity across the training steps is the same GMP and GraNet.
Table 3 shows the validation perplexity results of each algorithm across different density levels on aLMMA-60 and LLLMM-130M.. CHTSs stably outperforms SET, CHT, and Rigl., while CHTss are constantly better than GraNet and GMP. At 70% sparsity, CH'Tss is already able perform comparably to the fully connected model. It is important to note that Rigl. and GraNet exhibit subpar performance in this evaluation due to the use of bfloat16 precision in this configuration. This lower precision adversely impacts gradient accuracy, particularly in the early stages of training. Since both RigL and GraNet rely on gradient information to regrow new links, the imprecise gradients lead to regrowing the wrong links, thereby hindering their performance. To further validate this observation, we conduct additional experiments under FP32 precision, presented in Table IO The results confirm that RigL and GraNet perform significantly better under high-precision training, although they still fall behind CHTs and CHTss. Importantly, industry trends are increasingly moving toward precisiio training and inference improve efficiency [47 48). In this context, CHT's and CHTss demonstrate greater robustness compared to RigL. and GraNet, making them better aligned with practical deployment needs under aeduccdd-precssion settings.
5 Conclusion and Discussion
We advance current knowledge in brain-inspired dynamic sparse training, proposing the Cannistraci- Hebb Training soft rule with sigmoid gradual density decay (CHTss). First, we introduce matrix multiplication mathematical formula for GPU-friendly approximation of the CH link predictor. This significantly reduces the computational complexity of CHT and speeds up the running time, enabling the implementation of CHT's in large-scale models. Second, we propose a Cannistraci- soft rule (CHTs), which innovatively utilizes a soft sampling rule for both removal and regrowth links, striking a balance for epitopological exploration and exploitation. Third, we propose the Bipartite Receptive Field (BRF) model initialize the sparse network topology in prain-inspired manner, enabling the network to preferentially capture spatially closer features. Finally, in transformer-bant based models, we integrate CHT' with a sigmoid gradual density decay strategy (CHTss). Empirically, CHT's demonstrate a remarkable ability to achieve ultra-sparse configurations-ug to 99% sparsity in MLPs for image classification suusfiiation-ssupassinnn fully connected networks. Notably, the regrowth process under CHTs does not rely gradients. With the sigmoid gradual density decay, CHTss surpasses
training

---

the fully connected Transformer using only 5% density and achieves comparable language modeling performance. Moreover, CHTs and CHTss achieve language modeling performance comparable to the fully connected LLaMA-60M and LLaMA-130M models, while CHT's even surpasses the fully connected LLaMA-1B counterpart. We describe the limitations of this study and future works in Appendix A
Acknowledgements
This work was supported by the Zhou Yahui Chair Professorship award of Tsinghua University (to CVC), the National High-Level Talent Program of the Ministry of Science and Technology of China (grant number 20241710001, to CVC).
References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397) 2023.
[2] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint (rxXxx:225.000666 2022.
[3] David A Drachman. we have brain to spare?, 2005.
[4] Christopher Walsh. Peter huttenlocher (1931-2013) Nature, 502(74700: 2013.
[5] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.
[6] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances Neural Information Processing Systems, 33:20744 20754, 2020.
[7] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 2943-2952 PMLR, 2020.
[8] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory eccoooicc sparse training framework the edge. Advances in Neural Information Processing Systems, 14:20838-20855 2021.
[9] Yingtao Zhang, Jialin Zhao, Wenjing Wu, Alessandro Muscoloni, and Carlo Vittorio Cannistraci. Epitopological learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. In The Twelfth International Conference on Learning Representations, 2024.
[10] Anthony Holtmaat and Karel Svoboda. Experience-depenndent structural synaptic plasticity in the mammalian brain. Nature Reviews Neuroscience, 10(9):647-66 658. 2009.
[11] Carlo Vittorio Cannistraci. Gregorio Alanis-Lobato, and Timothy Ravasi. From fink-predittion in brain connectomes and protein interactomes to the coomunity-paradeee in complex networks. Scientific reports, 3(1):1613, 2013.
[12] Simone Daminelli, Josephine Maria Thomas, Claudio Durán, and Carlo Vittorio Cannistraci. Common neighbours and the aaal-commuitty-prradiea for topological link prediction in bipartite networks. New Journal of Physics, 13111) nov 2015.
10

---

[13] Claudio Durán, Simone Daminelli, Josephine M Thomas, V Joachim Haupt. Michael Schroeder, and Carlo Vittorio Cannistraci. Pioneering topological methods for network- based drug drug-target prediction by exploiting a brain-network self-organization theory. Briefings in Bioinformatics, 19(6): 182-1202, 04 2017.
[14) Carlo Vittorio Cannistraci. Modelling self-organization in complex networks via a brain- inspired network automata theory improves link reliability in protein interactomes. Sci Rep, 8(1):2045 23222, 10 2018.
[15] Vaibhav et al Narula. Can local local-community paradiig and epitopological learning enhance our understanding of how local brain connectivity is able process, learn and memorize chronic pain? Applied network science, 2(1), 2017.
[16] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bant learning applied to document recognition. Proc. IEEE, 86(11)2278-23324 1998.
[17] Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[18] Han Xiao. Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint (XXiv:770..0744: 2017.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[20] Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english-germen german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70-74. Association for Computational Linguistics, 2016.
[21] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Marcello Federico, Sebastian Stüker, and François Yvon, editors, Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2-17, Lake Tahoe, California, December 4-5 2014.
[22] Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, al. Findings of the 2017 conference on machine translation (wmt17). Association for Computational Linguistics, 2017.
[23] Ameya Prabhu, Girish Varma, and Anoop Namboodiri. Deep expander networks: Efficient deep networks from graph theory. Proceedings of the European Conference on Computer Vision (ECCV), pages 20-35, 2018.
[24] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based on connection sensitivity. In 7th International Conference on Learning Represen- tations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[25] Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference Machine Learning, pages 4690-4721. PMLR, 2022.
[26] James Stewart, Umberto Michieli, and Mete Ozay. Data-free model pruning at initialization via expanders. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4518-4523. 2023.
[27] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv:1711.051366 2017.
[28] Mike Lasby, Anna Golubeva, Utku Evei, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.02299 2023.
[29] Donald Hebb. The organization of behavior. emphnew york, 1949.
11

---

[30] Alessandro Muscoloni, Umberto Michieli, Yingtao Zhang, and Carlo Vittorio Cannistraci. Adaptive network automata modelling of complex networks. Preprints, May 2022.
[31] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parameteerizatonn? in-time over-parametterizatinn in sparse training. In International Conference on Machine Learning, pages 7008. PMLR, 2021.
[32] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[33] Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774 2023.
[34] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprini arXiv:2306. 2023.
[35] Paul Erdös and Alfréd Rényi. On the evolution of random graphs. Publication of the Mathemat- ical Institute of the Hungarian Academy of Sciences, 5:17-60, 1960.
[36] Duncan Watts and Steven H Strogatz. Collective dynamics of 'small-worldd networks. Nature, 39366684))4440-442, 1998.
[37] Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science, 286(5439):509-512 1999.
[38] Maria Ercsey-Ravasz, Nikola T Markov, Christophe Lamy, David C Van Essen, Kenneth Knoblauch, Zoltán Toroczkai, and Henry Kennedy. predictive network model of cerebral cortical connectivity based on distance rule. Neuron, 80(1):184-197, 2013.
[39] Danielle S Bassett and Edward Bullmore. Small-world brain networks. The Neuroscientist, 12(6))512-522) 523,, 2006.
[40] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9999 090889992,2222 2021.
[41] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
[42] Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.
[43] Aaron Gokaslan and Vanya Cohen. Openwehtext corpus. http:://Skylion07..iiii io/ OpenWebTextCCoruu 2019.
[44] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[45] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for seneral-purposs language understanding systems, 2020.
[46] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient IIm training by gradient low-rank projection, 2024.
[47] Pengle Zhang, Jia Wei, Jintao Zhang, Jun Zhu, and Jianfei Chen. Accurate int8 training through dynamic bloek-level fallback. arXiv preprint arXiv:2503.0804 080400 2025.
[48] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Advances in Neural Information Processing Systems, 37:87766-87800, 2024.
12

---

[49] Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. Spdf: Sparse pre-training and dense fine-tuning for large language models. In Uncertainty in Artificial Intelligence, pages 2134-21 PMIR, 2023.
[50] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5533-5543, Virtual, 13-18 Jul 2020. PMLR.
[51] P ERDdS and A R&wi. On random graphs i. Publ. math. debrecen, 6(290-297):18, 1959.
[52] Ming Li, Run-Ran Liu, Linyuan Lü, Mao-Bin Hu, Shuqi Xu, and Yi-Cheng Zhang. Percolation on complex networks: Theory and application. Physics Reports, 907:1-68, 2021.
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (tti:2307.002888 2023.
[54] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[55] Aleksandra I. Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacek Tabor. Fantastic weights and how to find them: Where to prune in dynamic sparse training, 2023.
[56] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
[57] Jie Ji, Gen Li, Lu Yin, Minghai Qin, Geng Yuan, Linke Guo, Shiwei Liu, and Xiaolong Ma. Advancing dynamic sparse training by exploring optimization opportumities. In Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11606-21619. PMLR, 2024.
[58] Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating compressedddecoompsssee training of deep neural networks. arXiv preprint arXiv:2106.12377 2021.
[59] Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter E. Latham, and Yee Whye Teh. Powerpropagation: A sparsity inducing weight reparameterisation. arXiv preprint 1 2021.
[60] Jie Ji, Gen Li, Jingjing Fu, Fatemeh Afghah, Linke Guo, Xiaoyong Yuan, and Xiaolong Ma. single-step, sharpness-aware minimization is all you need to achieve efficient and accurate sparse training. In Advances in Neural Information Processing Systems, 2024.
[61] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03620. 2022.
[62] Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, and Shiwei Liu. Dynamic sparsity is channel-level sparsity learner. In Advances in Neural Information Processing Systems, 2023.
[63] Gen Li, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, and Xiaolong Ma. Neurrev: Train better sparse neural network practically via neuron revitalization. In International Conference on Learning Representations, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We mentioned all of our innovations and the contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitation is discussed in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper contains all the information reproduce. The codes are available in the Supplementary Information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The codes and instructions are available in the Supplementary Information.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/t.ps..ccppullc//guui /
www.SSubbisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper contains all the information to reproduce.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the results of MLP, include the standard error of the 3-seeds runs for each method. For the larger models, because of the resource limitation, we cannot run three seeds before submission. We will include three seeds results during revision period.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the computation resources requirements in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This research adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed broader impacts in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Assets used in the paper properly credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Table 4: Hyperparameters of MLP on Image Classification Tasks.
Hyper-parameter
MLP
Hidden Dimension
1568 (3072 for CIFARIO)
Hidden layers
3
Batch Size
32
Training Epochs
100
LR Decay Method
Linear
Start Learning Rate
0.025
End Learning Rate
2.5e -4
(fraction of removal)
0.3
Update Interval (for DST)
I epoch 0.9
Momentum
Weight decay
5e -4
Table 5: Hyperparameters of Transformer on Machine Translation Tasks. inoam refers to learning rate scheduler that incorporates iterative warm-up phases, specifically designed for dynamic sparse training (DST) methods. The purpose is allow newly regrown connections to accumulate momentum, preventing potential harm the training process. For the fully connected (FC) baseline, only the standard noam scheduler is used.
Hyper-parameter
Multi30k
IWSLT14
WMT17
Embedding Dimension
512
512
512
Feed-forward Dimension
1024
2048
2048
Batch Size
1024 tokens
10240 tokens
12000 tokens
Training Steps
5000
20000
80000
Dropout
0.1
0.1
0.1
Attention Dropout
0.1
0.1
0.1
Max Gradient Norm
0
0
0
Warmup Steps
1000
6000
8000
Learning rate Decay Method
inoam
inoam
inoam
Iterative warmup steps
20
20
20
Label Smoothing
0.1
0.1
0.1
Layer Number
6
6
Head Number
8
8
Learning Rate
0.25
2
2
(fraction of removal)
0.3
0.3
0.3
Update Interval (for DST)
100 steps
100 steps
100 steps
Limitations and Future Work
A potential limitation of this work is that the hardware required to accelerate sparse training with unstructured sparsity has not yet become widely adopted. Consequently, this article does not present a direct comparison of training speeds with those of fully connected networks. However, several leading companies 49 (50] have already released devices that support unstructured sparsity in training.
For future work, we aim to develop methods for automatically determining the temperature for soft sampling at each epoch, guided by the topological features of each layer. This could enable each layer to learn its specific topological rules autonomously. Additionally, we plan test CHT's and CHTss in larger LLMs such as LLaMA-7b to evaluate the performance in scenarios with denser layers.
B Broader Impact
In this work, we introduce novel methodology for dynamic sparse training aimed at enhancing the efficiency model training. This advancement holds potential societal benefits by increasing interest in more efficient AI practices. However, the widespread availability of advanced artificial
21

---

Table Hyperparameters of LLaMA-60M, LLaMA-130M, and LLaMA-IB on Open inoam refers to learning rate scheduler that incorporates iterative warm-up phases, specifically designed for dynamic sparse training (DST) methods. The purpose is to allow newly regrown connections to accumulate momentum, preventing potential harm to the training process. For the fully connected (FC) baseline, only the standard noam scheduler is used.
Hyper-parameter
LLaMA-60M
LLMMA-130M
LLaMA-1B
Embedding Dimension
512
768
2048
Feed-forward Dimension
1376
2048
5461
Global Batch Size
512
512
512
Sequence Length
256
256
256
Training Steps
10000
30000
100000
Learning Rate
3e-3 (Ie-3 for FC)
3e-3 (Ie-3 for FC)
3e-3 (4e-4 for FC)
Warmup Steps
1000
10000
10000
Learning rate Decay Method
inoam
inom
inoam
Iterative warmup steps
20
20
20
Optimizer
Adam
Adam
Adam
Layer Number
12
Head Number
12
32
C (fraction of removal)
0.1
0.1
0.1
Update Interval (for DST) 100 steps
100 steps
100 steps
Table 7: Performance comparison of different dynamic sparse training methods on
MNIST, Fashion MNIST (FMNIST), and EMNIST datasets trained MLP at 99% sparsity. ACC represents accuracy, and ANP denotes the active neuron percolation rate, indicating the final size of the network. Accuracies present standard error taken over three seeds. The best dynamic sparse training method for each dataset is highlighted in bold, and the performances that surpass the fully connected model are marked with "*"
MNIST
FMNIST
EMNIST
Method
ACC (%)
ANP
ACC (%)
ANP
ACC (%)
ANP
FC
98.78 + 0.02
90.88 0.02
87.13 0.04
CHTs"
98.81 0.04*
20%
90.93 0.03
89%
87.61 + 0.07
24%
CHTs"
98.76 + 0.05
27%
90.67
0.05
73%
87.82 + 0.04
CHT
98.48 + 0.04
29%
88.70 + 0.07
30%
86.35 + 0.08
21%
RigL
98.61 + 0.01
29%
89.91 + 0.07
55%
86.94 0.08
28%
SET
98.14 = 0.02
100%
89.00 = 0.09
100%
86.31
0.08
100%
CHTss
98.83 + 0.02*
32%
90.81
0.11
40%
87.52 + 0.04
35 %
GraNet
98.81 0.00*
35%
89,98
0.06
53%
86.94 0.03
45%
GMP
98.62 + 0.03
58
90.29 0.19
69%
0.09 86.93
75 %
P Refers to regrowth method CH3
CH3_L.3p.
" Refers to the regrowth method CH2 L.3n.
neural networks, particularly large language models (LLMs), also presents risks of misuse. It is essential to carefully consider and manage these factors to maximize benefits and minimize risks.
C Cannistraci-Hebb epitopological rationale
The original CHT framework leverages the Cannistraci-Han Hebb link predictor on Length paths (CH3- L3p) metric for link regrowth. Given two seed nodes u and in network, this metric assigns score
CH3-L3p(u,u)-
Σ
(5)
de, det
22

---

Table 8: Perplexity (PPL) results across different sparsities (0.7, 0.8, 0.9, 0.95) for CHTss and CHTss under different regrowth strategies (Fixed and Uniform) and r settings on LLaMA60M.
Fixed
Uniform
Sparsity
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
70%
28.16
28.39
28.25
28.32
30.11
28.12
28.43
28.56
80%
30.22
29.84
30.04
30.03
30.19
29.86
30.23
30.06
CHTs
90%
33.32
33.37
33.03
33.77
33.45
33.36
33.88
33.72
95%
37.29
37.51
37.24
37.46
37.23
36.47
37.33
37.67
70%
27.62
30.05
27.82
28.43
27.62
27.74
27.74
27.68
80%
29.00
29.00
29.66
32,91
29.49
29.69
29.09
29.24
CHTss
90%
31.51
31.67
31.65
31.59
31.66
32.61
31.68
31.42
95%
38.66
35.31
36.24
37.50
42.20
37.40
35.36
35.10
Table Perplexity (PPL) results across different sparsities (0.7, 0.8, 0.9, 0.95) for CHTs and CHTss under different regrowth strategies (Fixed and Uniform) and settings on LLaMA-130M,
Fixed
Uniform
Sparsity
r 0.0
0.1
y= 0.2
r 0.3
0.0
0.1
r 0.2
0.3
70%
20.24
20.16
20.10
20.25
20.62
20.18
20.15
20.20
80%
21.33
21.37
21.36
21.48
21.34
21.40
21.40
22.49
CHTs
90%
23.72
23.76
23.76
23.94
23.74
23.73
23.71
24.99
95%
28.05
26.45
26.90
26.91
26.78
27.97
29.05
27.10
70%
20.63
19.88
19.93
19.85
21.43
19.90
20.93
19.94
80%
20.71
22.60
20.86
20.70
20.73
20.74
20.72
20.82
CHTss
90%
22.58
22.72
22.61
22.51
22.53
22.59
22.60
23.12
95%
25.28
25.12
25.20
25.12
25.07
25.15
25.23
25.12
Here, and denote the seed nodes, while 21 and are common neighbors on the L3 path [30,. walk of three consecutive links that connects represents the number of external local community links (eI.CL.) of node i, with default increment of to prevent division by zero. Path-based link prediction has demonstrated its effectiveness on both real-world networks (30) and artificial neural networks [9]. However, this method incurs a high computational cost due to the need to compute and store all length-three paths, resulting in a time complexity of O(N dd), where N is the number of nodes and d is the network's 's average degree. This complexity prohibitive for large models with numerous nodes and higher-density layers. To
via those two intermediate nodes. The term de;
Perplexity
ITOP
90
Soft
0.7
Random
80
Deterministic
0.6
Perplexity
70
Soft
ITOP
Random
Deterministic
0.3
50
0.2
40
0.1
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Steps
Steps
Figure 3: Comparison of link regrowth strategies in CHTs using a LLaMA- model trained on OpenWebTes for 5000 steps. The left plot shows validation perplexity (lower better), while the right plot reports the in-time over-paran parameeerizationn (ITOP) rate, which measures the cumulative proportion of links activated during training. Results are presented for three strategies: Soft, Random, and Deterministic regrowth.
23

---

L2
L3 local community
nLCL
local community
eLCL
ILCL
CN
seed
U
V
?
L2 paths
L3 paths
Figure 4: Cannistraci-Hebb epitopological rationale. [30 The figure illustrates an explanatory example of topological link prediction using the Cannistraci-Hebb epitopological rationale based on either L2 or L3 paths. The two black nodes represent the seed nodes whose unobserved interaction to be assigned a likelihood score. White nodes denote the common neighbours (CNs) of the seed nodes at either L2 or L3 distance. Together, the set of CNs and the internal local community links (iLCL) constitute the local community. Different link types are color-coded: green for nLCLs, red for external local community links (eLCLs), and white for iLCLs. The L2 (path length 2) and L3 (path length 3) paths associated with the illustrated communities are highlighted. Notably, in artificial neural networks (ANNs), linear layers correspond to bipartite networks, which inherently support only L33 path predictions, as shown in Figure
address this issue, we introduce a more efficient, node-based paradigm that eliminates the reliance on length-three paths between seed nodes. Instead, this approach focuses on the common neighbors of seed nodes. The node-based version of CH3-L3p, denoted as CH2-L3n, is defined as follows:
di
(H)
(6)
de
SEL3
Here, u and v denote the seed nodes, while 2 is the common neighbor on the L3 path [30], a walk of three consecutive links that connects represent the number of internal local community links (iLCLs) and external local community links (eLCLs) of node i, with default increment of I to prevent division by zero. Internal local community links (iLCLs) are those that connect nodes belonging to the same local community. Contrarily, external local community links (eLCLs) connect nodes belonging to different communities. Figure gives visual representation of L2 and L3 paths between two seed nodes u and defining their local community.
via two of those intermediate nodes. The terms diž de
D Sparse topological initialization
Correlated sparse topological initialization. Correlated Sparse Topological Initialization (CSTI) a physics-informed topological initialization. CSTI generates the adjacency matrix by computing the Pearson correlation between each input feature across the calibration dataset and then selects

---

the predetermined number of links, calculated based on the desired sparsity level, as the existing connections. CSTI performs remarkably better when the layer can directly receive input information. However, for layers that cannot receive inputs directly, it cannot capture the correlations from the start since the model is initialized randomly, as in the case of the Transformer. Therefore, in this article, we aim to address this issue by investigating different network models to initialize the topology, to improve the performance for cases where CSTI cannot be directly applied.
Bipartite scale-free model. In artificial neural networks (ANNs), fully connected networks are inherently bipartite. This article explores initializing bipartite networks using models from network science. The Bipartite Scale-Free (BSF) [9] network model extends the concept of scale-freeness to bipartite structures, making them suitable for dynamic sparse training. Initially, the BSF model generates monopartite Barabási-Albert (BA) model [37], well-established method for creating scale-free networks in which the degree distribution follows a power law (2=2.76 in Figure [5) Following the creation of the BA model, the BSF approach removes any connections between nodes of the same type (neuron in the same layer) and rewires these connections to nodes of the opposite type (neuron in the opposite layer). This rewiring is done while maintaining the degree of each node constant to preserve the power-law exponent 7.
Bipartite small-world model. The Bipartite Small-World (BSW) network model [9] is designed to incorporate woald-world properties and a high clustering coefficient into bipartite networks. Initially, the model constructs regular ring lattice and assigns two distinct types of nodes to it. Each node is connected by an equal number of links to the nearest nodes of the opposite type, fostering high clustering but lacking the small-world property. Similar to the Watts-Strogatz model (WS) [36), the BSW model introduces a rewiring parameter, 3, which represents the percentage of links randomly removed and then rewired within the network. At 1, the model transitions into an Fidōs-Réény model [51), exhibiting small-world properties but without a high clustering coefficient, which is
popular as the topological initialization of the other dynamic sparse training methods (5
Bipartite receptive field model. The Bipartite Receptive Field (BRF) model is a random network generation technique designed to mimic the receptive field phenomenon in the brain networks. The process involves adding links to the adjacency matrix of the bipartite network, with the connectivity structured around the main diagonal according to a parameter E [0, 1]. A low value of r results in links that are primarily clustered around the diagonal, while a higher value of r leads to a more random connectivity pattern. Specifically, a bipartite adjacency matrix with links near the diagonal indicates that adjacent nodes from the two layers are linked, whereas links far from the diagonal correspond to more distant node pairs. Mathematically, consider an N x M bipartite adjacency matrix Mijiim,..... I......... where M represents the input size and N represents the output size. Each entry of the matrix mi.j is set to if input node is connected to output node j, and 0 otherwise. scoring function St,5 is assigned to each connection in the adjacency matrix based on its distance to the main diagonal. This score is given by:
S.j dij
where
(8) is the distance between the input and output neurons. Therefore, St., represents the distance from the diagonal, raised to the power of 1_r The parameter r controls how structured or random the adjacency matrix is. As - 0, the scoring function becomes more deterministic, with high scores assigned to entries near the diagonal and low scores to entries farther away. Conversely, as r + 1, all scores St., become more uniform, leading to a more random, less structured adjacency matrix. The next step is to determine the degree distribution for the output nodes. This can either be fixed, assigning the same degree to all output nodes, or uniform, where the degrees are randomly sampled from uniform distribution. Hence, we propose two variations of the BRF model: the Bipartite Receptive Field with fixed sampling (BRFf), in which the degrees of output nodes are fixed, and the Bipartite Receptive Field with uniform sampling (BRFu), where the degrees of the output nodes follow a uniform distribution. This represents an additional enhancement to the WS scheme, which offers no way to control how connections are allocated among the output nodes. In conclusion, to run the BRF model, the user should input an output degree distribution and a spatial dependent distance randomness.
dy manfli - 11, -i.---i-1,166 - M) - (j-
25

---

Bipartite Small-World network model
Resudar network (f
= 0.25
P 0.5
# = 0.75
Excés- Kétyy network @
Biportito Stale Proe network modal
Bipartite Receptive Field - Fixed Sampling
Poweriow 276)
Regular
Bipartite Receptive Field Uniform Sampling
Regular
0.25
IL75
The adjacency matrix of the Bipartite Scale-Free (BSF) network model compared to those of the Bipartite Small- World (BSW) network, the Bipartite Receptive Field with fixed sampling (BRFy), and the Bipartite Receptive field with uniform sampling (BRF,) as parameters / and vary between and 1. a) The BSF model inherently forms scale-free network characterized by power-law distribution with = 2.76. b) As 3 changes from 0 to 1, the network exhibits reduced clustering. It important network. c) As increases towards 1, the adjacency matrix becomes more random, while sampling the output neurons' degrees from fixed uniform distribution.
Figure 5:
that when B the BSW model does not qualify
woll-worlld
E Equal Partition and Neuron Resorting to enhance bipartite scale-free network initialization
As indicated in SET and CHT 91. trained sparse models typically converge to scale-free network. This suggests that initiating the network with a scale-free structure might initially enhance performance. However, starting directly with Bipartite Scale-Free model (BSF, power-law exponent 2.76) does not yield effective results. Upon deeper examination, two potential reasons emerge:
The BSF model generates hub nodes randomly. However, this random assignment of hub nodes to less significant inputs leads to less effective initialization, which is particularly detrimental in CHT, which merely utilizes the topology information to regrow new links.
As demonstrated in CHT, in the final network, the hub nodes of one layer's output should correspond to the input layer of the subsequent layer, which means the hub nodes should have high degree both sides of the layer. However, the BSF model's random selection disrupts this correspondence, significantly reducing the number of Credit Assignment Paths (CAP) [ in the model. CAP is defined the chain of transformation from input to output, which counts the number links that go through the hub nodes in the middle layers.
To address these issues, we propose two solutions:
Equal Partitioning the First Layer: We begin by generating a BSF model, then rewire the connections from the input layer to the first hidden layer. While keeping the out-degrees of the output neurons fixed, we randomly sample new connections to the input neurons until each of the input neurons' in-degrees reaches the input layer's average in in-degree. This
26

---

a)
b)
BSW network model with different 8
Different link removal methods
0.35
0.4
0.30
0.25
0.3
0.20
0.2
0.10
0.05
Figure 6: The ablation test of the 8 of the bipartite small world (BSW) model and the removal methods in CHTs. a) evaluates the influence of the rewiring rate 8 the model performance when initialized with the BSW model. b) assesses the influence of link removal selecting from the weight magnitude (WM), weight magnitude soft (WM-soft), relative importance (RI), and relative importance soft (RI-soft). We utilize the win rate of the compared factors under the same setting across each realization of seeds for all experiment combinations on MLP. The factor with the highest win rate is highlighted in orange.
a) BRF nelwork model with different
b) Different Link-Removal Melhods
0.6
0.4
0.3
0.2
0.2
0.0
0.0
0.0
0.25
0.5
0.75
1.0
WM
WM-soft
RI
RI-soft
Figure 7: The ablation test of the parameter in the bipartite receptive field (BRF) model and the removal methods in CHT's using the BRF initialization technique. a) evaluates the influence of the parameter on the model performance when initialized with the BRF model. b) assesses the influence of link removal in the CHTs model with BRF initialization. We utilize the win rate of the compared factors under the same setting MLP. The factor with the highest win rate is highlighted in orange.
each realization of 3 seeds for all experiment combinations on
approach ensures all input neurons are assigned equal importance while maintaining the power-law degree distribution of output neurons.
Resorting Middle Layer Neurons: Given the mismatch in hub nodes between consecutive layers, we suggest permuting the neurons between the output of one layer and the input of the next, based on their degree. higher degree in an output neuron increases the likelihood of connecting to a high-degree input neuron in the subsequent layer, thus enhancing the number of CAPs.
As illustrated in Figure 8 while the two techniques enhance the performance of the BSF initialization. they remain inferior to the BSW initialization. As noted in the main text, achieving scale- freeness more effective when the model is allowed to learn and adapt dynamically rather than being directly initialized as a predefined structure.
Network percolation and extension to Transformer.
We have adapted network percolation 52 9 to suit the architecture of Transformer after link removal. The core idea is to identify inactive neurons, which are characterized by having
27

---

a)
BSF netwark model with different strategies
b)
A4E (Tro Spect
CST
0.4
RSV
BSF Racorl
B9F
Win cate 180)
0.2
0.2
0.0
6SF
BSP-EP
BSP Posort BSP EP RaSorT RasorT
Deleer ST
Figure 8: The Performance of the bipartite scale-free model and two enhanced techniques. a) shows the win rate of the Bipartite Scale-Free network model (BSF) with the different techniques. EP stands for equal partition of the first layer, and Resort refers to reordering the neurons based on their degree. b) assesses the comparison between Correlated Sparse Topological Initialization (CSTI), the Bipartite Scale-Free (BSF) model with the best solution from a), and the Bipartite Small-Wors (BSW) model with 3 = 0.25.
connections on either one or both sides within layer of neurons. Such neurons disrupt the flow of information during forward propagation or backpropagation, In addition. Layer-wiss wiier computation of the CH link prediction score further implies that neurons without connections on one side are unlikely to form connections in the future. Therefore, network percolation becomes essential to optimize the use of remaining links.
As shown in Figure I network percolation encompasses two primary processes: cl) inactive neuron removal remove the neurons that lack connections on one or both sides; c2) incomplete path adjustment to remove the incomplete paths where links connect to the inactive neurons after cl). Typically applied in simpler continuous layers like those an MLP, network percolation requires modification for more complex structures. For example, within the Transformer's 's self self-attention module, the outputs of the query and key layers undergo dot product operation. It necessitates percolation in these layers to examine the activity of the neurons in both output layers at the same position, Similar interventions are necessary in the up_proj and gate_proy layers of the MI.P module in the LLaMA model family 11
G Baseline Methods
G.1
Fixed Density Dynamic Sparse Training Methods
SET
Removes connections based on weight magnitude and randomly regrows new links.
RigL Removes connections based on weight magnitude and regrows links using gradient information, gradually reducing the proportion of updated connections over time.
A state-of-the-art (SOTA) gradient-free method that removes links with weight magnitude and regrows links based on CH3-L3 scores. CHT often applied with early stopping to mitigate its computational complexity when working with large models.
CHT 9
G.2
Gradual Density Decrease Dynamic Sparse Training Methods
[1]: Prunes the network with weight magnitude and gradually decreases the density Although originally pruning method, GMP is treated dynamic sparse training method in their implementation [41, as stores historical weights and allows pruned weights reappear during training, since, during training, the pruning threshold might change.
GMP
54
based Equation 10
g Implements two-stage density decrease strategy described in the original work. removes links based on the combination of weight magnitude and 0.01 ogradiett and regrows new links randomly.
MEST FARSS
28

---

GraNet [40]:
Gradually decreases density using Equation 10, Similar to RigL, GraNet removes links based on the weight magnitude and regrows new links with the gradient of the existing links.
Table 10: Float32 Precision Comparison on LLaMA- 130M. Bold values denote the best performance among DST methods. Lower perplexity corresponds to better model performance. represents the initial sparsity for DST methods employing a density decay strategy.
Method
Sparsity
70%
80%
FC
17.07
RigL
18.34
19.64
CHTs
17.99
19.25
GraNet (8; = 0.5)
17.92
18.79
CHTss (s; 0.5)
17.76
18.69
H Ablation and Sensitivity Tests
An overall ablation test To fully assess each component's effectiveness, we conduct several ablation and sensitivity tests that help us understand how select sparse topological initialization and identify the best link removal and regrowth methods. We first made a global test for all the components in Table [T] which shows the effectiveness of each element introduced by this article. The node-based and path- path-based link regrowth methods have comparable performance, but the node-based versions are much faster.
Sparse topological initialization. For sparse topological initialization, we compare BRF, BSW, BSF, and CSTI [9] across three image classification datasets, shown in Figure Sp. The results indicate that when the inputs can directly access task-relevant information, CSTI consistently achieves the best performance. In general, BRF and BSW perform similarly under these conditions, but outperform the BSF initialization.
To further validate findings, evaluate BRF and BSW network initializations machine trans- lation tasks using Transformer models. Figure 9 and Figure I0 present the performance comparisons between BSW and BRF the Multi30k and TWSLT datasets, while Figure 11 shows the win-rate analysis. These comparisons demonstrate that BRF consistently outperforms BSW across most cases. Additionally, Figure 7h analyzes the impact of the receptive field range BRF initialization for MNIST, Fashion MNIST, and EMNIST tasks using MLPs, with results indicating that r = 0.25 yields the best performance.
Building on this prior knowledge, we further evaluate BRF on LLaMA-60N and LLaMA-130M models, testing values in the range [0, 0.3] and comparing two different degree distributions. The results, shown in Table and Table indicate that on LLaMA models, the choice r and distribution has limited impact. While 0.1 wins slightly more often, the improvements remain marginal. Finally, Table reporrts the best performance combinations of and degree distributions derived from these evaluations.
Link removal. We first conduct a simple evaluation of the link removal methods introduced in this article when changing the and 8 inside Figure 1p2) on Figure/bb) and Figure 7p). The removal methods are selected from Weight Magnitude (WM), Weight Magnitude soft (WMs), Relative Importance (RI), and Relative Importance soft (RIs). For WM we fix the hyperparameters
1 and 0.5; for WMs we fix = 1 and we let
1; for RI we fix the hyperparameters a and increase linearly from 0.5 to 0.9; for RIs we fix a 0 and let d increase linearly from 0.5 to 0.9. From the results, it can be observed that WMs performs the best most cases. We compare these methods with those in (55] in Table 12 two machine translation tasks. The results indicate that using WMs as a link removal method generally outperforms the alternatives.
We also evaluate how to define the softness in WMs. During sampling, we have hyperparameter to decide the temperature of the scores that convert to the probability of being removed. We perform test using a linear decay solution, since, generally, the weights in the model become more reliable as
29

---

Table 11: Ablation results of Transformer Multi30K and IWSLT datasets 90% sparsity. The scores indicate BLEU scores, the higher the better. Bold values denote the best performance among DST methods.
Multi30K
IWSLT (90% sparsity)
Variant
(90% sparsity)
CHT
28.38
19.91
CH'sss
32.68 (2.42 hours)
24.82 (18 hours)
without node-based implementation
CHTss
28.92
21.88
without soft sampling
CHTss
30.35
21.60
without sigmoid decay (= CHTs)
CHTss
32.79 (0.25 hours) 24.57 (1.5 hours)
(full model)
Table 12: Performance comparison of CHTss and CHTss at 90% sparsity across different removal methods. The tested dataset is Multi30K, and the reported metric is BLEU, which is the higher the better.
Remove Method
CHTs
CHTss
set wm wm__sof sof_soft
28.82
25.76
28.17
31.15
30.35
32.79
ri ri_soft
28.91
32.20
27.86
31.86
MEST
28.70
32.07
snip sensitivity
28.23
31.66
29.02
29.73
Rsensitivity
28.18
30.67
training progresses. Figure shows the variation in BLEU scores as we change the starting and ending values of the 8 parameter in the soft weight magnitude removal method on transformer models. Recalling that we define the temperature by T = T= Multi30k, a high starting temperature produces better performance. This is motivated by the fact that loss decreases very fast through epochs, meaning that weights learned quickly, and we can deterministically remove weights with high reliability. In more complex datasets, like IWSLT, low starting temperatures are preferred. This is because during the early stages of training, weights are learned slowly, meaning that deterministic removal can be less reliable. To be more consistent, we select start 8 0.5 and end 8 0.9 for all the tasks in the main article.
T we observe that for simple benchmark like
Density Decay Strategy. To further demonstrate the benefit of the density decay strategy, we conduct an ablation study comparing the sigmoid-based density decay strategy with cubic-based density decay strategy. We evaluate the performance on the LLaMA-60M model using the C4 dataset and report the perplexity. We show the number of density decay steps and a comparison between the cubic density decay that GraNet [40] and GMP [1]] introduced and our proposed sigmoid density decay strategy. The results in the Table B show that the sigmoid- based decay consistently achieves lower PPI. than the cubic-based decay in all the density decay steps, and both strategies outperform the baseline fixed sparsity (CHTss) These results confirm the motivation for introducing the sigmoid density decay strategy. The table below shows the PPL of CHT'ss with the two density decay strategies over different density decay steps.
Curvature of the Density Decay. In CHTss, the parameter is dictates the pruning schedule's shape: lower values yield smoother decay, while higher values create a sharper curve. We varied k in 2, 4, 6. 8, 10 on LLaMA60M at 70% and 95% sparsity, using perplexity as the evaluation metric. Results on Table 14 show that overly sharp decay curves (larger k) cause instability. the article, we used & = 6 for all experiments.
30

---

Table 13: Comparison of CHTss variants (Sigmoid- vs. Cubic-based) under different density decay steps. Baseline (CHTs) achieves 35.59. Lower values indicate better performance.
Density Decay Steps
1000
2000
3000
4000
5000
6000
7000
8000
9000
CHTss (Sageoid-based)
35.20
35.19
35.24
34.63
34.68
34.63
34,46
34.50
34.39
CHTss (Cubie-based)
35.23
35.21
35.29
35.03
34.96
39.23
34.96
34.79
34.94
Table 14: Perplexity comparison of LLaMA60M under different initial sparsity and lt values at 70%
and 95% sparsity levels. Lower is better.
LLaMA60M (70% % sparsity)
k=2.0
k=4.0
k=6.0
k=8.0
1=100.0
27.61
27.74
27.74
27.82
29.79
0.2
27.60
28.81
27.81
29.82
34.05
0.3
27.66
27.67
27.78
27.59
27.58
0.4
27.93
27.82
27.78
27.73
27.71
0.5
28.05
27.82
27.62
27.83
28.13
LLaMA6UM (95% % sparsity)
k=2.0
4.0.0
k=6.0
8=8.0
k= 10.0
0.1
37.00
36.02
35,86
36.42
36.59
0.2
36.83
36.08
35.42
38.39
35.87
0.3
36.67
35.30
35.53
35.67
35.78
0.4
35.71
35.48
35.42
35.44
35.63
0.5
36.76
35.70
35.59
35.72
36.05
Removal Fraction Ç. This parameter determines the proportion of existing connections that are pruned during each pruning-regrowth cycle. It directly controls the amount of structural change introduced to the network at each step. our sensitivity analysis on Table T3] we evaluated a range of ( values 10.1, 0.2, 0.3, 0.4, 0.5) to assess their effect on the convergence and final performance of the CHTs model. The tests were performed on MLPs (CIFAR-10) at 90%, 95%, and 99% sparsities. Results are averaged over three seeds. When the sparsity is lower, we need lower ( to keep the model trained more stably, and when the sparsity is higher, a higher Ç is required to encourage the topological exploration. In all experiments involving MLP and Transformer, we use Ç of 0.3, and for LLaMA models, we use 0.1 for all the experiments.
CHTs CHTss. In most experiments, CHT's (with the sigmod density decay schedule) con- sistently outperforms CHT's. There are, however, a few counter-examples (e.g.. LLLaM--BB at 70% target sparsity on OpenWebText) where CHTs slightly surpasses CHTss. To understand this outlier behavior, we first note that in the LLaMA-IB Open WebText run the schedule only decreased density from 50% 30%-a small change relative the fixed 50% baseline-yielding very similar perplexities (14.66 vs. 15.15). When we extended the schedule to a much lower final density 5%, the advantage of CHTss became clear: CHTss reached 16.51 PPL versus 18.93 for CHTs. This aligns with our ITOP analysis: at 30% density, over 90% of candidate links appear during the evolution of both methods, but 5% density CHTs touches only ~20% of links while CHTss still explores -90%. In other words, density-decal confers a broader search over plausible connections precisely where search matters most-at high sparsity. We further corroborated this with cross-scale study on six I.LaMA sizes (20M(-)1B) B) at 70%, 90% and 95% sparsity on Table 16) averaged validation perplexity differences are not significant at 70% (Wilcoxon (p=0.688)). but become significant in favor of CHTss at 90% and 95% (p=0.031 for both). Taken together, these results indicate that occasional wins of CHTs arise in regimes where the decay shallow and the effective search space is already well covered; they do not suggest that the sigmoid density decay strategy fails larger models or particular tasks. Rather, its benefit scales with the target sparsity: the more aggressive the final sparsity, the more CHTss helps by exposing and testing larger set of candidate links during training.
31

---

CHT on Multi0k Top Average Performence by Sparsity and Model
CHTBs MultB0k Tap Average Performance by Sparsity and Model
33.25
BFU
32.75
32,50
32,00
29.3
24
31.50
28.5
0.7
0.8
09
0.95
07
0.3
055
Sparsit
Sparsity
Figure 9: Top average BLEU for WS (grey) and BRF initialization methods the Multi30k translation dataset of CHTs (left) and CHTss (right, with sigmoid decay) at different sparsity levels. Error bars denote the standard error across three seeds.
CHTs on IWSLT Tap Average Performance by Sparsity and Model
CHTss on ISST "ap Average Perfarmance by Sparsity and Model
Network Initalitation
Retwork Initalizztion
22.5
BAF
25,25
25.00
BLFU
21.0
24.50
24.23
25.73
0.9
0.95
0.6
5.05
Spansity
Sparsty
Figure Top average BLEU for WS (grey) and BRF initialization methods on the IWSLT translation dataset of CH'Ts (left) and CHT's (right, with sigmoid decay) at different sparsity levels. Error bars denote the standard error across two seeds.
OFF
Figure 11: Win rates of BRF against WS over CHT's and CHTss models on different datasets
(Multi30k and IWSLT) and different sparsities (0.9 and 0.95 for IWSLT and 0.7, 0.8, 0.9, 0.95 for Multi30k).
32

---

Table 15:
Test aceuracy (%) on CIFAR-10 under varying sparsity ratios and values. Mean + standard deviation over multiple runs. Bold indicates best per column.
CIFAR-10 (90%)
CIFAR-10 (95%)
CIFAR-10 (99%)
68.73 + 0.27
69.28 + 0.08
63.45 = 0.04
0.2
68.63 + 0.10
69.78 + 0.09
67.30 = 0.21
0.3
68.47 + 0.10
69.37 + 0.10
68.08 = 0.02
0.4
67.90 + 0.10
69.01 + 0.08
67.83 0.19
0.5
67.47 + 0.14
68.56 + 0.15
67.24 0.14
Table 16: CHTs vs. CHTsss across sparsity levels (validation perplexity: lower is better).
Sparsity
Mean CHTs +
Mean CHTss + p-value
Significance
0.70
23.81
23.88
0.688
Not significant
0.90
28.47
26.51
0.031
Significant
0.95
31.89
29.19
0.031
Significant
I Integral of Sigmoid Density Decay and Cubic Density Decay
In this section, show the formula for the proposed Sigmoid Density Decay and the Cubic decay implemented in GraNet 40) and GMP [54]. For the Cubic function, it is formulated as:
(1--~ to nAI
f
= (5: -8)
where I E {lo, to + .l, to + n. 8, is the initial sparsity, 8y is the target sparsity, to is the starting epoch of gradual pruning, 4 the end epoch of gradual pruning, and Al is the pruning frequency.
Since our work focuses MIP, Transformer, and LLMs, where FLOPs are linearly related to the density of the linear layers, the FLOPs of the whole training process are linearly related to the integral of the density function across the training time. The integral of the cubic decay function from to tz
ap Sparsity)
322
-20
-214
21.10
20
28:33
30.00
3111
3C.0
13.35
21.57
23
21.00
-220
09
25.03
15.55
29/79
23
06
65
Figure 12: Investigating the level of randomness in link removal strategies. Top BLEU scores of the transformer model using CHTss with weight magnitude soft removal strategy, as the initial and final values of 8 take values in (0.1, 0.25, 0.5, 0.75, 0.8).
33

---

1s:
1'' (st 08) (1 t 1 to
dl
ndi.
(10)
(8i 85) - 10).
For the sigmoid decrease, the integral is:
5 to
1
(s 83)
dt
-k( 16(11- 99t00
(11)
1 1
(si 8g) 10)
To maintain consistency in the computational cost (FLOPs) during training compared the cubic decay strategy, we reduce the number of steps in the sigmoid-based gradual density decrease by half,
J Historical weights
Inspired by GMP [54 A1), we incorporate historical weights into CHTs and CHTss implementa- tion. During training. we maintain a historical weight matrix that records previously learned weights throughout the training process. When CHTs and CHTss predict new links, initialize them using their corresponding historical weights specifically, the values they held before being pruned. this way, CHTs and CHTss enable weight recovery with preserved memory, allowing the model to retain valuable prior information.
K Pseudo code of node-based CH link predictor
We present the pscudocode for the node-based CH2-L3 regrowth method in Algorithm I which comprises three main steps.
Step 1: We compute the sets of path length-2 (L2) neighbors from nodes U to U and from V to V. The computational complexity of this step
Om (dm + (dmda))s
where (dmdy) denotes the average product of degrees dm and d, across all relevant node pairs. In the case of an ultra-sparse network (e.g., with average degree close
this simplifies O + n).
Step We treat each destination node along the UU and VV paths as a potential common neighbor and compute the CH2-L3 score according to Equation (3) in the main text. This step has time complexity of
((m² + n
Step 3: We aggregate the CH2-L33 scores across all hop-3 nodes. This step requires
Ommn (dm) + nm (dn))
time. Under the ultra-sparse assumption, this further reduces O + nm).
Combining all steps, the overall time complexity the node-based CH2-L3 regrowth procedure is
O(mn (dm)) + nm (d,)).
Since all operations can be implemented in matrix-wist wise manner leveraging GPU acceleration assuming all matrices as dense, the total time complexity becomes
Onm + mn²).
34

---

Algorithm CH2-L3n
Require: Binary bipartite adjacency matrix Auv E {0, 1 Ensure: Score matrix ""*x" with S|i > 0 only if Auv [i.]]
U-to-V edges
function CH2-L3N(AUv)
2:
Avu +- Atv
V - edges
dy - row-suml A
4: dy + row-sum (Avu)
Step 1: two-step paths inside each partition > U -
5:
UU + Ave
VV Ave
-V-U-V
> Step 2: CH2-L3n preparatory scores
7:
init evU t 0mxm: EVV Opxn for - to m do for j est t du [i] U. -1
8:
9:
m s.t. j * and JU[i.J] > do
10:
U +1
11:
evo e0o
# According to Equation (3)
ext + 1
12:
end for end for
14:
for
f- 1 to n do
for b to s.t. b # a and VV (a, bj > do
16:
est - lb - VV [a. b)
evvla. bl + VV bj
17:
# According Equation (3)
est +
18:
end for end for
19:
>. Step final CH2-L3n scores
20:
init S 0m xn
21:
for < to m do
22:
for 1 to n s.l. Auv (i, a] 0 do
23:
Suv + Σ evu [i,]] Auv Li, a]
=
24:
Svu + Σ evvla,, b] lv
b-
25:
S/i. a) + Suv + Svv
26:
end for
27:
end for
28:
return S
29:
end function
L Extra results of LLaMAlb
Language modeling. We present comparison of CH'Ts, CHTss, and fully connected network on language modeling tasks using the LLaMA- 1B model Table The results clearly demonstrate that CHTs outperform the fully connected (FC) baseline at 70%, even at high sparsity 95%, CHTss achieves perplexity of 16.51, which is remarkably close to the FC baseline.
Zero-shot performance. We compare CHTs, CHTss, and fully connected (FC) network on zero-shot benchmarks, summarized in Table 20 At 90% and 95% sparsity, CHTs often surpasses the FC baseline, suggesting improved generalization to unseen data. At 70% sparsity, CHTs is comparable to FC, whereas at higher sparsity (90-95%) it exceed FC in accuracy. This pattern appears at odds with perplexity trends, so we provide additional analysis for readers evaluating pretrained models GLUUESSpprGGL.E in zero-shot setting.
key confound label imbalance in several tasks; some datasets contain markedly skewed posi- tive/negative distributions. In practice, observed that models sometimes emit nearly the same
35

---

Table 17: Validation perplexity of different dynamic sparse training (DST) methods on Open- Web using LLaMA-1B across varying sparsity levels. Lower perplexity corresponds to better model performance. The performances that surpass the fully connected model are marked with "*"
Sparsity
0.7
0.9
0.95
FC
14.62
CHTs
14.53*
17.14
18.93
CHTss
15.15
15.62
16.51
Table 18: Label imbalance in selected GLUE//upprGLLU tasks (positive vs. negative counts and proportions).
Dataset
Pos (count, %)
Neg (count, %)
CoLA
721 (69.13%)
322 (30.87%)
MRPC
279 (68.38%)
129 (31.62%)
QQP
14,885 (36.829) (6.)
25,545 (63.
BoolQ
2,033 (62.17%)
1,237 (37.83%)
label for most inputs, yielding deceptively strong accuracy if that label matches the dataset majority class. Table 18 reports the class ratios for representative tasks.
To mitigate this issue, we report Matthews correlation coefficient (MCC) instead of accuracy. MCC robust to class imbalance and better reflects the quality of binary predictions. Table T9 shows averaged MCC over GLUE/SuperGLUE under multiple sparsity regimes.
The 70% sparse models (MCC 0.028 for CHTs; 0.033 for CHTss) are on par with the dense FC
baseline (MCC 0.031), but MCC declines as sparsity increases; at 95% sparsity, negative MCC indicates predictions are anticorrelated with ground truth (worse than random). We attribute part of this degradation to compute- compute-limited limited pretraining budgets (e.g., up to 8.9B tokens for LLaMA-1B in our runs), which are insufficient to reach the regime where shott performance is reliable at extreme sparsity. Recommendation: for zero-shot evaluation on imbalanced datasets, prefer MCC (or similarly balanced metrics) over raw accuracy, especially when comparing across sparsity levels and training budgets.
M
Extra Related Works
More recently, BiDST has elegantly reframed DST as a bi-level optimization problem to co-optimize the weights and the sparsity mask simultaneously [57]. Other methods improve the training dynamics itself, such as AC/DC, which alternates between sparse and dense phases (58], or Powerpropagation, which introduces in the often-chaotic loss landscape sparse models, S2-SAM provides an efficient, plug-and-pll and--ladyyyyyy sharpness-aware optimizer [60). Another thrust of research adapts DST for specific architectural or hardware advantages. For example, SLaK utilizes dynamic sparsity to enable massive, high- performance kernels in CNNs [6T]. CHASE provides a practical method translate unstructured dynamic sparsity into hardware-Fie friinddl,, channel-level sparsity (62). Addressing network health, NEURREV proposes mechanism to revitalize "dormant neurons" that can emerge during training
manduityin weight reparameterization [59]. To enhance generalization
63
N
Experiments compute resources
All experiments were conducted on NVIDIA A100 80GB GPUs. ML. and Transformer models were trained using single GPU, while LLaMA models were trained using eight GPUs in parallel.
36

---

Table 19: Average MCC on zero-shot GLUE/SuperGLLUE. FC denotes the dense fully connected
model.
Metric
FC
70% CHTs
70% CHTss
90% CHTs
90% CHTss
95% CHTs
95% CHTss
AVG MCC
0.031
0.028
0.033
0.031
0.022
-0.003
-0.007
Table 20: Zero-shot evaluation of LLaMA-IB across GLUE and SuperGLUE. ACC scores are shown. Values are mean sd over 5 seeds (Im-eval 56 Red values indicate the top scores over models and sparsities for benchmark.
70 %6 sparsity
90 % sparsity
95 % sparsity
Dataset
FC
CHTs
CH
CHTs
CHTss
CHTs
CHTss
CoLA
40.27 + 1.52
38.83 + 1.51
44.20 + 1.54
50.53 1.55
67.88 1.45
68.84 1.43
31.83 +
MNLI
32.89
32.77 +
32.65
32.83
32.73
32.91
MRPC
39,95
40.44 2.43
36.03 2.38
66.42 2.34
38.48 2.41
67.65 2.32
46.81
2.47
QNLI
0.668
50.96 - 0.68
49.70
50.36
49.79
49.26
51.42 = 0.68
QQP
47.94
43.40 + 0.25
49.12
50.57
4.22
3.292
41.70 + 0.25
RTE
47.29 3.01
46.93 + 3.00
46.57
52.71
52.35
49.46 + 3.01
SST-2
65.71
52.98 =
49.66
49.08 1.69
1.621
1.699
49.08 +
WNLI
5..70
5.98
5.97
5.93
5..70
5.970
40.85 5.88
Hellaswag
28.98 0.45
28.75 0.45
28.70
2.45
28.13
0.445
27.53 + 0.45
Boolq
41.07 +: 0.86
50.89 + 0.87
48.62 0.87
44.59 4: 0.87
47.46 0.87
0.87 55.38
52.14 à: 0.87
CB
48.21 # 6.74
50.00 = 6.74
50.00 - 6.74
50.00 2 6.74
50.00 : 6.74
37.50
6.53
48.21 6.74
Copa
65.00 + 4.79
62.00 =
64.00 : 4.82
66.00 4.76
63.00 4.85
4.92 60.00
60.00 + 4.92
Average
46.50
45.60
46.38
49.24
48.53
49.47
44.31
37


Reviews:
[{"review_id": "NVGAM0DfGV", "reviewer": "Reviewer_tKGL", "summary": "This study introduces brain-inspired methods to enhance dynamic sparse training (DST) in artificial neural networks (ANNs), addressing limitations in existing approaches like the Cannistraci-Hebb training (CHT). The proposed bipartite receptive field (BRF) initializes sparse connectivity, while a GPU-friendly CHT approximation reduces computational complexity. The CHT soft rule (CHTs) improves link sampling flexibility, and a sigmoid-based gradual density decay (CHTss) further optimizes sparsity. Experiments demonstrate that BRF outperforms prior models, CHTs achieves better accuracy than dense networks with just 1% connectivity in MLPs, and CHTss excels in Transformer-based translation tasks at 5% sparsity. At 30% connectivity, both methods surpass other DST techniques in language modeling and zero-shot tasks, proving efficient and scalable for ultra-sparse ANNs.", "questions": "See Weaknesses.\n1. The paper acknowledges that hardware support for unstructured sparsity is not yet widespread, which limits the immediate practical deployment of the method. This could delay real-world adoption.\n2. The method introduces several hyperparameters. While some sensitivity analysis is provided, it is not exhaustive. The optimal settings might vary across tasks, which could complicate deployment.\n3. The paper claims full reproducibility, but some results are based on limited runs due to resource constraints. This could affect the reliability of the reported improvements.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel brain-inspired method (CHTss) for dynamic sparse training (DST) that leverages the Cannistraci-Hebb theory, offering a unique perspective on sparse neural network training. The integration of topological principles from neuroscience is a significant contribution.\n2. The proposed method achieves impressive results, outperforming fully connected networks in various tasks (e.g., MLPs at 99% sparsity, Transformers at 5% density). The empirical results are well-documented and supported by extensive experiments across multiple datasets and architectures.\n3. The paper addresses the high computational complexity of previous methods \n4. The study evaluates the method on diverse tasks (image classification, machine translation, language modeling) and architectures (MLPs, Transformers, LLaMA), demonstrating broad applicability.\n\nWeaknesses:\n1. The paper acknowledges that hardware support for unstructured sparsity is not yet widespread, which limits the immediate practical deployment of the method. This could delay real-world adoption.\n2. The method introduces several hyperparameters. While some sensitivity analysis is provided, it is not exhaustive. The optimal settings might vary across tasks, which could complicate deployment.\n3. The paper claims full reproducibility, but some results are based on limited runs due to resource constraints. This could affect the reliability of the reported improvements.", "was_revised": true, "final_justification": "I will maintain the positive score.", "rebuttal": "We sincerely thank the Reviewer for the thoughtful and encouraging review of our work, particularly for recognizing the novelty of our brain-inspired method and the broad applicability across multiple architectures and tasks. We will reply to the weaknesses you raised in detail.\n\n**Weakness 1: Hardware support for unstructured sparsity is not yet widespread. This could delay real-world adoption.**\n\n**Reply:** We appreciate the reviewer’s comment and would like to respond from two perspectives:\n- Our method (CHTs) is ideal for edge AI and robotics, where GPU computing isn't feasible due to hardware and power limits. It generates lightweight, sparse models perfect for micro-drones, micro-robots, and other embodied edge devices, enabling efficient real-time training without GPUs.\n- While current GPUs favour dense computations, new hardware from companies like Cerebras is emerging to efficiently support unstructured sparsity (Lie, Sean. \"Harnessing the Power of Sparsity for Large GPT AI Models.\" Cerebras Systems Blog (2022)). Our method is future-proof, aiming to fully leverage dynamic sparse training and inference as this hardware becomes more accessible.\n\n**Weakness 2: The method introduces several hyperparameters. While some sensitivity analysis is provided, it is not exhaustive. The optimal settings might vary across tasks, which could complicate deployment.**\n\n**Reply:** We fully understand the Reviewer's concern about the task-specific hyperparameter tuning that could probably boost the performance of our methods in each task. To clarify, our experiments are designed with minimal task-specific hyperparameter-tuning. The primary performance gains are achieved through the introduction of:\n- The novel **brain-inspired receptive field (BRF)** topological initialization (Figures 9–11).\n- The **soft sampling strategy** for link removal and regrowth, which balances exploration and exploitation (Figure 3, Table 12).\n- The **sigmoid-based density decay** that broadens the search space of connections, particularly at high sparsity (Tables 2, 3, 11, and 12).\n- The **node-based implementation** that provides a significant runtime speed-up (Figure 2, Table 12).\n\nThese core innovations introduce several additional hyperparameters. However, we found that the model performance remains relatively stable across a reasonable range of values. To substantiate this, we added a new section in the Appendix presenting sensitivity analyses of these hyperparameters. Due to space limitations, we summarize the key findings here:\n\n- Soft sampling factor **δ**. As shown in **Figure 7**, we observe that higher initial temperatures perform better on simple benchmarks, whereas lower initial values are preferable on more complex tasks like IWSLT. Across our LLM experiments, we used a δ starting at 0.5 and ending at 0.9.\n- The BRF initialisation randomness parameter **r**. Results in **Tables 9 and 10** indicate that values of 𝑟 in [0, 0.3] yield quite stable performance.\n- Curvature of the Density Decay Curve **k**. In CHTss, the parameter k dictates the pruning schedule's shape: lower k values yield a smoother decay, while higher values create a sharper curve. We varied k in {2, 4, 6, 8, 10} on LLaMA60M at 70% and 95% sparsity, using perplexity as the evaluation metric. Results show that overly sharp decay curves (larger k) cause instability. In the article, we used k = 6 for all experiments.\n\n**LLaMA60M** (70% sparsity)\n|Initial Sparsity|k=2.0|k=4.0|k=6.0|k=8.0|k=10.0|\n|--:|--:|--:|--:|--:|--:|\n|0.1|**27.61**|27.74|27.74|27.82|29.79|\n|0.2|**27.60**|28.81|27.81|29.82|34.05|\n|0.3|27.66|27.67|27.78|27.59|**27.58**|\n|0.4|27.93|27.82|27.78|27.73|**27.71**|\n|0.5|28.05|27.82|**27.62**|27.83|28.13|\n\n**LLaMA60M** (95% sparsity)\n|Initial Sparsity|k=2.0|k=4.0|k=6.0|k=8.0|k=10.0|\n|--:|--:|--:|--:|--:|--:|\n|0.1|37.00|36.02|**35.86**|36.42|36.59|\n|0.2|36.83|36.08|**35.42**|38.39|35.87|\n|0.3|36.67|**35.30**|35.53|35.67|35.78|\n|0.4|35.71|35.48|**35.42**|35.44|35.63|\n|0.5|36.76|35.70|**35.59**|35.72|36.05|\n\n- Removal Fraction (**ζ**) — This parameter determines the proportion of existing connections that are pruned during each pruning-regrowth cycle. It directly controls the amount of structural change introduced to the network at each step. In our sensitivity analysis, we evaluated a range of ζ values (0.1, 0.2, 0.3, 0.4, 0.5) to assess their effect on the convergence and final performance of the CHTs model. The tests were performed on MLPs (CIFAR-10) at 90%, 95%, and 99% sparsities. Results are averaged over three seeds. When the sparsity is lower, we need a lower **ζ** to keep the model trained more stably, and when the sparsity is higher, a higher **ζ** is required to encourage the topological exploration. In all experiments involving MLP and Transformer, we use a **ζ** of 0.3, and for LLaMA models, we use 0.1 for all the experiments. \n\n|ζ| CIFAR-10 (90%)| CIFAR-10 (95%)| CIFAR-10 (99%)|\n|-:|:-|:-|:-|\n|0.1 | **68.73 ± 0.27** | 69.28 ± 0.08 | 63.45 ± 0.04 |\n|0.2 | 68.63 ± 0.10 | **69.78 ± 0.09** | 67.30 ± 0.21 |\n|0.3 | 68.47 ± 0.10 | 69.37 ± 0.10 | **68.08 ± 0.02** |\n|0.4 | 67.90 ± 0.10 | 69.01 ± 0.08 | 67.83 ± 0.19 |\n|0.5 | 67.47 ± 0.14 | 68.56 ± 0.15 | 67.24 ± 0.14 |\n\n**Weakness 3: The paper claims full reproducibility, but some results are based on limited runs due to resource constraints. This could affect the reliability of the reported improvements.** \n\n**Reply:** We have now added results averaged over three random seeds for all Transformer experiments in **Table 2**. These updated results provide a more reliable estimate of performance and confirm the robustness of our reported improvements. Tests on the CHT model were run on a unique seed due to time limitations.\n| Method | Multi30k (95%) | Multi30k (90%) | IWSLT (95%) | IWSLT (90%) | WMT (95%) | WMT (90%) |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| FC | 31.38 ± 0.38 | | 24.48 ± 0.30 | | 25.49± 0.15 | |\n| SET | 28.99 ± 0.28| 29.73 ± 0.10 | 18.53 ± 0.05 | 20.13 ± 0.08 | 20.19 ± 0.12| 21.61 ± 0.28|\n| RigL| 29.94 ± 0.27| 30.26 ± 0.34 | 20.53 ± 0.21 | 21.52 ± 0.15 | 20.71 ± 0.21| 22.22 ± 0.10|\n| CHT| 27.79| 28.38| 18.59| 19.91| 19.03| 21.08|\n| CHTs| 28.94 ± 0.57| 29.81 ± 0.37 | 21.15 ± 0.10 | 21.92 ± 0.17 | 20.94 ± 0.63 | 22.40 ± 0.06 |\n| MEST$_{EM\\\\&S}$ | 28.89 ± 0.26| 30.04 ± 0.52 | 19.56 ± 0.10 | 21.05 ± 0.21 | 20.70 ± 0.07| 22.22 ± 0.10|\n| GMP| 30.51 ± 0.82| 30.49 ± 0.40 | 22.76 ± 0.82 | 22.82 ± 0.53 | 22.47 ± 0.10| 23.37 ± 0.08|\n| GraNet| 31.31 ± 0.31| 31.62 ± 0.48* | 22.53 ± 0.12 | 22.43 ± 0.09 | 22.51 ±0.21| 23.46 ± 0.09|\n| CHTss| **32.03 ± 0.29*** | **32.86 ± 0.16*** | **24.51 ± 0.02*** | **24.31 ± 0.04** | **23.73 ± 0.43** | **24.61 ± 0.14** |\n\nFor the LLaMA models, due to resource limitations, we are currently unable to complete 3-seed evaluations before the rebuttal deadline. However, we fully commit to providing the 3-seed results for this model before the camera-ready version if the article is accepted. This will ensure that all claims in the paper are supported by statistically consistent results. \n\nIn addition, to enable full reproducibility, we have attached the code with the instructions to run in the submitted supplementary. We will open-source the code if the article is accepted for publication."}, {"review_id": "GKazvAwNnO", "reviewer": "Reviewer_QCkw", "summary": "This paper proposed a new Dynamic Sparsity Training (DST) method called Cannistraci-Hebb training soft rule (CHTs) to overcome the training complexity problem of existing methods, and achieved equivalent even superior results compared to fully connected networks.", "questions": "Questions:\n1. Can CHTss be implemented efficiently on GPU to reduce training time compared to the backpropagation algorithm?\n2. To my understanding, the basic building block of artificial neural networks is fully connected layers, which should only contain the input and output nodes. So why does the author put a network consists of 2 fully connected layers in Fig.1, while it is not widely used in Transformers?\n3. The result of Llama-130M on GLUE dataset seems to underperform older models such as BERT[1]. Why does it happen and does it have an impact of the correctness of the conclusion?\n\n[1] Devlin, Jacob et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” North American Chapter of the Association for Computational Linguistics (2019).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a modification to a DST method and significantly increases the accuracy and training efficiency of the method on various tasks.\n2. The result of the paper suggests that sparse networks may achieve equivalent performance to dense networks.\n\nWeakness: \n1. The abstract of the article is a bit too long and somewhat difficult to understand. For example, the author mentioned bipartite receptive field (BRF) twice, but I’m still confused about the benefit of this module, and in line 26, the author said that “we propose a sigmoid-based gradual density decay strategy”, but it still seems unclear to me what is the motivation for designing such a strategy. I hope that the author could make the abstract shorter and clearer in logic to help more readers understand the core of this paper.\n2. In most of the experiment, CHTss outperforms CHTs, but in the experiment of Llama-130M on SuperGLUE (Table 4) and Llama-1b 70% sparsity (Table 3) on OpenWebtext, CHTs seems to outperform CHTss. Could this mean that sigmoid density decrease is likely to fail on larger networks or some specific tasks? I hope that the author could provide some insights or theoretical analysis on this phenomenon.\n3. The experiment in the supplementary material shows that CHTs performs close to fully connected network at sparsity level 70%, and surpass the fully connected networks when the sparsity level increases to 90% and 95%. This seems counter-intuitive to me as performance generally decreases when sparsity increases. Could the author provide a theoretical explanation for this phenomenon?", "was_revised": true, "final_justification": "Thank you to the authors for their response, which has adequately addressed my concerns. I will adjust my score accordingly. Additionally, I recommend that the authors include their reply to Weakness 3 in the appendix for the benefit of future readers.", "rebuttal": "We thank the Reviewer's insightful suggestions. We could see that the main concern raised by the reviewer is about the zero-shot performance on GLUE and SuperGLUE. We will give our clarification one by one.\n\n**Weakness 1: The unclear Abstract and the motivation for the introduction of the sigmoid density decay.** \n\n**Reply:** Thank you for your constructive suggestions. We have revised the abstract to address your concerns:\n- We removed the sentence in Line 23 that mentioned BRF twice to improve clarity and avoid redundancy.\n- We clarified the motivation behind the sigmoid-based gradual density decay strategy by adding the following sentence: “Additionally, we propose a sigmoid-based gradual density decay strategy to enable a smoother decay during the pruning stage, leading to an enhanced framework referred to as CHTss.”\n- To further demonstrate the benefit of this strategy, we conducted an ablation study comparing the sigmoid-based density decay strategy with a cubic-based density decay strategy. We evaluated the performance on the LLaMA-60M model using the C4 dataset and report the perplexity (PPL; lower is better). We show the number of density decay steps and a comparison between the cubic density decay that GraNet and GMP introduced and our proposed sigmoid density decay strategy. The results in the table below show that **the sigmoid-based decay consistently achieves lower PPL than the cubic-based decay in all the density decay steps**, and **both strategies outperform the baseline fixed sparsity (CHTs)**. These results confirm the motivation for introducing the sigmoid density decay strategy. The table below shows the PPL of CHTss with the two density decay strategies over different density decay steps.\n\nBaseline: CHTs 35.59\n|Density Decay Steps|1000|2000|3000|4000|5000|6000|7000|8000|9000| \n|-|-|-|-|-|-|-|-|-|-|\n|CHTss (Sigmoid-based)|35.20|35.19|35.24|34.63|34.68|34.63|34.46|34.50|34.39|\n|CHTss (Cubic-based)|35.23|35.21|35.29|35.03|34.96|39.23|34.96|34.79|34.94|\n\n**Weakness 2: Why in some cases CHTs outperform CHTss? Could this mean that sigmoid density decrease is likely to fail on larger networks or some specific tasks?**\n\n**Reply:** We thank the reviewer for this insightful question. It is true that in a few cases (e.g., LLaMA-130M on SuperGLUE and LLaMA-1B at 70% sparsity on OpenWebtext), CHTs slightly outperforms CHTss. In the reply of **Weakness 3**, we have discussed the zero-shot performance. Regarding the OpenWebText results:\n- In the LLaMA-1B experiment, the density decay schedule reduced density only from 50% to 30%, which is a relatively small change compared to the fixed 50% sparsity baseline. As a result, the performance difference between CHTs and CHTss in this setup is minimal (14.66 vs. 15.15 PPL). However, when we extended the experiment to a final density of 5%, CHTss achieved a PPL of 16.51, significantly outperforming CHTs (18.93). This suggests that the benefit of CHTss becomes more pronounced at higher sparsity levels by trying more possible links during training. By computing the ITOP$^1$ [1], we find that at 30% density, more than 90% of links appear in the evolution of the models for both CHTs and CHTss, while at 5% density, CHTs only involves around 20%, while CHTss still involves around 90%. This gives an explanation of why the density decay strategy is better when the final density is lower.\n\nTo support our hypothesis, we ran additional experiments on six LLaMA model sizes (20 M → 1 B) for both CHTs and CHTss at 70 %, 90 %, and 95 % sparsity. For each sparsity level we computed the mean validation perplexity (lower is better) and used a two‑sided Wilcoxon signed‑rank test on the six paired values to assess whether the difference between CHTs and CHTss is statistically significant. As the table below shows, CHTss outperforms CHTs significantly at the higher sparsities (90 % and 95 %; p = 0.031 each), whereas at 70 % sparsity the difference is not significant (p = 0.688). We have added this analysis in the **Appendix** to clarify this.\n|Sparsity|Mean CHTs|Mean CHTss|p-value|Significance|\n|-|-|-|-|-|\n|0.7|23.81|23.88|0.688|Not significant|\n|0.9|28.47|26.51|0.031|Significant|\n|0.95|31.89|29.19|0.031|Significant|\n\n**Weakness 3: Why are the zero-shot evaluations of LLaMA1b, 90% sparsity, and 95% sparsity better than fully connected networks and 70% sparsity?** \n\n**Reply:** Thank you for this insightful comment. We agree that the result appears counter-intuitive, this phenomenon is probably caused by the imbalance of the datasets.\n-  In some of the datasets in GLUE and SuperGLUE, the positive and negative labels are very imbalanced. When we printed the output of the answer for each model, we found that they can usually only give the same reply for all the questions. Therefore, the accuracy is computed mainly based on the ratio of the different labels in the dataset.\n|Dataset|Pos (count, %)| Neg (count, %)|\n|-|-|-|\n|cola|721(69.13%)| 322 (30.87%)|\n|mrpc|279 (68.38%)| 129 (31.62%)|\n|qqp|14,885 (36.82%)| 25,545 (63.18%)|\n|boolq|2,033 (62.17%) |1,237 (37.83%)|\n- To address this, instead of using accuracy as the metrics, we switched to the Matthews correlation coefficient (MCC)$^2$, which is a measurement to address the imbalance of the dataset. We have the following results on GLUE and SuperGLUE on zero-shot evaluation:\n|Metric|FC|70% CHTs|70% CHTss|90% CHTs|90% CHTss|95% CHTs|95% CHTss|\n|-|-|-|-|-|-|-|-|\n|AVG of MCC|0.031|0.028|0.033|0.031|0.022|-0.003|-0.007|\n\nThe results show that models with 70% sparsity achieve performance (MCC of 0.028 for CHTs and 0.033 for CHTss) comparable to the dense, fully connected (FC) model (MCC of 0.031). However, performance degrades significantly as sparsity increases. At 95% sparsity, the models yield negative MCC scores. A negative MCC indicates that the models' predictions are often the opposite of the correct answer, performing worse than random guessing.\n\n**Question 1: Can CHTss be implemented efficiently on GPU to reduce training time compared to the backpropagation algorithm?**\n\n**Reply:** Thank you for the question. We would like to clarify that the evolution processes of both CHTs and CHTss are already implemented efficiently on GPU, involving only a few matrix multiplications.\n\nIf the reviewer’s question refers to a comparison of the evolution time with other methods such as GraNet (which uses gradients for regrowth), both of the evolution methods are quite fast. For example, on LLaMA-1B, the average evolution step time for GraNet is 1.92s, while for CHTs and CHTss it is 1.27s. The evolution time of both methods is eligible.\n\n**Question 2: Why does the author put a network consists of 2 fully connected layers in Fig.1, while it is not widely used in Transformers?**\n\n**Reply:** Thank you for the question. We provide three clarifications:\n- One of the key components of CHTs is the **network percolation** step, which is only meaningful when there are multiple layers. This step enables the removal of inactive nodes, allowing the algorithm to induce both unstructured and structured sparsities. Therefore, we intentionally used two layers in Figure 1 to clearly illustrate how this process operates.\n- Although the figure uses two layers for illustration, such structures are in fact widely used in neural networks, including Transformers. For instance, in GPT and Transformer, the Feed-Forward Network (FFN) sublayer consists of two consecutive linear layers. In LLaMA specifically, the gate_proj and up_proj layers operate in parallel and are followed by the down_proj layer.\n- As mentioned in **Appendix F**, to also introduce network percolation in Transformers, we also treat the q and k projections as two continuous layers, the v and o projections as another two continuous layers, and similarly, the FFN layers as consecutive layers for applying the network percolation step. \n\nTo address the reviewer’s concern and avoid any ambiguity, we have added a sentence in the caption of **Figure 1** to clarify this:\n“As explained in Appendix F, the two-layer illustration also reflects how we treat q/k, v/o, and FFN layers as consecutive layers when applying the network percolation step in Transformers.”\n\n**Question 3: Llama-130M underperforms older models such as BERT[1] on GLUE. Why?**\n\n**Reply:** Thank you for raising this point. We believe the observed difference is primarily due to differences in training budgets and evaluation settings and an inherent advantage of BERT model:\n- In our article, the LLaMA-130M model was only pretrained for 40k steps, while in the original BERT paper, the model was pretrained for 1M steps, which is 25× more iterations. This naturally leads to a performance gap.\n- The GLUE results reported in the BERT paper are based on fine-tuned models, while in our work, we report zero-shot evaluation.\n- More importantly, BERT is the bidirectional encoder, so each token attends to the entire sequence—an advantage for classification and other tasks that require global context. In contrast, decoder‑only models such as LLaMA process tokens left‑to‑right, making them naturally stronger at generative tasks like open‑ended text generation.\n\n$^1$: ITOP computes how many links have been involved during the whole dynamic sparse training process.\n\n$^2$: MCC gives a single score between -1 and 1. Towards 1 indicates perfect prediction, towards -1 indicates total disagreement, and 0 means predictions are random guessing.\n\n[1] Do we actually need dense over-parameterization? in-time over-parameterization in sparse training. S Liu, L Yin, DC Mocanu, M Pechenizkiy. ICML 2021"}, {"review_id": "lqAkeuLzj2", "reviewer": "Reviewer_kVtz", "summary": "The paper proposes a brain-inspired sparse training framework that initializes networks with receptive-field topologies, prunes and regrows connections using a gradient-free Cannistraci-Hebb rule, and gradually increases sparsity with a sigmoid schedule. This approach lets MLPs, Transformers, and a 130 M-parameter LLaMA variant retain or surpass dense accuracy while keeping only 1% to 30% of their original links.", "questions": "1. When training language models, up to what sparsity level can the proposed method still outperform the original dense model, or is it impossible to exceed the dense baseline at higher sparsities (over 50%)?\n2. Does the proposed framework also work for convolutional architectures such as VGG and ResNet on standard image tasks?\n3. For large model like LlaMA-1B, what is the cost of constructing and storing the BRF topology—both in terms of GPU memory and wall-clock time?\n4. If the initial BRF sparsity is set too low or too high, does the sigmoid-decay curve need retuning? Could the two components be learned jointly?", "limitations": "Please see weakness and question.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The framework is grounded in Cannistraci-Hebb network science, yielding a principled, gradient-free regrowth rule.\n2. Probabilistic removal and regrowth mitigate overlap between deleted and added links, improving topology exploration and final accuracy.\n3. The proposed framework demonstrates performance across MLPs, sequence-to-sequence Transformers and decoder-only LLMs.\n\nWeakness:\n1. Performance depends on several hyperparameters. For example, the soft-sampling temperature $\\delta$, removal fraction $\\zeta$, BRF randomness $r$, and sigmoid-decay curvature $k$—yet the paper provides little systematic analysis of how results vary with these settings.\n2. The paper contains too many CHI-style notations, which makes it exhausting to read. Figure 1 is overly cluttered and hard to follow, and the subsections in Section 3 are not tightly connected.\n\n3. Some important related works [1][2][3][4][5][6][7] need to be discussed. \n[1] AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks, NeurIPS 2021\n[2] Powerpropagation: A sparsity inducing weight reparameterisation, NeurIPS 2021\n[3] Dynamic sparsity is channel-level sparsity learner, NeurIPS 2023\n[4] More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity, ICLR2023\n[5] NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization, ICLR2024\n[6] Advancing Dynamic Sparse Training by Exploring Optimization Opportunities, ICML2024\n[7] A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training, NeurIPS 2024", "was_revised": true, "final_justification": "The reviewer solved most of the questions. I will raise my score.", "rebuttal": "We sincerely thank the Reviewer for the thoughtful and encouraging feedback. Your positive comments on our work are greatly appreciated. Below, we respond point‑by‑point to the identified weaknesses and questions and describe the additional analyses now included in the revised manuscript.\n\n**Weakness 1: Little systematic analysis of how results vary with different parameter values**\n\n**Reply:** We fully understand the Reviewer's concern about the task-specific hyperparameter tuning that could probably boost the performance of our methods in each task. To clarify, our experiments are designed with minimal task-specific hyperparameter-tuning. The primary performance gains are achieved through the introduction of:\n- The novel **brain-inspired receptive field (BRF)** topological initialization (Figures 9–11).\n- The **soft sampling strategy** for link removal and regrowth, which balances exploration and exploitation (Figure 3, Table 12).\n- The **sigmoid-based density decay** that broadens the search space of connections, particularly at high sparsity (Tables 2, 3, 11, and 12).\n- The **node-based implementation** that provides a significant runtime speed-up (Figure 2, Table 12).\n\nThese core innovations introduce several additional hyperparameters. However, we found that the model performance remains relatively stable across a reasonable range of values. To substantiate this, we added a new section in the Appendix presenting sensitivity analyses of these hyperparameters. Due to space limitations, we summarize the key findings here:\n\n- Soft sampling factor **δ**. As shown in **Figure 7**, we observe that higher initial temperatures perform better on simple benchmarks, whereas lower initial values are preferable on more complex tasks like IWSLT. Across our LLM experiments, we used a δ starting at 0.5 and ending at 0.9.\n- The BRF initialisation randomness parameter **r**. Results in **Tables 9 and 10** indicate that values of 𝑟 in [0, 0.3] yield quite stable performance.\n- Curvature of the Density Decay Curve **k**. In CHTss, the parameter k dictates the pruning schedule's shape: lower k values yield a smoother decay, while higher values create a sharper curve. We varied k in {2, 4, 6, 8, 10} on LLaMA60M at 70% and 95% sparsity, using perplexity as the evaluation metric. Results show that overly sharp decay curves (larger k) cause instability. In the article, we used k = 6 for all experiments.\n\n**LLaMA60M** (70% sparsity)\n|Initial Sparsity|k=2.0|k=4.0|k=6.0|k=8.0|k=10.0|\n|--:|--:|--:|--:|--:|--:|\n|0.1|**27.61**|27.74|27.74|27.82|29.79|\n|0.2|**27.60**|28.81|27.81|29.82|34.05|\n|0.3|27.66|27.67|27.78|27.59|**27.58**|\n|0.4|27.93|27.82|27.78|27.73|**27.71**|\n|0.5|28.05|27.82|**27.62**|27.83|28.13|\n\n**LLaMA60M** (95% sparsity)\n|Initial Sparsity|k=2.0|k=4.0|k=6.0|k=8.0|k=10.0|\n|--:|--:|--:|--:|--:|--:|\n|0.1|37.00|36.02|**35.86**|36.42|36.59|\n|0.2|36.83|36.08|**35.42**|38.39|35.87|\n|0.3|36.67|**35.30**|35.53|35.67|35.78|\n|0.4|35.71|35.48|**35.42**|35.44|35.63|\n|0.5|36.76|35.70|**35.59**|35.72|36.05|\n\n- Removal Fraction (**ζ**) — This parameter determines the proportion of existing connections that are pruned during each pruning-regrowth cycle. It directly controls the amount of structural change introduced to the network at each step. In our sensitivity analysis, we evaluated a range of ζ values (0.1, 0.2, 0.3, 0.4, 0.5) to assess their effect on the convergence and final performance of the CHTs model. The tests were performed on MLPs (CIFAR-10) at 90%, 95%, and 99% sparsities. Results are averaged over three seeds. When the sparsity is lower, we need a lower **ζ** to keep the model trained more stably, and when the sparsity is higher, a higher **ζ** is required to encourage the topological exploration. In all experiments involving MLP and Transformer, we use a **ζ** of 0.3, and for LLaMA models, we use 0.1 for all the experiments. \n\n|ζ| CIFAR-10 (90%)| CIFAR-10 (95%)| CIFAR-10 (99%)|\n|-:|:-|:-|:-|\n|0.1 | **68.73 ± 0.27** | 69.28 ± 0.08 | 63.45 ± 0.04 |\n|0.2 | 68.63 ± 0.10 | **69.78 ± 0.09** | 67.30 ± 0.21 |\n|0.3 | 68.47 ± 0.10 | 69.37 ± 0.10 | **68.08 ± 0.02** |\n|0.4 | 67.90 ± 0.10 | 69.01 ± 0.08 | 67.83 ± 0.19 |\n|0.5 | 67.47 ± 0.14 | 68.56 ± 0.15 | 67.24 ± 0.14 |\n\n**Weakness 2: too many CHI-style notations. Figure 1 is overly cluttered. The subsections in Section 3 are not tightly connected.**\n\n**Reply:** While the term “CHI-style notations” is not entirely clear to us, we interpret the concern as referring to an excessive use of mathematical symbols, abbreviations, or formalism that may hinder readability. In response, we have reduced the number of mathematical notations and abbreviations where possible. For example, we revised the paragraph on ELM in Subsection 3.1 (L141-151) as follows:\n``Epitopological Local Minima occur when a large fraction of the newly added connections are the same ones that were just pruned in the preceding step. Such a cycle of removal and regrowth indicates that the model is trapped, preventing it from effectively exploring more optimal network topologies.\nThis definition is crucial for understanding the challenges of the original Cannistraci-Hebb Training (CHT), where the rate of overlap between removed and regrown links becomes significantly high within just a few epochs, leading to a rapid convergence toward an ELM. Previously, CHT implemented a topological early stop strategy to avoid predicting the same links iteratively. However, it will stop the topological exploration very fast and potentially trap the model within the ELM.\"\n\nAdditionally, we reordered subsections 3.1 and 3.2 to align with the sequential steps of our DST method and increased the font size of the content in Figure 1.\n\n**Weakness 3: Missing related works**\n\n**Reply:** We thank the Reviewer for pointing out these omissions. We have now integrated the suggested references into the Related Works section. Specifically, we added the following paragraph at **L113**:\n``More recently, BiDST has elegantly reframed DST as a bi-level optimization problem to co-optimize the weights and the sparsity mask simultaneously. Other methods improve the training dynamics itself, such as AC/DC, which alternates between sparse and dense phases, or Powerpropagation, which introduces a sparsity-inducing weight reparameterization. To enhance generalization in the often-chaotic loss landscape of sparse models, S2-SAM provides an efficient, plug-and-play sharpness-aware optimizer. Another thrust of research adapts DST for specific architectural or hardware advantages. For example, sLaK utilizes dynamic sparsity to enable massive, high-performance kernels in CNNs. Chase provides a practical method to translate unstructured dynamic sparsity into hardware-friendly, channel-level sparsity. Addressing network health, NeurRev proposes a mechanism to revitalize \"dormant neurons\" that can emerge during training.\"\n\nWe will also ensure these references are included in the final camera-ready version.\n\n**Question 1: At what sparsity level can the proposed method still outperform the dense baseline?**\n\n**Reply:** To address the reviewer's question, we have run additional tests of the CHTs model across two sizes of models, LLaMA models (60M and 1B), with sparsities from 50% to 95%. The table below compares the evaluated models (PPL as evaluation).\n\n**LLaMA60M** (Dense baseline: 26.56)\n|Model|50%|60%|70%|80%|90%|95%|\n|-:|:-|:-|:-|:-|:-|:-|\n|CHTs|27.24|27.60|28.12|29.84|33.03|36.47|\n|CHTss|27.24|27.65|27.62|29.00|31.42|35.10|\n\n**LLaMA1B** (Dense baseline: 14.62)\n|Model|60%|70%|90%|95%|\n|-:|:-|:-|:-|:-|\n|CHTs|14.48* |14.53*|17.14|18.93|\n|CHTss|14.77|15.15|15.62|16.51|\n\nThe performance clearly shows that it is tough for a sparse method to outperform the dense model when the model is small, but on a larger model like LLaMA1B, CHTs outperforms the dense model with sparsity lower than 70%.\n\n**Question 2: Application to Convolutional architectures**\n\n**Reply:** As noted in the Limitation section, we are actively investigating the application of topology-based dynamic sparse training to CNNs. Now we have some preliminary results on MNIST using LeNet. The results of CHTs are close to those of the dense model and show improvements over the non-evolving static method. However, since implementing CHTs on CNNs involves several additional innovations, we may prefer to present these findings in a separate article.\n| |Accuracy|\n|-:|-:|\n|Dense|98.89|\n|50% CHTs|98.75|\n|50% Static sparse|98.45|\n\nTo show that our methods are also able to handle vision tasks, we evaluated ViT-Base on ImageNet, where our method achieved higher accuracy than the dense baseline at up to 80% sparsity:\n\n**Dense ViT-Base: Accuracy 78.5**\n|Sparsity|Accuracy|\n|-:|-:|\n|0.6|79.88*|\n|0.7|79.38*|\n|0.8|79.05*|\n|0.9|77.54|\n|0.95|71.61|\n\nWe will include these results in the final version.\n\n**Question 3: the cost of constructing and storing the BRF topology**\n\n**Reply:** The BRF is generated layer-by-layer and directly merged into the weight matrix to form the sparse weights. Therefore, the memory usage is negligible (<1GB). The generation time per layer (size 2048×2048) is approximately 3 seconds, and for LLaMA1B, the total cost is about 590s (6m).\n\n**Question 4: If the initial BRF sparsity is set too low or too high, does the sigmoid-decay curve need retuning? Could the two components be learned jointly?**\n\n**Reply:** This is a very sound question. We agree that the interplay between the initial sparsity and the sigmoid decay schedule can influence the training dynamics, especially in the early stages. To explore this, please refer to the Table and analysis in **Reply to Weakness 1** that compares different initial sparsities and corresponding values of the curvature parameter **k** used in the sigmoid decay.\n\nCurrently, we do not jointly learn the model’s sparsity and the decay schedule. However, we truly value this insight and will consider joint optimization or learnable scheduling as a direction for future work."}, {"review_id": "H2Utmiqn09", "reviewer": "Reviewer_QgTr", "summary": "This work improves brain-inspired dynamic sparse training (DST) by addressing key limitations of the Cannistraci-Hebb (CH) method. The experiments are comprehensive and the results are promising.", "questions": "NA.", "limitations": "Yes.", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Please note that I am not an expert in this field.**\n\nStrengths\n---\n\n1. Compelling Bio-Inspired Foundation.\nThe motivation that translates brain network principles into dynamic sparse training (DST) is well-grounded. This bio-plausible method presents a credible way to advancing current AI.\n\n2. Rigorous Technical Contributions.\nThe work demonstrates good mathematical and algorithmic innovation, including BRF initialization, GPU-optimized CH approximation, CHTs soft rule, CHTss framework and so on.\n\n3. Extensive and Convincing Validation.\nExperiments cover diverse domains, like vision classification, Transformers, language modeling, across recognized datasets. All results seem to be very promising.\n\n4. Field-Advancing Potential.\nThis paradigm that leverages neural-biological principles, e.g., receptive field, Hebb plasticity, might catalyze a new generation of efficient AI, going beyond gradient-based optimization.\n\nWeaknesses\n---\n\n1. Accessibility for Cross-Disciplinary Audiences.\nDespite being technically rigorous, the manuscript assumes substantial prior knowledge of network neuroscience (e.g., Cannistraci-Hebb rules) and sparse training literature. Non-expert readers may struggle with it.", "was_revised": true, "final_justification": "Just keep my original judgement.", "rebuttal": "We sincerely thank the Reviewer for the thoughtful and encouraging feedback. We are truly grateful for your recognition of the bio-inspired foundation, technical contributions, and the broad validation of our work.\n\nRegarding the noted weakness in accessibility for cross-disciplinary audiences, we fully understand the concern and agree that making the manuscript approachable to non-experts is important. In the revised version, we have created a new section in the Appendix named \"glossary of network science and Cannistraci-Hebb rules\" to introduce some basic notions and concepts of network science and CH rules involved in this article.\n\nWe believe these changes will significantly improve the accessibility of the manuscript for broader audiences. We thank the Reviewer again for recognizing our work."}]

Author rebuttal:

Meta-review: This paper proposes a novel methodology for dynamic sparse training in the context of high sparsity regime. The proposed method was studied and demonstrated on various architectures. It is worth highlighting that it obtains impressive results and competitive performance with dense neural networks training for language modelling, a task and architecture (LLaMA family) which is very scarcely studied in the dynamic sparse training literature. The rebuttal addressed most of the reviewers’ concerns. All four reviewers unanimously recommended acceptance. Addressing the concerns raised in the reviews and including the rebuttal discussions in the final version will improve the paper quality even more.
Decision: Accept (poster)