Title: Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
Venue: NeurIPS 2025
Accepted: yes
Keywords: dynamic sparse training

network science

epitopological Learning

efficient training
Abstract: This study aims to enlarge our current knowledge on the application of brain-inspired network science principles for training artificial neural networks (ANNs) with sparse connectivity. Dynamic sparse training (DST) emulates the synaptic turnover of real brain networks, reducing the computational demands of training and inference in ANNs. However, existing DST methods face difficulties in maintaining peak performance at high connectivity sparsity levels. The Cannistraci-Hebb training (CHT) is a brain-inspired method that is used in DST for growing synaptic connectivity in sparse neural networks. CHT leverages a gradient-free, topology-driven link regrowth mechanism, which has been shown to achieve ultra-sparse (1\% connectivity or lower) advantage across various tasks compared to fully connected networks. Yet, CHT suffers two main drawbacks: (i) its time complexity is $\mathcal{O}(N\cdot d^3)$- N node network size, d node degree - hence it can be efficiently applied only to ultra-sparse networks. (ii) it rigidly selects top link prediction scores, which is inappropriate for the early training epochs, when the network topology presents many unreliable connections. Here, we design the first brain-inspired network model - termed bipartite receptive field (BRF) - to initialize the connectivity of sparse artificial neural networks. Then, we propose a matrix multiplication GPU-friendly approximation of the CH link predictor, which reduces the computational complexity to $\mathcal{O}(N^3)$, enabling a fast implementation of link prediction in large-scale models. Moreover, we introduce the Cannistraci-Hebb training soft rule (CHTs), which adopts a flexible strategy for sampling connections in both link removal and regrowth, balancing the exploration and exploitation of network topology. Additionally, we propose a sigmoid-based gradual density decay strategy, leading to an advanced framework referred to as CHTss. Empirical results show that BRF offers performance advantages over previous network science models. Using 1\% of connections, CHTs outperforms fully connected networks in MLP architectures on visual classification tasks, compressing some networks to less than 30\% of the nodes. Using 5\% of the connections, CHTss outperforms fully connected networks in two Transformer-based machine translation tasks. Finally, with only 30\% of the connections, both CHTs and CHTss achieve superior performance over other dynamic sparse training methods, and perform on par with—or even surpass—their fully connected counterparts in language modeling across various sparsity levels within the LLaMA model family. The code is available at: https://github.com/biomedical-cybernetics/Cannistraci-Hebb-Training-Soft-Rule-.


Peer reviews:
comments: SUMMARY:
This work improves brain-inspired dynamic sparse training (DST) by addressing key limitations of the Cannistraci-Hebb (CH) method. The experiments are comprehensive and the results are promising.

STRENGTHS AND WEAKNESSES:
**Please note that I am not an expert in this field.**

Strengths
---

1. Compelling Bio-Inspired Foundation.
The motivation that translates brain network principles into dynamic sparse training (DST) is well-grounded. This bio-plausible method presents a credible way to advancing current AI.

2. Rigorous Technical Contributions.
The work demonstrates good mathematical and algorithmic innovation, including BRF initialization, GPU-optimized CH approximation, CHTs soft rule, CHTss framework and so on.

3. Extensive and Convincing Validation.
Experiments cover diverse domains, like vision classification, Transformers, language modeling, across recognized datasets. All results seem to be very promising.

4. Field-Advancing Potential.
This paradigm that leverages neural-biological principles, e.g., receptive field, Hebb plasticity, might catalyze a new generation of efficient AI, going beyond gradient-based optimization.

Weaknesses
---

1. Accessibility for Cross-Disciplinary Audiences.
Despite being technically rigorous, the manuscript assumes substantial prior knowledge of network neuroscience (e.g., Cannistraci-Hebb rules) and sparse training literature. Non-expert readers may struggle with it.

QUESTIONS:
NA.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a brain-inspired sparse training framework that initializes networks with receptive-field topologies, prunes and regrows connections using a gradient-free Cannistraci-Hebb rule, and gradually increases sparsity with a sigmoid schedule. This approach lets MLPs, Transformers, and a 130 M-parameter LLaMA variant retain or surpass dense accuracy while keeping only 1% to 30% of their original links.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The framework is grounded in Cannistraci-Hebb network science, yielding a principled, gradient-free regrowth rule.\n2. Probabilistic removal and regrowth mitigate overlap between deleted and added links, improving topology exploration and final accuracy.\n3. The proposed framework demonstrates performance across MLPs, sequence-to-sequence Transformers and decoder-only LLMs.\n\nWeakness:\n1. Performance depends on several hyperparameters. For example, the soft-sampling temperature $\\delta$, removal fraction $\\zeta$, BRF randomness $r$, and sigmoid-decay curvature $k$—yet the paper provides little systematic analysis of how results vary with these settings.\n2. The paper contains too many CHI-style notations, which makes it exhausting to read. Figure 1 is overly cluttered and hard to follow, and the subsections in Section 3 are not tightly connected.\n\n3. Some important related works [1][2][3][4][5][6][7] need to be discussed. \n[1] AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks, NeurIPS 2021\n[2] Powerpropagation: A sparsity inducing weight reparameterisation, NeurIPS 2021\n[3] Dynamic sparsity is channel-level sparsity learner, NeurIPS 2023\n[4] More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity, ICLR2023\n[5] NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization, ICLR2024\n[6] Advancing Dynamic Sparse Training by Exploring Optimization Opportunities, ICML2024\n[7] A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training, NeurIPS 2024\n\nQUESTIONS:\n1. When training language models, up to what sparsity level can the proposed method still outperform the original dense model, or is it impossible to exceed the dense baseline at higher sparsities (over 50%)?\n2. Does the proposed framework also work for convolutional architectures such as VGG and ResNet on standard image tasks?\n3. For large model like LlaMA-1B, what is the cost of constructing and storing the BRF topology—both in terms of GPU memory and wall-clock time?\n4. If the initial BRF sparsity is set too low or too high, does the sigmoid-decay curve need retuning? Could the two components be learned jointly?\n\nLIMITATIONS:\nPlease see weakness and question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new Dynamic Sparsity Training (DST) method called Cannistraci-Hebb training soft rule (CHTs) to overcome the training complexity problem of existing methods, and achieved equivalent even superior results compared to fully connected networks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper proposes a modification to a DST method and significantly increases the accuracy and training efficiency of the method on various tasks.\n2. The result of the paper suggests that sparse networks may achieve equivalent performance to dense networks.\n\nWeakness: \n1. The abstract of the article is a bit too long and somewhat difficult to understand. For example, the author mentioned bipartite receptive field (BRF) twice, but I’m still confused about the benefit of this module, and in line 26, the author said that “we propose a sigmoid-based gradual density decay strategy”, but it still seems unclear to me what is the motivation for designing such a strategy. I hope that the author could make the abstract shorter and clearer in logic to help more readers understand the core of this paper.\n2. In most of the experiment, CHTss outperforms CHTs, but in the experiment of Llama-130M on SuperGLUE (Table 4) and Llama-1b 70% sparsity (Table 3) on OpenWebtext, CHTs seems to outperform CHTss. Could this mean that sigmoid density decrease is likely to fail on larger networks or some specific tasks? I hope that the author could provide some insights or theoretical analysis on this phenomenon.\n3. The experiment in the supplementary material shows that CHTs performs close to fully connected network at sparsity level 70%, and surpass the fully connected networks when the sparsity level increases to 90% and 95%. This seems counter-intuitive to me as performance generally decreases when sparsity increases. Could the author provide a theoretical explanation for this phenomenon?\n\nQUESTIONS:\nQuestions:\n1. Can CHTss be implemented efficiently on GPU to reduce training time compared to the backpropagation algorithm?\n2. To my understanding, the basic building block of artificial neural networks is fully connected layers, which should only contain the input and output nodes. So why does the author put a network consists of 2 fully connected layers in Fig.1, while it is not widely used in Transformers?\n3. The result of Llama-130M on GLUE dataset seems to underperform older models such as BERT[1]. Why does it happen and does it have an impact of the correctness of the conclusion?\n\n[1] Devlin, Jacob et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” North American Chapter of the Association for Computational Linguistics (2019).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces brain-inspired methods to enhance dynamic sparse training (DST) in artificial neural networks (ANNs), addressing limitations in existing approaches like the Cannistraci-Hebb training (CHT). The proposed bipartite receptive field (BRF) initializes sparse connectivity, while a GPU-friendly CHT approximation reduces computational complexity. The CHT soft rule (CHTs) improves link sampling flexibility, and a sigmoid-based gradual density decay (CHTss) further optimizes sparsity. Experiments demonstrate that BRF outperforms prior models, CHTs achieves better accuracy than dense networks with just 1% connectivity in MLPs, and CHTss excels in Transformer-based translation tasks at 5% sparsity. At 30% connectivity, both methods surpass other DST techniques in language modeling and zero-shot tasks, proving efficient and scalable for ultra-sparse ANNs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces a novel brain-inspired method (CHTss) for dynamic sparse training (DST) that leverages the Cannistraci-Hebb theory, offering a unique perspective on sparse neural network training. The integration of topological principles from neuroscience is a significant contribution.\n2. The proposed method achieves impressive results, outperforming fully connected networks in various tasks (e.g., MLPs at 99% sparsity, Transformers at 5% density). The empirical results are well-documented and supported by extensive experiments across multiple datasets and architectures.\n3. The paper addresses the high computational complexity of previous methods \n4. The study evaluates the method on diverse tasks (image classification, machine translation, language modeling) and architectures (MLPs, Transformers, LLaMA), demonstrating broad applicability.\n\nWeaknesses:\n1. The paper acknowledges that hardware support for unstructured sparsity is not yet widespread, which limits the immediate practical deployment of the method. This could delay real-world adoption.\n2. The method introduces several hyperparameters. While some sensitivity analysis is provided, it is not exhaustive. The optimal settings might vary across tasks, which could complicate deployment.\n3. The paper claims full reproducibility, but some results are based on limited runs due to resource constraints. This could affect the reliability of the reported improvements.\n\nQUESTIONS:\nSee Weaknesses.\n1. The paper acknowledges that hardware support for unstructured sparsity is not yet widespread, which limits the immediate practical deployment of the method. This could delay real-world adoption.\n2. The method introduces several hyperparameters. While some sensitivity analysis is provided, it is not exhaustive. The optimal settings might vary across tasks, which could complicate deployment.\n3. The paper claims full reproducibility, but some results are based on limited runs due to resource constraints. This could affect the reliability of the reported improvements.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}