SpikingBrain: Spiking Brain-inspired Large Models
Authors: Yuqi Pan
Yupeng Feng
JingHao Zhuang
siyu ding
Han Xu
Zehao Liu
Bohan Sun
Yuhong Chou
Xuerui Qiu
Anlin Deng
Anjie Hu
Shurong Wang
Peng Zhou
Man Yao
Jibin Wu
jian yang
孙国梁
Bo XU
Guoqi Li
Venue: Accepted by TMLR ()

Abstract: Mainstream Transformer-based large language models (LLMs) face significant efficiency bottlenecks: training computation scales quadratically with sequence length, and inference memory grows linearly. These constraints limit their ability to process long sequences effectively. In addition, building large models on non-NVIDIA computing platforms poses major challenges in achieving stable and efficient training and deployment. To address these issues, we introduce SpikingBrain, a new family of brain-inspired models designed for efficient long-context training and inference. SpikingBrain leverages the MetaX GPU cluster and focuses on three core aspects: (1) Model Architecture: linear and hybrid-linear attention architectures with adaptive spiking neurons; (2) Algorithmic Optimizations: an efficient, conversion-based training pipeline compatible with existing LLMs, along with a dedicated spike coding framework; (3) System Engineering: customized training frameworks, operator libraries, and parallelism strategies tailored to the MetaX hardware. Using these techniques, we develop two models: SpikingBrain-7B, a linear LLM, and SpikingBrain-76B, a hybrid-linear MoE LLM. These models demonstrate the feasibility of large-scale LLM development on non-NVIDIA platforms, and our training framework supports weeks of stable training on hundreds of MetaX GPUs with Model FLOPs Utilization (MFU) at expected levels. SpikingBrain achieves performance comparable to open-source Transformer baselines while using exceptionally low data resources (continual pre-training of approximately 150B tokens). Our models also significantly improve long-context efficiency and deliver inference with (partially) constant memory and event-driven spiking behavior. For example, SpikingBrain-7B achieves more than 100× speedup in Time to First Token (TTFT) for 4M-token sequences. Furthermore, the proposed spiking scheme achieves 69.15% sparsity, enabling low-power operation. Overall, this work demonstrates the potential of brain-inspired mechanisms to drive the next generation of efficient and scalable large model design.

Published in Transactions on Machine Learning Research (03/2026)
SpikingBrain: Spiking Brain-inspired Large Models
Yuqi Pan Yupeng Feng', Jinghao Zhuang', Siyu Ding', Han Xut.4 Zehao Liu Bohan Sun', Yuhong Chou Xuerui Qiu Anlin Deng', Anjie Hu H6. Shurong Wang Peng Zhou', Man Y20. Jibin Wu', Jian Yang! Guoliang Sun Bo Xul.2* & Guoqi Li
Institute of Automation, Chinese Academy of Sciences
Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model Kee Laboratory of Brain Cognition and Brain-inspired Intelligence Technology
"Beijing Academy of Artificial Intelligence 'The Hong Kong Polytechuic University "Zhongguancun Academy "Beihang University $Zhejiang University
LuxxiTech Metea Integrated Circuit Co., Ltd.
Reviewed on OpenReview:
Abstract
Mainstream Transformer-based large language models (LLMs) face significant efficiency bottlenecks: training computation scales quadratically with sequence length, and inference memory grows linearly. These constraints limit their ability to process long sequences effectively. In addition, building large models on non-NVIDIA computing platforms poses major challenges in achieving stable and efficient training and deployment. To address these issues, we introduce SpikingBrain, a new family of brain-inspired models designed for efficient long-context training and inference. SpikingBrain leverages the Metaxp GPU cluster and focuses on three core aspects: i) Model Architecture: linear and hybrid-linear attention efficient, urchitectures with adaptive spiking neurons; ii) Algorithmic Optimizations: conversion-based training pipeline compatible with existing LLMs, along with a dedicated spike coding framework; iii) System Enginecring: customized training frameworks, op- erator libraries, and parallelism strategies tailored to the MetaX hardware. Using these techniques, we develop two models: SpikingBrain-7B, a linear LLM, and SpikingBrain- 76B, a hybrid-linear MoE LLM. These models demonstrate the feasibility of large-scale LLM development on non-NVIDIA platforms, and our training framework supports weeks of stable training hundreds of MetaX GPUs with Model FLOPPs Utilization (MFU) at expected levels. SpikingBrain achieves performance comparable to open-source Transformer basclines while using exceptionally low data resources (continual pre-training of 150B tokens). Our models also significantly improve long- contect efficiency and deliver inference with (partially) constant memory and event-driven spiking behavior. For example, SpikingBrain-77B achieves more than 100x speedup in Time to First Token (TTFT) for 4M-token sequences. Further- more, the proposed spiking scheme achieves 69.15% sparsity, enabling low-power operation. Overall, this work demonstrates the potential of brain-inspired mechanisms to drive the next generation of efficient and scalable large model design.
1
Introduction
Recent advances in large language models (LLMs) built on the Transformer architecture (Vaswani et al.) 2017 have been driven by the scaling law (Kaplan et al. 2020), which suggests that performance improves with larger model sizes and more data (OpenAl 2025 Google Deep 2025 Anthropic 2025). However,
* *Corresponding authors: xubo@in.ac.oo and guoqi liia...i@i
htttps:/ http///wwwwm/tttte www.metxx-tech.oom
2The code for this project publicly available at

---

Published in Transactions on Machine Learning Research (03/2026)
this scale-driven approach comes with significant challenges: extremely high training costs, substantial energy consumption, and complex deployment pipelines. Therefore, achieving high performance and energy efficiency under limited resources has become a critical research goal. To address this, our work draws inspiration from brain mechanisms. We explore novel architecturcs, training paradigms, and spike coding schemes to develop efficient, brain-inspired LLMs that move beyond the traditional Transformer framework.
A further objective is to validate the training and deployment of such models on non-NVIDIA computing clusters. We use open-source Transformer checkpoint (Qwen2.5-7B-bas (Yang et al. 2024b) example) together with our efficient development framework to train and evaluate two models on the MetaX GPU cluster. The models, SpikingBrain-77 and SpikingBrain-70B-A 12B, undergo end-to-end validation, including continual pre-training (CPT), long-context extension (up to 128k tokens), and supervised fine-tuming (SFT). We also adapt the vLLM inference framework to demonstrate deployment feasibility on MetaX hardware.
As illustrated in Figure I SpikingBrain constructs model architectures aligned with brain-inspired mechanisms; these architectures are developed via an efficient. conversion pipeline and supported by adaptations for non- NVIDIA clusters. Our main technical contributions are follows:
Hybrid Linear Architectures: Moving away from quadratic self-attention, we design hybrid models combining linear, local, and standard attention modules. We explore two hybrid strategies: inter-layer sequential (SpikingBrain--B) and intra-layer parallel Baatkingrrain-66--22222
The former achieves linear complexity and excels at long-context efficiency; the latter provides stronger language modeling capability through more sophisticated architectural design. Notably, the modcling of linear attention (Katharopoulos et al. 2020) also resonates with neuronal dynamics in biological (He et al. 2024b).
systems
Inspired by event-driven biological neurons, we propose a spiking scheme that converts activations into integer spike counts and expands them into sparse spike trains. This enables addition-based, event-driven computation. Several encoding formats are supported, including binary {0, 1}, ternary {-1, 0, 1}, and bitwise spike coding. These sparse, event-driven representations provide the basis for our low-power inference design and may also inspire the development of next-generation neuromorphic hardware (Roy et al. 2019 Schuman et al. 2022
Adaptive Threshold Spiking:
Frenkel et al. 2023).
Efficient Model Conversion: i) Attention modules: Using a unified attention map analysis, we convert quadratic attention modules into sparse boddnngwwnndww and low-rank linear attention, by remapping the weights of existing Transformer models (Kasai et al., (021). ii) FFN modules: For SpikingBrain-76B, we utilize an MoE upcycling technique (He et al.] 2024a that replicates dense FFN weights to create sparse experts, increasing capacity with minimal compute and memory overhead. These techniques, together with the dedicated spike coding framework, reduce continual training and inference costs, enabling efficient long-context handling with less than 2% of the compute required for training from scratch.
Large-Scale Training and Inference on MetaX: Both models are trained on hundreds of MetaX C550 GPUs, covering the entire pipeline, from dala preprocessing to distributed training and inference. We adapt frameworks, operators, and communication primitives to ensure stability. This work represents, to our knowledge, the first large-scale training of brain-inspired LLMs on a NVIDIA platform, achieving stable training at 76B parameters.
Our design philosophy holds that linear attention modules exhibit modeling characteristics highly analo- gous to human memory mechanisms, particularly through their use of compressed, continuously updated memory states Gu 2025). In addition, the cuxture-of-Expertt (MoE) component reflects a principle of modular specialization, akin to the distributed and specialized processing observed in biological neural networks (O'Doherty et al. 2021). By combining network-level sparsity (via MoE) with neuron-level spiking sparsity, our approach offers a robust multi-scale efficiency strategy. Taken together, these results point toward a promising direction for developing more efficient and biologically plausible large-model architectures that extend beyond standard transformer-based L.MSs through brain-inspired mechanisms.
2

---

Published in Transactions on Machine Learning Research (03/2026)
SpikingBrain: Spiking Brain-inspired Large Models
Construction
Training
i
Brain-inspired Mechanisms
Model Architecture
Development Pipeline
(Hybrid) Linear Models
V Communicate via Spikes
Open-source Transformer models
Event Driven
Spike Encoding
V Adaptive Sparsity
v Firing Activity Control
Integeo/splke
Conversion- Truining
Need about 150B tokens Long-context applicable
V Modular Sparsification
MoE/FFN
v Functional Specialization
More general
More efficient
Performance comparable
Compressed Memory
Continuously Updated
Train from Scratch
Markov Properties
Hybrid Efficient Attention
Need more than 10T tokens
/ Dendirtic Dynamics
Mostly limited context
Support
Inspire
Adaptation on Non-NVIDIA GPU Clusters
Next- sostl-gnnration Neuromorphic hardware design
Training Framework
Operators
Spiking Sparsity
Resource
Communication Primitives
Parallelism Strategies
100 times 4M Time toae--tt--iitsteeeee co-first-tokat token Speedup
Figure 1: Overview of SpikingBrain. Inspired by brain mechanisms, SpikingBrain integrates hybrid efficient attention, MoE modules, and spike encoding into its architecture, supported by a universal conversion pipeline compatible with the open-source model ecosystem. This enables continual pre-training with less than 2% of the data while achieving performance comparable to mainst open-source models. We further adapt frameworks, operators, parallel strategies, and communication primitives for non-NVIDIA (MetaX) clusters, ensuring stable largo-scale training and inference. SpikingBrain achieves over 100x speedup in TTFT for 4M-token sequences, while spiking delivers over 69% sparsity at the micro level. Combined with macro-level MoF. sparsity, these advances provide valuable guidance for the design of next-generation neuromorphic chips.
SpikingBrain Models
Inspire the design
Computing Platform
Non-NVIDIA GPU (e.g., MetaX)
Neuromorphic Hardware
CPU NVIDIA GPU
Activation Forms
Integer Spike Counts
Sparse Spike Trains
Figure 2: Compatibility of SpikingBrain models across diverse computing platforms. SpikingBrain models can be deployed on CPUs and both NVIDIA and non-NVIDIA GPUs using integer activation formats, also inspiring the design of neuromorphic hardware leveraging event-driven sparse spike representations.
3

---

Published in Transactions on Machine Learning Research (03/2026)
In summary, this work presents efficient brain-inspired LLM development MetaX GPUs, producing two models: SpikingBrain-7B and SpikingBrain-7667 MoF. They achieve (near-)linear complexity, high training stability, and competitive performance through continual pre-training, requiring only 2% of the data typically nceded for training comparable LLMs from scratch. Inference shows substantial speodups, with SpikingBrain-78 reaching over 100x speedup in TTFT (Time to First Token) for 4M-token imputs. We further deploy compressod 1B SpikingBrain model on CPU-based mobile inference framework, achieving a 15.39x speedup sequence length of 256k. The proposed spiking scheme also yields ~ 69% sparsity, offering strong support for reducing power consumption. These findings highlight the potential of brain-inspired design for future neuromorphic hardware and enable efficient model scaling across diverse computing platforms (see Figure 2
2
Model Architecture
2.1
Core Components
2.1.1
Attention Mechanisms
The core temporal interaction module of LLM is the attention mechanism, which projects input tokens into QKV vectors. At each time step, the current query qe interacts with past keys and values k,, v.(8 & t) to produce the output Oμ.
Softmax Attention Standard softmax attention (Vswaani et al. 2017 captures global, token-wist
interactions across the entire sequence:
E expl 1 Jv.
O softmax(QK" O OMV;; o,
(1)
Σ exp(gik.)
Here, M a causal mask where Mij 1 j and M = -00 if j. For a sequence of length n, softmax
attention provides high arithmetic intensity and parallelism during training but incurs O(n²) computation. During inference, maintaining the key-value cache adds an O(n) memory footprint, becoming bottleneck
for long-context tasks.
Sliding Window Attention (SWA) To reduce this quadratic computational cost. many linear-complexity variants have been proposed. One example is SWA (Beltagy et al. 2020 Child et al. 2019 Jiang et al. 2023), which limits the attention scope to a fixed local window of size w, capturing fine-grained local interactions:
0 softmax(QQK O M');; O, Ei exp(qık, \v.
Σ w+ exp(qik.)
Here, M' is windowed causal mask where M' = if + +isj training complexily and inference memory reduce to On)) and 0(1) respectively, as is a fixed constant independent of n.
S otherwise M', is -00. Thus,
Linear Attention Another common variant is linear attention (Katharpoollos et al. 2020 Yang et al.
2024c Dao Gu 2024), which removes the softmax function to achieve linear complexity and can be reformulated as state-based linear recurrence:
(QKT о M)V;
(3)
0 Σ (qkk = qi Σ (k] - = S.S,, where S, = =S + k, Va.
(4)
Here, M a causal mask where M,j if 2 j and M, = if < Linear attention supports chunk-wieg parallelism during training and maintains fixed-size state in recurrent form (Sun et al. 2023: Yang al. 2024ch, reducing computational complexity and inference memory to O(n) and O(1).
4

---

Published in Transactions on Machine Learning Research (03/2026)
Hybrid Attention Each attention mechanism has distinct strengths: softmax attention excels at global retrieval, SWA is efficient for local context, and linear attention effectively compresses longerange information. Hybrid Attention aims to combine these methods to balance efficiency and accuracy, Two common paradigms
arc: i) Inter- lnter-layer sequential hybridization (Lieber et al. 2024 Ren et al.) 2024 De et al. 2024 Blakeman
et al.
2025
Qwen Team 2025
Team et al. 2025), where different attention types are slacked across layers:
attentionl (x;); 0, =attention((o))
ii) Intra-layer parallel hybridization (Dong al. 2024 Zuo et al. 2025). where different attention modules process the same input in parallel and outputs are merged:
o attentionl(x.). of = attention2(x.),
w1 0 + W2 o
(6)
Both approaches are widely adopted, enabling flexible trade-offs between modeling accuracy and efficiency by adjusting the proportions of each attention type. Our SpikingBrain-71 employs inter-layer hybridization of linear attention and SWA, whereas SpiiingBrain-76B uses intra-layer hybridization that combines linear attention, SWA, and full softmax attention.
2.1.2 of (MoE)
The core idea of sparse MoE [Lepikhin et al. 2020 Fedus et al. 2022) is to enhance model capacity by introducing N parallel expert networks into the original ferd-forward network (FFN) layer, while dynamically selecting the most relevant experts for each input token x through a router W,:
p p= (W,,), I (i ps (op-k(p))
(7)
MoE' activation(x)
Σ Pi (x). VE
(8)
Here, each expert E-(.) is FFN. The router produces probability vector p. where is typically softmax or sigmoid function. For cach token, only the top-k experts with the highest probabilities are activated (with I denoting the set of activated experts). The final MoE output is then computed as the weighted sum of these selected experts.
Unlike traditional dense layers, MoE does not activate all experts simultancously. Instead, it leverages sparse activation to significantly increase parameter capacity and expressiveness while keeping computational cost nearly unchanged. This property enables MoE to combine efficiency with high performance in both training and inference. Prior studies (Dai et al. 2024 Yang et al., 2024a: Liu et al. 2024h further suggest that introducing shared experts that are always active, and retaining a few dense FFN layers in shallow stages of the model can improve training stability and overall performance.
In practice, dense models can be efficiently expanded into MoE models using the upcycling technique (Ko- matsuzaki et al., 2022 He et al. (2024a), which scales parameter capacity without sacrificing the original performance. The procedure involves: replicating the dense FFN weights across all experts so that the upcycled model initially matches the dense bascline; ii) rescaling expert outputs appropriately to maintain consistency with the output scale of the original model.
2.1.3 Spiking Neuron Modeling
Spiking neurons are the core components of Spiking Neural Networks (SNNs) (Mass 1997). and their modeling generally falls into two paradigms: simplified approximations and detailed simulations. The Hodgkin -uxley (HH) model (Hodgkin & Huxley 1952) represents the latter, accurately describing the membrane potential dynamics of biological ncurons through a set of nonlincar differential equations. It focuses on simulating the synergistic eflects of transmembrane ion fluxes. Although such models can faithfully replicate fine-scale bioelectrical phenomena of neurons (e.g., complex firing patterns and refractory periods), they entail high computational complexity due to the involvement. of multiple high-order differential operations. Thus, HH-based models are currently mainly suitable for physiological studies on small neuron populations and cannot meet the efficiency and latency requirements of LLMs.
5

---

Published in Transactions on Machine Learning Research (03/2026)
By contrast, simplified models such the Leaky Integrate-and-Fing (LIF) model, are more commonly used in practice (Yao et al. 2023 2024a). The LIF neuron is first-order approximation of soma dynamics incorporating the temporal dimension. For an input token x (extended over time, where X₃ is the input at the f-th time stop), the membrane potential Vi and spike output S, of the ncuron can be formulated as below:
1(1 -s)V, + Vreset 8 X(+1 Ave Vin St + X++1
hard reset soft reset
Vi+1 =
St = if V1 2 Vhh else 0.
Here, the membrane potential accumulates charge in the soma, À represents a decay factor (mimicking the natural leakage of ion potential), where Vth represents a fixed firing threshold, and Vreset the reset potential, which is usually set to 0. While this model simplifies the ion-based mechanisms and retains the core logic of natural neurons, it still exhibits several limitations for large-scale models (Luo al.) 2025 Yao et al. 2025 Fan et al. 2025 i) The temporal dynamics (mainly introduced by the decay factor and reset mechanism) lead to increased training complexity and instability; ii) Even when integrated into pre-trained models, redundant complexity still persists, making it complicated to build models and simulate the original values; iii) The fixed threshold can cause suboptimal spike generation, leading to many neurons becoming silent or aver-activated, posing challenges for optimization and impeding the balance between accuracy and energy efficiency.
To address these limitations, we propose Adaptive-threshole Spiking Neurons, which simplify LIF neurons for enhanced computational efficiency and modeling accuracy, while drawing inspiration from adaptive neuronal thresholding:
Adaptive Threshold: Inspired by adaptive dynamic properties of biological neurons, the firing threshold Vn, is designed as a dynamic value correlated with the membrane potential. This prevents neurons from being over-excited or over-quiescent, maintaining moderately active state from a statistical perspective.
Simplified Temporal Computation: The decay factor is eliminated, and the soft-reset mechanism is adopted, enabling the conversion from contimuous values to integer spike counts in a single step. During optimization, the temporal dimension merged for stability and computational efficiency on GPUs. For inference, the temporal dimension can be re-expanded, and energy-efficient computation can be achieved using sparse event-driven asynchronous hardware Yaao et al. 2024b).
Our modeling preserves essential characteristics of biological neurons and is appropriately simplified to eliminate redundant computations. It not only leverages the energy efficiency advantages of biological neural systems but also holds promise for effective enginecring implementation in pre-trained LLMs when combinod with specific asynchronous hardware.
2.2 Integrated Model Architectures
Through lightweight training, a base Transformer can be converted into efficient attention variants of different forms, alongside MoE upcycling and spiking neuron coding (see Section 3 As a case study, we develop two models from the Qwen2.5-7B-bnn 7B-base checkpoint: SpikingBrain-7BB, a pure linear model optimized for long-context efficiency, and SpikingBrain-76B. a hybrid- MoE model designed to balance efficiency and performance. Their architectures illustrated in Figure 3 Both models are integrated with HuggingFace and vLLM inference frameworks, supporting deployment on single- and multi-GPU settings. Additionally, shown in Figure 2 SpikingBrain models can be deployed across diverse computing platforms, including CPUs as well as both NVIDIA and non-NVIDIA GPUs. Moreover, by expanding integer activations into equivalent sparse spike representations, the models provide insights that can inspire the design of next-generation neuromorphic chips.
SpikingBrain-7B
The 7B model achieves purely lincar complexity by interleaving linear attention and sliding window attention (SWA) layers with a fixed 4K window a 1:1 ratio. The FFN modules adopt the same SwiGLU design as the base model. Spike coding is applied to convert the activations of all linear
6

---

Published in Transactions on Machine Learning Research (03/2026)
Sprase MoF.
SpikingBrain-7B
SpikingBrain-763 (A12B)
Forward Pass
Backward Pass
Experts 0,1,..115
Head
Linear Projections (During Inference)
Top-k
Head
Shared Expert
FFN
Weight
Router
SWA
MoE/FFN
x14
Neuromorphic Hardware
288
GPU
FFN
SWA/FA+LA
Hybrid Attention
LA
Integers
+ Spikes +
Embed
Sink Token
Norm
Norm
Embed
Activation
SWA/FA
LA
Inter-layer Hybrid
Spiking Coding
Intra-layer Hybrid
Figure 3: Integrated architectures of SpikingBrain models. FA: Full Softmax Attention; SWA: Sliding Window Attention; LA: Linear Attention. (Left) SpikingBrain-7B is a linear model with inter-layer hybridization. (Middle) Spike coding converts activations into integer counts for GPU execution or into spike trains for event-driven neuromorphic hardware. (Right) SpikingBrain-76üi is a hybrid-linear MoE model with lntra-layer hybridization, configured with 128 sink tokens, 16 routed experts, and 1 shared expert. Seven dense FFNs are located at layers [1, 2, 3, 5, 7, 9, 11], with all other FFNs implemented as MoE layers. Attention modules are arranged as 'LA FA" at layers [7, 14, 21, 28), and 'LA + SWA' at all other layers.
projection layers in the model, matrix multiplications for parameter projections dominate the model's FL.OPs. In this architecture, the inter-layer hybrid design is optimized for extreme efficiency, owing to its simplicity in attention computation, model parallelism and sequence parallelism. SWA layers captures precise local patterns, while linear attention layers efficiently compresses long-range information. The model thus provides linear-time training complexity and constant memory usage during inference, regardless of sequence length, offering substantial efficiency gains for long-context processing. practice, we implement lincar attention using Gated Linear Attention Module Yang et al. (202cc) (GLA), where the gating vector go, derived from low-rank projection parameters, enhances expressivily and recurrent modeling capability:
S. = diag(g)) @ +
(10)
O, = S,S
(11)
SpikingBrain-76B The 76B model integrates linear attention and SWA in a 1:1 intra-layer parallel hybrid configuration, while standard full-attention layers are interleaved at a 1:6 ratio across layers. This intra-layer hybrid design is optimized for stronger language modeling capabilities, as enables more intricate attention computation and incorporates additional attention parameters and FLOPs. During parallel hy bybidiization, outputs of both attention branches normalized to ensure consistent scale and to prevent instability in the early stages of training. In addition, 128 learnable sink tokens (Xiao et al. 2023 Dong et al. 2024 are introduced to mitigate the attention-sink phenomenon in softmax attention and to enhance the flexibility of SWA for local modeling. Specifically, 128 trainable embeddings are prepended to the input; these lokens are attended by all other tokens and also attend to each other without causal masking. This functionality is implemented by customizing FlashAttention (Dao] 2024) kernels. The model also employs Gated Lincar Attention Modules, but in this case, the gating vector is tied directly to the key vector (Chou et al. 2024 Qin et al. 2024), i.e., g: 1 - k,, eliminating the need for extra gating parameters. The FFN modules adopt
7

---

Published in Transactions on Machine Learning Research (03/2026)
sparse Mixturo-of- Diitturo-of-Epeerts (MoE) design: each MoE layer consists of 16 routed experts (top-1 activated) and one shared expert, so that only about 15% of parameters (12B) are activated per token. To stabilize training and control parameter growth, 7 dense FFN layers are preserved out of 28 total layers (specifically at layers 1, 2, 3, 5, 7, 9, and 11). Similarly, spike coding is applied to the activations of all lincar projection layers shroughout the model.
Connections with Brain Mechanisms Our architectural choices are closely aligned with principles observed biological brains. i) Linear attention modules exhibit modeling properties analogous to human memory, relying on compressed and continuously updated states (Gu 2025). At each step, they retrieve information only from the current memory, showing Markov-like behavior. From biological perspective, their stateful temporal recurrence can be viewed dynamics (Wang et al. 2025). ii) The Soxxturr-of-xxpertt (MoE) component embodies the principle of modular sparse activation and functional specialization, reminiscent of the distributed and specialized processing found in neural circuits f0'Doherty et al. 2021). iif) Our spike coding scheme draws inspiration from event-driven and adaptively sparse neuronal activation in biological systems (Maass) 1997). By combining network-level sparsity (MoE) with neuron-level spiking sparsity, our approach enables on-demand allocation of computation and provides robust (wo-scale efficiency stralegy. Collectively, these results suggest promising pathway for designing large-model architectures that are both efficient and biologically plausible
abstract computational analogy to neuronal dendritic
(Li et al. 2024).
3 Training Paradigm
The development of our SpikingBrain models relies on a highly efficient and universal conversion pipeline that is compatible with the open-source model ecosystem. Specifically, we start from a Transformer base model and perform targeted transformations on three key modules-hybrid attention modules, MoE modules, and spike-coding modules supported by multi-stage training (including CPT and SFT) to adapt the model to the new brain-inspired architectures.
3.1 Attention Map Correspondence
Here, we provide brief analysis of the relationships among different attention maps to illustrate the feasibility of transferring parameters across attention types. First, the attention map in pro-trained Transformers is defined as:
A softmax QK O M) E Ruxn
(12)
On this basis, SWA can be interpreted as a spansified version of this map with a strong recency bias Beltagy
et al. 2020 Child et al.) 2019 Zaheer al., 2020):
A = (QK @ M').
(13)
Moreover, linear attention can be regarded as low-rank approximation of the attention map, where the maximum rank of A is d (the dimensionality of the query/key vectors) (Chen et al. 2021 Wang et al. 2020
A = QK M.
(14)
Leveraging this attention-map correspondence, we can initialize the QKV projection parameters of Spik- ingBrain directly from a pre-trained Transformer checkpoint (Kasai et al. 2021), and this applies to full attention, linear attention, and SWA. By reusing the learned QK similarity and performing lightweight training on a small amount of data, the attention can be adapted to these local or low-rank cases. Since SWA maintains local modeling precision and linear attention retains global interaction, combining them in a hybrid paradigm provides a closer approximation to the original attention map (Dass et al. 2023 Chen et al. 2021 Arora et al. 2024bb. This enables a smoother transition during conversion and accelerates loss convergence.
Finally, due to the generality of this attention-man map formulation, we can construct diverse attention pat- terns sparse, local, linear, hybrid and transfer them from any pre-trained softmax attention model. This
8

---

Published in Transactions on Machine Learning Research (03/2026)
allows for fiexible trade-offs between performance and efficiency by adjusting factors such as the proportion of retained full-attention layers or the window size of local attention.
To ensure stable convergence during the conversion stage, we follow several key practices:
Apply non-negative activation to the QK vectors in linear attention (Kasai et al. 2021 Mercat et al. 2024). such as ReLU in SpikingBrain-7B or Sigmoid in Spiking Because softmax attention maps are inherently non-negative, the converted attention must preserve this property to enable smooth transfer. Normalization, as in softmax attention, is applied after the linear attention outputs.
Keep newly introduced parameters low-rank, such as normalization layers, gating components and sink tokens. In conversion training, the learning rate is typically lower and the dataset smaller than in pre-training, making it difficult to optimizc a large number of randomly initialized parameters. We also want the pre-trained weights lo guide optimization; therefore, we reuse all projection weights in the attention and FFN modules and minimize new parameters.
Perform long-context extension during conversion. Since efficient attention mechanisms scale sub-quadratically and are substantially more efficient on long contexts than the base Transformer, practical approach is restrict the training length and resource usage of the original quadratic model, and then integrate long-contexi extension with continual pre-training during conversion. This allows increasing the context length while maintaining training efficiency, as shown in the conversion pipeline of SpikingBrain (Section 3.4
Fully train the model during conversion to ensure performance, using learning rate warm-up up and either cross-architccture distillation (Wang et al., 2024 Zhang et al.) 2024b) or full-parameter training (Zhang et al. 2024c). For simplicity and to the memory capacity of MetaX GPUs, we adopt full parameeree training without freezing the backbone.
3.2
MoE Upcycling
To efficiently expand a dense model into an MoE architecture, we adopt the upeycling technique [Komatuuzaii
el al. 2022) which increases model capacity while reusing the knowledge encoded in the original parameters. At initialization, the FFN in the base dense model is replicated into N experts, and a randomized router is introduced. The router selects the top-k experts for each token with probability p and outputs their weighted sum, ensuring functional equivalence to the original dense FFN the start:
(x) == Ex == .....- = Ex(x) = FFN(x), at initialization.
(15)
p (W,), I (i pp k(p)) MoF activation(x)
(16)
Σ P Ex(x).
(17)
During training, stochastic routing and data noise gradually break the symmetry among experts, leading differentiated gradients and specialized expert weights. In addition to the N routed experts, our 76B model includes S shared experts (S = 1 in our setting) that are always active for all tokens, helping stabilize the conversion process.
Directly replicating and act ivatingg multiple experts, however, amplifies the output scale. To maintain consistency between MoE and dense FEN outputs, we rescale the expert weights (He et al. 2024a). The relationship between MoE activation OMOE and dense activation adense are derived as below, and the scaling factor a can be defined accordingly:
k N
(S S+ N
k
1
OMoE S adense
X adense
adenses
(18)
S+ N
We apply this factor both shared and routed experts initialization, so that Ex == a FFN holds for all 1 5 i S N at initialization. For Bpiking we use S 1, 1, N = 16, yielding 0.98.

---

Published in Transactions on Machine Learning Research (03/2026)
3.3 Spiking Driven LLMs
Inspired by biological computation mechanisms (event-driven and sparse activation) and aiming to balance performance with efficiency, we propose a dedicated spiking strategy that encodes activations as equivalent integer values and spike sequences. This method can be applied both during and after training, converting the activations of large models into spikes. To further improve energy efficiency, we quantize both model weights and the KV cache to INT8 precision in conjunction with the spiking process. Integrated with SpikingBrain' 's lightweight conversion pipeline, this approach requires no full fine-tuning; a small calibration set is sufficient. to optimize quantization paramcters. For SpikingBrain-77, the entire optimization process takes about 1.5 hours on single GPU with 15 GB memory, significantly reducing deployment. cost. while preserving accuracy.
Our activation spiking scheme follows a decoupled two-step approach: i) Adaptive- spiking during optimization: single-step generation of integer spike counts while maintaining appropriate neuronal firing activity. ii) Spike coding during inference: expansion of spike counts into sparse spike trains over virtual time steps. This approach enables the integer- formulation to support computationally efficient optimization on GPUs, while the expanded spiking formulation provides event-driven, energy-elficient inference when combined with specialized hardware.
3.3.1 Step 1: Adaptive Threshold Spiking
The first step, adaptive-threshold spiking, focuses on the single-step generation of integer spike counts. At this stage, activations are converted using a simplified adaptive-threshold neuron model. The core idea is to design a dynamic threshold that ensures neurons remain statistically -eicher over-excited or wver-quiscent-thheeee avoiding redundant spikes or information loss often caused by fixed thresholds. Specifically, we define:
T St.
X
SINT
(IL mean(abs(x)),
(19)
k
where x denotes the continuous floating- poiitt activations at the projection layer of the large model, equivalent to the accumulated multi-step inputs x; in conventional SNNs. (x(()) is the adaptive threshold determined by the membrane potential, & is a hyperparameter that controls the firing rate, and SINT is the aggregated integer spike count.
We adopt simplified adaptive-threshold spiking neurons to convert continuous activations into integer spike counts. By eliminating the decay factor (i.e., using an IF neuron model) and utilizing the soft-reset mechanism, the inference dynamics can be expressed as:
V++1
Vth St + Xi+1,
St 1 if V. (x( else 0.
(20)
During optimization, the temporal dimension can be collapsed into single-step computation:
VT x, SINT round
(21)
Vx)
This single-step integer formulation enables computationally efficient and stable optimization on GPUs, while adaptive threshold maintains appropriate firing rates. As a result, our spiking scheme preserves the essential characteristics of biological neurons, yet is sufficiently simplified to be practical for large-scale models.
For gradient computation in the optimization phase, since the spiking process is an approximate representation of the original membrane potential, the gradient of the spiking process can be skipped (i.e., using straight- through estimation), which maintains the gradient computation process consistent with conventional training.
Furthermore, controlling firing activity is the primary motivation for adopting adaptive-threshold neurons rather than LIF neurons in our SpikingBrain architecture. The effects of the adaptive threshold and hyperparameter k on firing activity can be summarized as follows:
Threshold dynamics: V(a(x) is determined by the mean absolute membrane potential. When the input patential is high, Via (x) increases, preventing excessive spiking and thus controlling sparsity to
10

---

Published in Transactions on Machine Learning Research (03/2026)
reduce redundant computation. When the input potential is low, Vun(x) decreases, allowing neurons to emit small number of spikes to retain key information and avoid accuracy loss from inactivity. Statistically, membrane potentials often follow long-tailed Gaussian-like distribution with occasional outliers. In this setting, the mean absolute value approximates 0.8 times the standard deviation, providing stable metric for regulating spike activity.
Effect of the hyperparameter k: k directly scales the threshold and thereby determines the distribution of spike counts. A larger & lowers (x), producing higher spike counts SINT and broader firing ranges. This is suited for accuracy-critical scenarios where additional computation is acceptable. A smaller k raises (x)(x), reducing spike counts and producing sparser encodings, which is advantagoous for low-power edge deployments. Tuning k thus provides a flexible trade-off between accuracy and efficiency. This trade-off is empirically evaluated using SpikingBrain-771 (Appendix A.8
Outlier handling: Large models often contain rare but high-magnitude activations (outliers) that significantly affect accuracy. The adaplive threshold, due its statistical basis, robust to these values. Neurons maintain stable activity for the majority of inputs while producing higher spike counts for rare outliers. This behavior resembles the burst response of biological neurons, ensuring that critical information carried by outliers is preserved. On specialized asynchronous hardware, these rare bursts have minimal impact on overall energy efficiency.
(a)
Aritifical Neuron
Neuron with Adaptive Threshold
density
Neuron with Coding Optimization
Dendhie
Dendrie
-V V. Skift in distribution membrance potential density
Dendrii Soma
Soma
Step
Soma
Step
Firing Rale Optimization
Vi Mi changes accordingly
Vis
Adaptive Threshold
Messbrance
Potential
Spiking Count
ensures
range [-7.7]
Time Step Optimization
(b)
Spile
Spike
Binary Spike Coding
Ternary Spike Coding
(Exitation)
(Bidirectional)
Spilk Count
Spike Cownt
(Inhibition)
(Two's Complement)
Spike Count
Spike Cownt
Ternary Spike Coding
Bitwise Spike Coding
Figure 4: Schematic of three spike coding schemes. (a) An adaptive threshold maps membrane potential to spike counts, which expanded over virtual timesteps into sparse spike trains, enabling the conversion from continuous activations to discrete spikes.
Binary: binary uses {0,1} to represent "spike//o-ssiike, while ternary uses 1, to encode both excitatory and inhibitory events. Compared with binary, ternary reduces both timesteps and firing rate by half. (c) Bitwise vs. Ternary: spike counts are unfolded into binary bits across timestops to achieve temporal compression. In high-count scenarios, the required timesteps are far fewer than ternary, leading to significantly higher efficiency.
(b) Ternary
11

---

Published in Transactions on Machine Learning Research (03/2026)
3.3.2 Step 2: Spike Coding
During inference, the integer spike count SINT generated in Step 1 is expanded into a sparse spike sequence with values {-1,0, 1 along the time dimension to support event-driven computation, which formulated as:
T SINT Σ St y= ViL(x)
T
Ws
(22)
where W the weight matrix and y is the output of the linear projection layer. Because each 8, takes values from {0, 10,,, matrix multiplications are replaced by event-driven accumulations, thereby improving computational efficiency.
To expand SINT into multi-step spikes st, we design three encoding schemes tailored to different application needs. The primary goal is to replace dense matrix multiplications with sparse, event-driven additions when supported by appropriate hardware, while optimizing the spike firing rate and mumber of time steps without sacrificing representational capacity. This reduces power consumption and increases computational sparsity.
This the most basic event-driven spike coding method. Each time a unit spike is fired (with value of 1). it represents an activation of the neuron state, and the spike count is accumulated over contimuous time steps. This coding scheme is intuitive and simple, with low computational overhead, making it suitable for scenarios with very low spike counts and effectively reducing system complexity. However, when representing large counts, this coding often requires many time steps to complete the representation. Additionally, the lack of directional optimization for spike firing results in a higher firing rate, further limiting system energy efficiency. Meanwhile, this encoding scheme only supports positive integers (unsigned sInt) which is not applicable to Adaptive Threshold Spiking.
Binary Spike Coding 1):
Ternary Spike Coding (, To improve neuronal expressivity and sparsity in event-driven computations, we introduce ternary spike coding. The core idca is extend the traditional binary spike coding by adding inhibitory spikes (-1), resulting in three firing states: {-1, 1, Compared to binary coding, which can only represent positive values, termary coding offers bidirectional expression capabilities. Here, '1" represents excitatory spikes, '-1' inhibitory spikes, and "0" the silent state, making it more aligned with the "excitation exccitation/nhhiittionn /inhibitiinnn' regulatory mechanism in biological neural systems. As shown in Figure 4 (b), ternary spike coding not only provides symmetric expression capabilities of 11 but also reconstructs the mapping from membrane potential to spike counts through a symmetric quantization strategy. This stralegy maps activations to a symmetric distribution [-k,..., ...... +k] rather than [0, 2, so that amw-ampit counts (e.g., =1) statistically occupy a larger probability mass, effectively absorbing high-frequency large counts from the tail of the original distribution. As a result, this scheme halves the number of time steps and reduces the spike firing rate by more than 2x, significantly improving sparsity and energy efficiency without sacrificing expressivity.
Bitwise Spike Coding: Bitwise coding can be viewed as an event sequence encoding method, where integer count values are expanded bit by bit into spike events over time steps. Each time step corresponds to one binary bit of the count value, significantly compressing the time dimension overhead. As shown in Figure 4 (c), this mechanism supports three implementation forms to accommodate different symbolic representations and precision requirements: i) Pure bitwise encoding, suitable for positive integers, provides extremely high temporal compression in high-count scenarios. For instance, count of 256 requires 256 consecutive time steps in binary coding, 128 in ternary coding, but only 8 steps in 8-bit bitwise encoding. ii) Bidirectional bitwise encoding uses +1 to represent each bit, replacing negative part with -1. This halves the time step emission rate and reduces the required number of time sleps by one (e.g.. 7 steps for a count of 256), while maintaining equivalence in representation. iii) Two's complement encoding incorporates sign information into the highest bit, supporting both positive and negative counts while retaining binary simplicity. Compared to ternary coding, which still requires many time steps for higher bit counts, the bitwise encoding scheme significantly compresses time steps, reducing total spike emissions by up to 8x (for an 8-bit.
12

---

Published in Transactions on Machine Learning Research (03/2026)
count). This effectively reduces overall spike communication overhead and computational load while maintaining precision, making it particularly suitable for high-precision, low-power, and time-sensitive neuromorphic computing tasks. Taking Bidirectional bitwise encoding as an example, its encoding formula and the equivalent computation matrix multiplication can be expressed as:
SINT Σ
2' s,. = Vxn(x)
Σ 2 tm1
(23)
3.3.3 Hardware Adaptation and Deployment Potential
Our spiking scheme can be executed on GPUs. By collapsing the temporal dimension into single step, it avoids the incompatibility between event-driven computation and the synchronous architecture of GPUs, enabling direct simulation and inference validation on general-purpose hardware.
However, the synchronous design of GPUs cannot fully exploit the event-driven and sparse-asynchrononn advantages of spiking signals (Schuman et al.) 2022 Frenkel et al. 2023). GPUs operate under fixed high-frequency clock cycles, unlike biological neural systems that remain idle without spikes and trigger computation only upon activation. To take full advantage of our design and leverage the low-power potential, implementing matrix operations on specialized asynchornous hardware architectures (e.g., neuromorphic chips, spiking processors) is required. These platforms natively respond to sparse spike events without requiring clock synchronization (Roy et al., 2019 Yao et al., 2024b): circuits remain in a quiescent, low-power state in the absence of spikes and perform addition operations only when spikes occur.
This approach maximizes energy efficiency and offers practical pathway for deploying low-power brain- inspired LLMs in edge scenarios such industrial control and mobile devices. It also outlines reference technology roadmap for developing next-generation energy-efficient neuromorphic hardware, supporting the shift of large-scale models from compute-driven to energy-optimized paradigm. Additional discussion of hardware implementation considerations is provided in Appendix
3.4 Multi-stage Conversion Pipeline
The multi-stage conversion pipeline consists of continual pre-training (CPT) with long-contert extension, followed by supervised fine-tuning (SFT). For demonstration purposes, all training data of SpikingBrain are sampled from high-quality open-souree datascts; in practice, domain-specific data can be incorporated for vertical adaptation.
The continual pre-training process comprises three stages which progressively extend the context window while adapting the model to new architectures. In the first stage, our models are trained on 100B tokens with a sequence length of 8k, aiming to transfer attention palterns toward local or low-rank variants and ensure loss convergence. Subsequently, the second and third stages extend the sequence length to 32k and 128k, respectively, cach trained with 20B to 30B tokens. The entire conversion process consumes about 150B tokens in total. Compared to the ~0T tokens typically required for training from scratch, the CPT approach requires only about 2% of the data, enabling efficient adaptation under resource and budget constraints. All three stages use the Matrix (Zhang et al. 2024a) dataset, with long-context data generated through a simple per-dlmmai packing strategy. The RoPE base remains consistent with the base model at 1M.
SFT is conducted in three stages, each employing domaîn-spocific data to progressively enhance the capabilities of the model in general knowledge, dialogues, and reasoning. The first stage focuses primarily on improving fundamental language understanding and domain knowledge, using the Infinity Instruct foundational (Li
et al. mathematical problem solving. This stage is trained with 500k samples under an 8k sequence length. The second stage specializes in dialogue ability and instruction following, utilizing the Infinity Instruct chat (Li et al. 2025) dataset that includes multi-turn conversations, task-oriented dialogues, and knowledge-based question answering. The data volume and sequence length remain consistent with the first stage. The third stage targets reasoning tasks, using a high-quality reasoning dataset (Liu al. [205 Madhusudhan et al. 2025) distilled via the DeepSeek-R1 (Guo et al. 2025 method, containing examples with detailed
2025) dataset covering various foundational topics such scientific knowledge, code interpretation, and
13

---

Published in Transactions on Machine Learning Research (03/2026)
chain-of-thoughtt annotations for mathematical proofs, logical reasoning, case analyses, and other multi-step inference problems. To ensure cross-linguistic reasoning, the dataset maintains a 1:1 Chinese- ratio. A total of 150k samples are used under a sequence length of 8k.
4 Implementation on the MetaX Cluster
Distributed training of train-inspired large models on the MetaX non-NVIDIA cluster poses several challenges. These include: ensuring stability for large-scale parallel training, sustaining intensive communication under long-sequence parallel topologies, and adapting CUDA and Triton operators for hybrid attention. In this work, introduce targeted optimizations address each of these challenges, which enable the successful training of both the SpikingBrainn77B and -76B models. This section details the adaptations for distributed training, operator customization, and the parallel topologies used in our conversion pipeline.
4.1 Distributed Training Adaptation
To enable efficient and stable training on non-NVIDIA GPU clusters, the MetaX software platform has implemented a series of hardware-aware adaptations across multiple dimensions, including MoE' optimization, compitation-commnicttoo overlap, memory optimization, auto-tuning, distributed cluster training, and kernel fusion. Some of these optimizations are designed as plug-in modules, enhancing the flexibility of the training framework.
For MoE training, four strategies are introduced to mitigate memory and computational pressure during the early training stages:
Hot-Cold Expert Optimization: To address communication hotspots caused by uneven to- ken routing in the early phases, frequently accessed experts are replicated locally. This reduces communication overhead until expert load stabilizes, after which the local copies are removed.
Adaptive Recomputation: When a heavily utilized expert processes tokens beyond set threshold, activation recomputation is triggered save memory. This technique is automatically disabled in later stages when the token distribution is balanced.
Multi-Graamuarity Recomputation (Korthikanti et al. 2022): To balance computation and memory under high memory pressure, experts support three recomputation levels: lightweight (activations and router), moderate (including FFN and shared experts), and full (entire MoE layer).
Length Alignment: Variations in token counts per expert affect the efficiency of GEMM. Token dropping and padding are applicd to unify input sequence lengths, improving overall computational efficiency.
To address high communication overhead distributod training, MctaX utilizes SDMA engines for intra-node high-speed data transfer. For lensor parallelism and expert parallelism, communication kernels are fused with compute kernels to reduce scheduling conflicts (NVIDIA 2025). Memory optimizations include fine- ane-ggrined offloading of transformer layer weights or optimizer states to the CPU, well as selective recomputation of normalization layers, activation functions, or individual transformer layers based on memory demand.
For long-sequence parallelism, the MetaX GPU cluster supports efficient multi-GPU and multi-node com- munication with stable training throughput. This alleviates synchronization challenges and improves both GPU memory utilization and compute efficiency. Inter-GPU connectivity enhanced through MetaLink and PCIe 5.0, eliminating intra-node bottlenecks. RDMA over InfiniBand 200/400G RoCE ensures low-latency, bandwihh inter-node communication, meeting the requirements of large-scale distributed training.
The software stack also incorporates auto-tuning and fast recovery mechanisms for large-scale clusters:
Auto- Tanto-T An automated tuning engine covers operators, memory, and communication. It. benchmarks common operators, models communication performance across network topologies, and explores parallel configuration spaces to recommend top-k strategies, minimizing manual effort.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Triton Adapter
CUDA to MACA
Triton Operator
JIT Compilation & Optimization
Call Layer Adaptation
CUDA Operator
Grid Search & Arch. Matching
Optimization Analysis
Cache Structure Specialization
Cache/Arch. Matching
MetaX Compiler
MetaX Library Replacement
MetaX Device METAX
Figure 5: Operator adaptation of SpikingBrain on MetaX GPUs. The adaptation involves two complementary pathways: Triton adaptation and CUDA migration to MACA framework, covering different operator subsets. Together, they form a unified hardware adaptation framework tailored for MetaX GPUs.
Fast. Checkpointing: The DLRover (Wang et al. 2023) Flash Checkpoint technique first. writes training state (model weights, optimizer, learning rate scheduler, etc.) to CPU memory before asynchronously persisting to distributed file system. This reduces I/O time by 85% and shortens recovery time after failures. The built-in profiling tools automatically instrument training jobs, monitor performance per layer, detect slow nodes, and trigger alerts to sustain high cluster utilization.
4.2 Operator Adaptation
The efficient adaptation of SpikingBrain large models to MetaX GPUs relies the comprehensive MetaX software ecosystem. The overall process can be divided into two major components: Triton adaptation and CUDA migration to the MetaX self-developed MACA framework. These two pathways larget different operator subsets in SpikingBrain models. While distinct in implementation, they complement each other and together constitute a complcte hardware adaptation framework tailored for MetaX GPUs (sce Figure
the Triton adaptation workflow, we designed four progressive stages based on the MetaX technology stack and Triton's compilation pipeline. The objective is to fully exploit the MetaX GPU's strengths in compilation optimization and instruction scheduling, while keeping the adaptation process transparent to higher-level applications:
JIT Compilation Optimization: During Triton's just-in-time compilation stage, the linear attention operators (Yang et al. 2024c) used in SpikingBrain are reorganized at the code level. By refining instruction pipelining and register allocation, the operators achieve a balance between memory access latency and computational density. This stage directly leverages Triton's hardwair--gnooticc general optimization strategies, enabling dynamic and efficient execution of compute kernels.
Grid Search and Architecture Matching: Systematic exploration of Block/Gric configurations is carried out and mapped to the scale of MetaX GPU streaming multiprocessors and thread concurrency features. By extending Triton's architectural support to the MetaX GPU family, we significantly improve the throughput of matrix multiplication in linear attention operators.
Cache Structure Specification: To minimize redundant computation and memory traffic, we introduce fixed, structured cache designs aligned with MetaX's on-chip memory hicrarchy. For instance, by optimixing reuse strategies for weight matrices and Key-Value sequences in line with
15

---

Published in Transactions on Machine Learning Research (03/2026)
L2 cache capacity and bandwidth properties, inference efficiency for long-sequence processing is markedly improved.
Target Code Generation via MetaX Compiler: Finally, Triton kernels are transformed by the MetaX compiler backend into executable target machine code for MetaX GPUs. Beyond standard code generation, the compiler introduces deep optimizations for tensor cores, SIMD instruction sets, and memory alignment constraints, ensuring operators run at near-peak hardware performance.
In the CUDA-to-MACA migration workflow, four tightly integrated stages are involved:
Call Layer Adaptation: Original CUDA APIs and runtime interfaces are encapsulated and redirected to the MACA framework, allowing seamless mapping without altering user-facing code. This significantly reduces migration overhead for developers.
Using performance profiling tools, we identify performance bottlenecks in SpikingBrain models, such long-sequence attention operators involving softmax, exp/sum, dot-product accumulation, and certain normalization operations. These operators reimplemented using native MACA optimizations to fully leverage tensor accelcration units.
Optimization Analysis:
Cache and Architecture Matching: Similar to the Triton adaptation process, cache- awarees is embedded into the MACA execution engine. Key data structures (e.g., intermediate accumulation matrices, positional encoding caches) are retained in high-speed cache, reducing global memory traffic and improving efficiency through hardware-specific optimizations.
Replacement with MetaX Acceleration Libraries: Finally, core baseline operators are substi- luted with MetaX's high-performance libraries. The Hotspot operator acceleration library is used to replace foundational calls, while advanced libraries such as mcFlashinfer,, specifically optimized for MetaX hardware, are employed to deliver sustained performance improvements at the hardware level.
It is worth emphasixing that the entire adaplation and migration process for brain-inspired large models on MetaX GPUs adheres to a user-friendly design philosophy. Whether through Triton kernel optimization or CUDA-to-MACA migration, end users can preserve their cxisting programming practices and interface calls. In most cases, no exlensive code modifications are required for the models to run efficiently on MetaX GPUs. Meanwhile, the MetaX ecosystem provides unified debugging and performance analysis tools, enabling developers to transparently monitor execution characteristics on hardware and tune performance needed.
4.3 Parallel Topology
The memory demands of training large language models often exceed the capacity of single GPU. To make training such models feasible, it is essential to employ efficient and scalable distributed training techniques alongside other memory reduction methods. These approaches distribute computational and storage loads while maintaining training efficiency. In this section, we present the four parallelization strategies and corresponding distributed training topologies employed to train SpikingBrain models on the MetaX cluster.
Data Parallelism (DP) Data parallelism Valliatt 1990) involves partitioning the training data into batches, each processed by a separate GPU. Each GPU maintains complete replica of the model and performs forward and backward passes independently, while gradient synchronization occurs only at the beginning of the backward propagation. Data parallelism offers low communication overhead and excellent scalability. As computing resources increase, the scale of data parallelism can grow accordingly, increasing the total batch size to enhance overall throughput. and reduce training time. During training, we employ ZeRO (Rajbhandari et al. 2020) eliminate redundancy of the optimizer states, effectively distributing the GPU memory pressure.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Pipeline Parallelism (PP) Pipeline parallelism divides the model into different stages by layers. Each GPU is assigned subset of layers, and intermediate activations are communicated between GPUs during the forward and backward passes using peer-to-peer (p2p) communication. We employ the 1FIB (Narayaaa et al.) 2021 Harlap et al. 2018) schoduling algorithm for efficient pipeline execution. Furthermore, incorporating Mixture-of-Expert (MoE) layers interspersed with dense FFN layers in SpikingBrain-700 helps mitigale the memory imbalance issue, where the first pipeline stage typically exhibits higher memory usage compared to other stages.
Expert Parallelism (EP) In ofxturre-f-Experte models, most parameters reside within the expert networks. Expert parallelism (Rajbhandari et al.) 2022) addresses this by partitioning experts across multiple GPUs. This requires all-to-all communication to dispatch and combine tokens. During the training of SpikingBrain-76ht we employ Grouped GEMM kernels to accelerate computation across experts, and an auxiliary loss is used to promote balanced token routing across experts.
Sequence Parallelism (SP) Training with long sequences incurs substantial memory costs for storing intermediate activations. Sequence parallelism tackles this by splitting the input sequence along its length dimension, with each GPU processes a contiguous chunk. Communication primitives like all-to-all (a2a) or peer-to-peer (p2p) are then used to exchange necessary information between devices, enabling the computation of attention over the entire sequence. For experiments, we employ DeepSpeed Ulyssss (Jacobs et al.) 2023) for elficient sequence parallelism and incorporate specialized algorithms for linear attention modules to further improve performance. For linear attention, AllGather communication is used to collect intermediate local states (Sun et al. 2025) when the SP size is small, while for larger SP sizes (e.g., across multiple nodes), the All-Scan primitive from ZeCO (Chou et al. 2025 sequence parallelism adopted to effectively eliminate communication overhead.
L Parallel Topology of SpikingBrain The SpikingBrain-7B model trained using the Colossal-AI et al. 2023b) framework. For 128k sequence length training, a parallel strategy of 32-way data parallelism and 8-way sequence parallelism is adopted. To reduce GPU memory consumption, techniques such
ZeRO-2 (Rajbhandari et al.) 2020 and activation recomputation (Chen et al. 2016 Korthikanti et al. 2022) are applied. Sequence parallelism is implemented with DeepSpeed Ulysses (Jacobs et al. 2023), which performs all-to-all communication within nodes. In addition, the number of attention heads is padded to satisfy the partition requirements.
The SpikingBrain-76B model is trained using the Megatron (Shoeybi al. 2019) framework. For 8k sequence length training, the parallel configuration consists of 128-way data parallelism, 8-way expert parallelism, and 4-way pipeline parallelism. ZeRO (Rajbhandari et al.] 2020) and select activation recomputation (Chen) et al. 2016 Korthikanti et al. 2022 are applied to reduce memory consumption. For extended contexts of 32k and 128k, 4-way and 8-way sequence parallelism are added on top of the base 8k parallel strategy, respectively.
The 76B hybrid model integrates linear allention and (local) softmax attention. The max atlention branch uses DeepSpeed Ulysses (Jacobs et al. 2023) with all-to-all communication and head padding to enable partitioning, while the linear attention branch employs the AllGather-based SP algorithm (LASP-2 (Sun)
et al. 2025 to reduce communication overhead.
5
Results
5.1
Downstream Performance
We conduct a comprehensive evaluation of the two converted efficient models, SpikingBrain-771 and -76B, across a broad set of downstream task( 3 They are compared against open-source baselines of similar scale, including Hybrid, Transformer, and MoE architectures, as well as the strong base model Qwen2.5-7B. To ensure fairness, all evaluations are performed under identical settings using the OpenCompass (Buitrago & Nystrom 2019 framework. In selecting evaluation metrics, place greater emphasis on metraining-oriente
3hhis section evaluates the effectiveness the architectural design and conversion pipeline. All models remain floating point format; results for spiking models are presented in Section
17

---

Published in Transactions on Machine Learning Research (03/2026)
benchmarks (e.g., MMLU, C-Eval), as these better indicate whether 200B tokens during efficient conversion can inherit the knowledge and modeling capacity of the base model. Moreover, these benchmarks are less confounded by data quality factors such as alignment performance after SFT. Note that, except for Qwen2.5, the other baselines are trained on limited Chincse data, resulting in disadvanlages on CMMLU and C-Eval. For more information on the baselines and benchmarks, please refer Appendix A
with fewer than
Pre-trained Models As shown Table 1 our SpikingRrain-77h linear model recovers nearly 90% of the base model's performance across benchmarks, reaching level comparable advanced Transformer models such Mistral-7B and Llama3-8B. This indicates that, with appropriate training strategies, linear attention architectures preserve strong general-purpose purpose modeling ability while substantially reducing complexity, However, due to the significant structural modifications of the attention mechanism, there remains certain performance gap between the 7B pure linear model and base Qwen2.5-7B. This suggests that the extent of recovery strongly influenced by architectural changes when pursuing extreme efficiency.
Table 1: Performance evaluation of the SpikingBrain--7B pre-trained model. All models are tested with the HuggingFace framework and evaluated using perplexity-basee based method.
SpakingBrain-
Falcon- Mamba
Qwen2.5 (base)
Mistral
Zamba-v1
Llama3.1
7B
Params
7B
7B
8B
7B
Tokens
+ +150B
5.8T
IT Hybrid
15T
18T
Complexity Type
Linear
Linear
Linear
Quadratic
Quadratic
Benchmarks
MMLU
65.84
63.24
62.56
58.19
65.74
74.21
CMMLU
71.58
42.50
44.58
38.42
52.44
81.73
ARC-C
43.32
47.53
45.13
37.18
51.96
44.04
HellaSwag
70.89
71.50
75.81
52.02
71.60
72.81
Ceval
69.80
41.93
47.04
36.40
51.46
81.60
As shown in Table 2 our SpikingBBainn-66B hybrid-linear MoE model nearly closes the performance gap with its base model by scaling parameter size and adopting more fine-grained attention designs. Despite using fewer activated parameters, it achieves performance comparable to, and in cases surassing, reprosentative Transformer models such Llama2-70B, Mixtral-8x 7B, and Gemma2-27B, Compared with BpininggBainn-7BB the 76B model delivers consistent improvements across all benchmarks, providing strong evidence that within the conversion framework, flexible choices in attention and FFN design allow a better balance between performance and efficiency, while enabling architectures tailored to specific application neods.
Table 2: Performance evaluation of the SpikingBrain-76BB 76B pre-trained model. All models are tested with the vLLM (won et al. 2023) framework and evaluated using a perplexity-based met
SpikingBrain-7700 (76B-A12B)
Jamba
Mixtral-8*7D
Qwen2.5 (base)
LLama2-70b
Gemma2-27B
(52B-A12B)
(47B- A13B)
Params Tokens Complexity Type
12B/76B
12B/52B
13B/47B
70B
27B
7B
+160B
2T
13T
18T
Hybrid
Hybrid
Quadratic
Quadratic
Quadratic
Quadratic
Benchmarks
MMLU
73.58
67.17
71.23
69.57
75.94
75.31
CMMLU
78.83
51.11
52.70
52.94
61.80
81.50
ARC-C
42.00
49.10
48.98
46.21
57.64
43.56
HellaSwag
73.31
79.39
79.42
79.15
79.34
73.37
Ceval
78.89
48.94
54.39
49.26
61.02
81.68
Instruct Models
We conduct a threo-stage SFT process on the converted pre-trained models endow them with instructionnfollowiii and chain-of-thought reasoning abilities. As shown in Table aligned
18

---

Published in Transactions on Machine Learning Research (03/2026)
models achieve performance on par with open-source chat baselines of similar scale across general knowledge, long-sequence modeling, and instruction following. Importantly, the SFT process does not cause overfitting degrade the general capabilities acquired during pretraining, indicating that the (hybrid) linear architectures maintain both stability and scalability under alignment training. Given current data limitations, we primarily aim to remonst effective aligument results obtained on MetaX computing clusters, leaving large-scale and higher-quality post-training alignment as future work.
Long-context Capability To assess real-workd long-context modeling and understanding, we evaluate SpikingBrran--002 (after stage-2 SFT) on six LongBench tasks (Bai et al. 2024). Given the baseline's maximum positional embeddings, we truncate prompts to 32k. As shown Table 5 our model competitive with other hybrid and Transformer baselines while using only four global softmax attention layers. Additionally, LongBench evaluates long-context modeling and understanding rather than recall performance, where linear attention exhibits known limitations. This limitation and its empirical analysis are discussed in Appendix A.4
Inference speed model under SP framework
Inference speed of model
140000
80
15.39x
Speed Ratio (SpikingBraiiii Lammm)))
SplkingBrain-78
Limall 2:38 (swei): 236.2 takemat)
120000
- Qwen2.5-78
104.1x
Decode Speed (tokensis)
50-
100000
TTFT (ms)
80
80000
30-
60000
7.52x
40000
26.5x
20. 10.10
4.04x
20000
2.4xx
3.17
256k
512k
1M
2M
4M
Sequence Length
Output Length (K tokens)
32
128
Figure 6: TTFT comparison under sequence par- allelism. Time to First. Token (TTFT) latency of SpikingBrain-77 7B compared with the Qwen2.5-7B base- line across different input lengths, measured on NVIDIA H100 GPUs For inputs beyond 2M tokens, direct eval- uation of Qwen2.5-7B constrained by resource limits and attention head count; results are therefore extrapo- lated using fitted scaling curve.
Figure 7: Decoding speed comparison at different output lengths on a CPU-based mobile inference framework. Results ported for the IB-parameter SpikingBrain linear model and the Llama3.2-11 baseline, both eval- uated on an Intel Core 15-12600KF CPU, By default, Q4_0 weight quantization is applicd to balance throughput and memory efficiency.
5.2 Long-context Efficiency
Beyond task performance, we also place particular emphasis on the efficiency of our models in long-sequence inference. Benefiting from the integration of (hybrid) linear attention with the MoE mechanism, both SpikingBrain models exhibit substantial inference speedups. Since the 7B model is more compact and shows stronger efficiency gains, we adapt it to the HuggingFace framework with multi-device requence-parally inference (ZeCO + p2p), successfully supporting efficient prefill up to 4M tokens. All measurements reported in Figure 6 and Table 7 are conducted on NVIDIA H100 GPUs and averaged over 10 runs. As shown in Figure and Table 7 at imput length of 1M tokens, the SpikingBrain-77 model achieves 26.5x speedup in TTFT (Time to First Token) compared with the Qwen2.5 baseline using full attention and all-to-all communication. Due to resource limitations and the number of attention heads, Qwen2.5 is difficult to evaluate at 4M tokens. Based on the fitted scaling curve, we conservatively estimate a speedup of over 100x, Moreover, as sequence length and GPU count increase proportionally, the time overhead of our 7B model remains nearly constant. These results demonstrate the scalability and superiority of our approach sarra-long-seueenee scenarios, providing strong support for the deployment of more efficient large language models in practical applications.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Model Weights After Training
Quantization Strategy and Weight Conversion to GGUF
Low-Precision Forward Computation
Produce the Output
Model Registration and Tensor Mapping
Load Quantized Model Weights
CPU-side
Construct and Optimize Computation Graph and Operators
Inference
Pipline
Figure 8: Overview of the CPU-side inference pipeline. The workflow includes four main steps: weight conversion and quantization, model registration and tensor mapping, graph and operator optimization, and quantized inference.
In addition, we perform a systematic evaluation of inference speed for both the 7B and 76B models without sequence parallelism enabled. The experiments are conducted on MetaX C550 GPUs, including HuggingFace BingleGGUU deployment and both single- and multi-GPU deployments using vLLM, As shown in Table R SpikingBrain-7B consistently achieves multiplicative specedd over Qwen2.5 at sequence length of 128k (1.41x HuggingFace and 2.75x on vLLM), while delivering performance comparable to even exceeding that of the Mistral model, which also employs SWA to achieve linear complexity. For the larger 76B model, inference speed is equally compelling: it not only significantly outperforms Llama2-70B, but even surpasses the MoE bascline Mixtral-8x 7B in settings with Expert Parallel (EP) enabled. These results highlight the broad applicability and performance advantages of our approach across model scales and inference frameworks.
For long-sequence training, we further measure Throughput per GPU Second (TGS) on SpikingBBrin-T77 and the Qwen2.5 baseline under the sequence parallelism framework, using MetaX C550 GPUs. The experiments show a 5.36x throughput advantage for our 7B model at a sequence length of 128k, consistent with the TTFT improvements observed during inference and attributable to its efficient attention design.
5.3 CPU-side Inference
In this section, we deploy compressed LB-parameter models, including the SpikingBrain linear model with SWA and GLA, as well as Llama3.2 (Dubey et al. 2024p. on a CPU-based mobile inference framework. The deployment leverages llama.cpp as the inference backend. As illustrated in Figure S the workflow consists of four main steps: i) Weight conversion and quantization, ii) Model registration and tensor mapping, iii) Graph and operator optimization, and iv) Quantized inference. See Appendix A.6 for details.
A key innovation of the SpikingBrain-II linear model lies in its inter-layer hybridization of linear attention and SWA, which avoids full computation over all historical Keys and Values. This design not only reduces KV-cache memory consumption but also significantly lowers inference latency. Compared with conventional Transformer- basedd models, our computation graph explicitly integrates these mechanisms to further optimize KV-cache management. We evaluate decoding speed at different output lengths using a 1 input. The hardware setup includes an Intel Core 15-12K0OKF CPU, 64 GB of memory, and Ubuntu 22.04.4 LTS, with the project compiled using CMake 3.28.3. As shown in Figure 7 the SpikingBrain-ID model maintains constant computation and memory overhead during decoding, resulting in stable throughput the output sequence length increases. In contrast, the 1lama3.2-1B baseline shows sharp decline in decoding speed due to full KV-cache computation. Overall, SpikingBrain-IB achieves speedups of 4.04x, 7.52x, and 15.39x at sequence lengths of 64k, 128k, and 256k, respectively.
This workflow allows the inference engine to scamlessly adapt brain-inspired models for CPU deployment, achieving substantial gains in computational efficiency, memory utilization, and hardware adaptability. It provides practical solution for running large models efficiently in resource-constrained environments.
20

---

Published in Transactions on Machine Learning Research (03/2026)
5.4 Performance the MetaX Cluster
The performance of the MetaX C550 GPU cluster is evaluated from two perspectives: training efficiency and system stability. For the Spiking model during the training phase, we achieve TGS (Token Per Second) of 1558 and an MFU (Model FLOPs Utilization) of 23.4% (8-way DP, 4-way PP, PP micro-batch size 2, global batch size 512); reflecting high computational efficiency and effective resource utilization. This result is comparable to the measured MFU of 25.8% on an NVIDTA A800 80GB GPU cluster under the same parallel configuration, despite the use of older software versions on MetaX due to compatibility constraints.
Regarding stability, continuous monitoring of the cluster operation confirms exceptional reliability and robustness. The training session remains uninterrupted for more than two weeks, underscoring the stability and maturity of the MetaX hardware and software ecosystem.
Overall, these results highlight the capability of the MetaX GPU cluster to efficiently and reliably support large-scale long-duration training tasks, enabling stable training of models with tens of billions of parameters. Additional cross-platform experiments comparing end-to-end model performance and operator-level behavior NVIDIA A100 and MetaX C550 are provided in Appendix A.7
5.5 Analysis of Spiking Scheme
We implement the joint optimization of spiking-based activation encoding and 8-bit fixed-point. weight, quantization (INT8 Quantization) on our pre-trained models. This combination achieves an effective balance between accuracy and energy efficiency, leveraging event-driven computation paradigm to reduce cost while preserving model performance.
Scheme Deployment and Accuracy Maintenance Specifically, we apply spike coding to the activation inputs of all linear projection layers in our models, while the corresponding weights are quantized using symmetric INT8. Quantization parameters are calibrated with a small set of 128 text samples. As shown in Table g evaluations on commonsense reasoning, MMLU, and CMMLU benchmarks indicate that the average performance drop under this scheme is limited to approximately 2% for both SpikingBrain-78 and SpikingBBrain-66B, confirming its effectiveness in preserving accuracy.
Table 3: Performance comparison of SpikingBrain before and after applying the spiking scheme. Evaluations are performed on commonsense reasoning, MMLU, and CMMLU benchmarks. The integration of Shaapiiv-thresholl spike coding with 8-bit weight quantization results in negligible performance degradation (approximately 1-3%).
Arc-c (norm)
HellaSwag (norm)
Winogrande
Arc-e
PIQA
MMLU
CMMLU
Avg.
SpikingBrainn-7
0.6992
0.8047
0.5566
0.6777
0.7949
0.6751
0,6904
0.6998
SpikingBrain-77 -B -W8ASpike
0.6895
0.7861
0.5410
0.6758
0.7979
0.6546
0.6677
0.6875
BrikingBrain -76B SpikingBrain -76B-76688 W8ASpike
0.7275
0.8125
0.5615
0.7000
0,8125
0.7247
0.7740
0.7304
0.7148
0.7949
0.5371
0.6863
0.8004
0.7081
0.7512
0.7133
Spike Distribution Characteristics To evaluate the sparsity and efficiency of the proposed spiking scheme, we analyze the statistical firing patterns under the bitwise-ternary ternary coding method. We collect the absolute values of input spike integers across all linear projection layers. The detailed distribution of spike counts is illustrated in Figure g Take the SpikingBrain-7B model for example, results show that 94.1% of values fall within [0,7], and only about 1% exceed 16. After expansion into bitwise- number of spikes fired per channel is only 1.13, while 18.4% of channels do not fire any spikes at all. Since 94.1% of values are within 7 (i.e., within the INT3 range), we further analyze the firing rate under a step size of 3. By filling inactive bits within the 3-step window with 0 the overall sparsity reaches about 69. 19.15%
spikes, the average
Spike Visualization Interface We provide a unified visualization interface to examine the spiking activity of neurons under different coding strategies. As shown in Figure 10 we compare Binary, Ternary, and Bitwise
21

---

Published in Transactions on Machine Learning Research (03/2026)
binary spike ratio within steps
binary spike ratio within steps
80
19%
86 86.49%
69
70
60
ratio (%)
aVg ratio (%6
40
19. 19.30%
30
17.89%
10.74%
20
9.69%
4.88%
2.8%%
spike
spike
Figure 9: Spike counts distribution of the bitwise spike coding scheme. Results are shown for SpikingBrain-7B (left) and SpikingBrain-7685 (right).
Spike Coding using (wo-dimensional lime neuron firing maps. This tool provides an intuitive means for analyzing spiking behaviors across different spike coding strategies:
Binary Spike Coding typically requires longer time steps and higher firing frequencies, resulting in a relatively dense overall spike distribution.
Ternary Spike Coding introduces positive and negative spikes, yielding a significantly sparser representation that reduces both the total spike count and the required time steps.
Bitwise Spike Coding further enhances compression through bit-expansion strategies, substantially reducing time overhead while preserving representational precision.
Energy Efficiency with Asynchronous Hardware Statistical analysis shows that during inference, 18.4% of channels remain completely inactive (i.e., emit spikes). Under the event-driven computing paradigm of asynchronous hardware, all weight-fetching operations for these inactive channels are skipped (including data transfer from off-chip DRAM to on-chip SRAM and from SRAM to compute units), thereby proportionally reducing memory access overhead.
For computation, we estimate and compare the energy consumption MAC (multiply accumulate) operations under different paradigms, based on published hardware energy consumption data at 45mm technology Yao
et al. 2024b):
Conventional FP16 MAC: Each operation consumes 1.5 pJ, requiring full floating- point multipli- cation and addition regardless of whether the activation effective.
INT8 MAC: Each operation consumes 0.23 pJ, but still depends on synchronous clock-driven execution, incurring energy costs for idle cycles.
Proposed scheme (event-driven spiking computing + INT8 weight quantization): The MAC energy consumption E is estimated based on 'spike-triggered weight additions", i.e,,
Average Spikes x EINTS Add-
(24)
With an average spike count of 1.13 and INT8 (weight) addition energy consumplion of approximately 0.03 pJ per operation, the average MAC energy cost about 0.034 pJ (1.13 x 0.03 pJ).
The results show that our scheme reduces energy consumption by 97.7% compared with FP16 MACs and by 85.2% compared with INT8 MACs, yielding energy-officiency improvements of up 43.48x and 6.76x, respectively. This demonstrates the offectiveness of combining spiking-driveen computation with quantization to significantly reduce energy overhead, while maintaining precision degradation within controllable range.
22

---

Published in Transactions on Machine Learning Research (03/2026)
However, these energy-efficieney estimates based idcalized assumptions regarding asynchronous hardware behavior. Hardware challenges underlying these estimates and corresponding directions toward practical realization are discussed in Appendix C.2
6 Discussion of Negative Results and Practical Challenges
In this section, we discuss experimental configurations and implementation strategies that did not yield the expected improvements, as well as challenges encountered during adaptation to MetaX hardware. Sharing these observations provides practical guidance and improves the transparency and reproducibility of our work.
Long-context data construction. Training on concatenated short texts, described in Section 3.4 rather than native long-context data, resulted in limited improvements in long-context performance. This suggests that downstream performance at extreme context lengths depends strongly on both the quantity and quality of long-context training data. Incorporating naturally long-context samples or recall-intensive training data is expected to further reduce the performance gap between SpikingBrain and the base Transformer model.
MoE upcycling initialization. Increasing expert diversity through partial random initialization of expert weights did not produce meaningful performance improvements. As a result, we adopt direct. copying and rescaling of expert weights during MoE upcycling, which provides stable and consistent performance.
Sequence parallelism stability. During multi-node requence-paralien (SP) training, we observed reduced GPU utilization and occasional out-of-memory errors or deadlocks. These issucs were primarily caused by asynchronous operator accumulation leading to excessive buffer pre-allocation, as well as inter-node load imbalance. We resolved these issucs by introducing explicit synchronization after SP communication primitives, which improved both stability and resource utilization.
Context length in reasoning SFT. Using an Sk context length during the reasoning SFT stage resulted truncation long chain-offthought sequences, limiting SpikingBrain's reasoning performance. Increasing the context length (e.g., 32k) enables more complete reasoning trajectories and improves downstream reasoning capability.
Software adaptation on MetaX platform. Due to software stack incompatibilities on the MetaX platform at the time of development, we adapted the Megatron framework to operate with older versions of PyTorch and Triton. This limited access to newer optimization features and required conservative kernel design choices. For example, reduced Triton block sizes and constrained autotuning configurations were used to ensure stability, but resulted in suboptimal performance compared to fully optimized implementations.
These observations highlight important, practical considerations for large-scale deployment and hardware adaptation of efficient and brain brain-inspired large language models.
7 Conclusion
This work provides a comprchensive demonstration of efficient brain-inspired large model training and inference on the MetaX GPU cluster. By integrating several key techniques including novel (hybrid) linear architectures beyond Transformers, sparse MoE' design, lightweight conversion training- we validate and implement practical development pipeline for spiking-based large models on a non-NVIDIA cluster with hundreds of GPUs.
threshold sparse spiking activation, and
We release two models as outcomes of this effort: the linear model SpikingBrain-77 and the MoE hybrid-linen model BraneeBBiii----112222 These models offer two main advantages: i) Training efficiency: With lincar or near-linear complexity, they substantially accelerate long-sequence training and match the performance of many open-source Transformer models while using less than 2% of the training data via continual pre-training.
23

---

Published in Transactions on Machine Learning Research (03/2026)
ii) Inference performance:
During inference, the models exhibit event-driven spiking behavior and maintain constant (SpikingBrain-7B) or layer-wise partially constant (Spiking(Brain-76B3) memory footprint. This yields ordgr-of-magnitude improvements in long-sequence efficiency and speed (e.g., over x00x TTFT acceleration at 4M tokens).
These results not only demonstrate the feasibility of efficient large-model training on non-NVIDIA platforms, but also outline new directions for the scalable deployment and application of brain-inspired models in future computing systems.
Broader Impact Statement
Our work introduces SpikingBrain, a framework designed to enhance the efficiency and accessibility of large language models (LLMs) across diverse hardware platforms, including non-NVIDIA GPUs and neuromorphic systems. By significantly reducing the energy consumption associated with LLM training and inference, methods contribute to the democratization of AI, potentially fostering innovation and rescarch in resource-constrained environments.
However, we recognize that the increased efficiency and broader deployment capabilities of LLMs, facilitated by our framework, also carries dual-use risks. A more widespread accessibility of LLMs could inadvertently amplify existing concerns related to the generation of harmful, biased, or unsafe content. These challenges, while inherent the scale of LLM technology, are potentially accelerated by tools that simplify their adoption.
We strongly advocate for the responsible development of efficient AI We encourage users of BrikingBrain to integrate robust safety guardrails and local moderation filters, especially for edge-case deployments. Our ultimate goal is to provide a toolset that balances computational performance with ethical responsibility, ensuring that the transition toward more efficient Al remains aligned with societal well-being.
Author Contributions
Yuqi Pan and Yuhong Chou designed the model architectures. Siyu Ding proposed the adaptive threshold spiking neuron and conducted spiseng-based training. Han Xu and Xuerui Qiu developed three spike coding techniques. Yuqi Pan, Yupeng Feng, and Zehao Liu carried out the framework adaptation on MetaX and model training. Jinghao Zhuang, Yupeng Feng, Zehao Liu, Yuqi Pan, Bohan Sun, and Yuhong Chou performed model evaluation. Anjie Hu and Anlin Deng contributed to code adaptation and refinement. Peng Zhou performed CPU adaptation and model testing. Yuqi Pan, Shurong Wang, Guoqi Li, Man Yao, Jibin Wu, Jian Yang, Guoliang Sun, and Bo Xu contributed to the discussion and analysis of experimental results. The project supervised by Bo Xu and Guoqi Li.
Acknowledgments
This work was partially supported by CAS Project for Young Scientists in Basic Research (YSBR-116), National Distinguished Young Scholars (62325603), National Natural Science Foundation of China (62236009, U22A20103), Beijing Science and Technology Plan (724110000422011), and Shanghai Neuhelium Neuromor- phic Intelligence Technology Co.Ltd.
24

---

Published in Transactions on Machine Learning Research (03/2026)
References
Anthropic. Introducing Claude Opus 4.1.
2025.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Mcasuring and improving recall in efficient language models. In Proceedings of 12th International Conference on Learning Represenlations (ICLR). ICLR, 2024a.
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. Simple linear attention language models balance the recall-throughuut tradeoff. In International Conference on Machine Learning, pp. 1763- 1840. PMLR, 2024b.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding, In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 3119 -1337, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ hatps3/vvvv//v /1/2024..cllooogg 172. URL https://acclatthloogy.oo wwt../rgg//0// wttp///2....lloooooo...oaa 172/
Tz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 055100 2020.
Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficck, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. Nemotron-h: A family of accurate and efficient hybrid mamaa-trannsoommr models. arXiv preprint ar 2an:2544.0662, 2025.
Paola A Buitrago and Nicholas A Nystrom. Open compass: accelerating the adoption of ai in open research. In Practice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning), pp. 1-9. 2019.
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34:17413-17426, 2021.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint ar Xrxi::1604.00 2016.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXi::190, 10509, 2019.
Yuhong Chou, Man Yao, Kexin Wang, Yuqi Pan, Ruijie Zhu, Yiran Zhong, Qiao Yu, Jibin Wu, Bo Xu, and Guoqi Li. Metala: Unified optimal linear approximation softmax attention map. In The Thirty-Eightt Annuual Conference Neural Information Processing Systems. Neural information processing systems foundation, 2024.
Yuhong Chou, Zehao Liu, Ruijie Zhu, Xinyi Wan, Tianjian Li, Congying Chu, Qian Liu, Jibin Wu, and Zejun Ma. Zeco: Zero communication overhead sequence parallelism for linear attention. arXiv preprint N.N::5577.01000 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXie preprinl arXiv: S5f::800.055 05,57, 2018.
Damai Dai, Chengqi Deng, Chenggang Zhao, Rx Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture-of of-exerrss language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1280-1297 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Tri Dao and Albert Gu. Transformers ssms: generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, pp. 10041 10071, 2024.
Jyotikrishna Dass, Shang Wu, Huihong Shi, Chaojian Li, Zhifan Ye, Zhongfeng Wang, and Yingyan Lin. Vitality: Unifying low-rank and sparse approxination for vision transformer acceleration with a linear taylor attention. In 2023 IEEE International Symposium on High-Performanee Computer Architecture (HPCA), pp. 415-428. IEEE, 2023.
Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffnn Mixing gated linear recurrences with local attention for efficient. language models. arXiv preprinl ar Xix::2400. 19427, 2024.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hybrid-head architecture for small language models. arviv preprint or xxXu:22411.1667, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. ar 2407, 2024.
Liangwei Fan, Hui Shen, Xiangkai Lian, Yulin Li, Man Yao, Guoqi Li, and Dewen Hu. A multisynaptic spiking neuron for simultancously encoding spatiotemporal dynamics. Nature Communications, 16(1) :7155,, 2025. doi: 10.1038//414467-055-77111
URL
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(1))): 2022.
Charlotte Frenkel, David Bol, and Giacomo Indiveri. Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence. Proceedings of the IEFE, (11(6):223-665)) 2023.
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Tbrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. ar preprint 4rXiv:2405.1611, 2024.
Google Deep.indd Gemini 2.5 Pro. http::///asppaaud.oooooo////////
2025.
Albert Gu. On the tradcoffs of state space models and transformers, 2025. URL bttps::/ggooballe sataaalll..ghhittt github
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepsock-r1: Incentivizing reasoning capability in 1lms via reinforcement learning. Xr preprint arXiv:2501.1200 2025.
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXie preprint ar Xrv: 2018.
Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shocybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts. arXiv preprint arXiv:2410. 110002200 2024a.
Linxuan He, Yunhui Xu, Weihua He, Yihan Lin, Yang Tian, Yujie Wu, Wenhui Wang, Ziyang Zhang, Junwei Han, Yonghong Tian, et al. Network model with internal complexity bridges artificial intelligence and neuroscience. Nature Computational Science, 4(8):584 599, 2024b.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arxiv preprint arXi::200900330 2020.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Alan L Hodgkin and Andrew F Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology, 117(4):500, 1952.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advonces in Neural Information Processing Systems, 36:62991 63010, 2023.
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Lcon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14 14509, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. X'xiv preprinl ar orXii:2401.04000 2024.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b. Arxi abss2231.06682, 2023. URL https://api. httpss//aap.semmanicccchooo semanticscholas org/
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXic preprint orXiu: 705.03551, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.088 000666,,,,,,,,,, 2020.
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A Smith. Finetuning pretrained transformers into rmns. In 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pp. 10630-10643. Association for Computational Linguistics (ACL), 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rmns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp- 5156-5165. PMLR, 2020.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arxin:221.. 2. 2.0m022, 2022.
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. arXiv preprint 2rXiv:2205.05193 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th. Symposium on Operating Systems Principles, 2023.
Dmitry Lepikhin, HyoukJoong Leo, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. Xixiv preprint ar Xiv:20006. 1 16668, 2020.
27

---

Published in Transactions on Machine Learning Research (03/2026)
Guoqi Li, Lei Deng, Huajin Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass. Brain- inspired computing: A systematic survey and future trends. Proceedings of the IEEE, 112(6):544456) 584, 2024. doi: 10.1109/JPROC:2224....
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Mcasuring massive multitask language understanding in chinese. arXiv preprint Xrv:22306.000 092122 2023a.
Jijie Li, Li Du, Hanyu Zhao, Bo-wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis to enhance language models. arXiv preprinl WrXi:2506.11116, 2025.
Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Paraliel Processing, pp. 766 775, 2023b.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Mcirom, Yonatan Belinkov, Shai Shalev- Shwarr-S, et al. Jamba: A hybrid transformer-manba language model. arXiv preprinl arxiv:2403. 19887, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprinl orXiv:22 12. 19437, 2024.
Cong Liu, Zhong Wang, ShengYu Shen, Jialiang Peng, Xiaoli Zhang, ZhenDong Du, and YaFang Wang. The chinese dataset distilled from deepseek-r1-671
110k, 2025.
Xinhao Luo, Man Yao, Yuhong Chou, Bo Xu, and Guoqi Li. Integer-valued training and spike-driven inference spiking neural network for pirfopeeformancc and energy-efficient object detection. In European Conference on Computer Vision, pp. 253-272. Springer, 2025.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1099):1659-1671, 1997.
Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Jash Mehta, and Toby Liang. Millions scale dataset distilled from r1-32b. htlps://uuggnnglcceeco
2025.
Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models. arXiv preprint 2rXi::2405.066/ 06640, 2024.
Deepak Narayanan, Mohammad Shocybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im arXiv preprint arXio:2104.04773, 2021.
NVIDIA. Communication
hatrrollkk//aaaunegggggaaeaaatec html 2025.
OpenAL. GPT-5. https:///ppnaal.com//
2025.
John P O'Doherty, Sang Wan Lee, Reza Tadayonnejad, Jeff Cockburn, Kyo ligaya, and Caroline J Charpentier. Why and how the brain weights contributions from a mixture of experts. Neuroscience & Biobehavioral Reviews, 123:14-23 23, 2021.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 5336-5358, Scattle, Unitod States, July 2022. Association for Computational Linguisties. URL https://aaclaatooooogg. org/2222 naacl-main.391
28

---

Published in Transactions on Machine Learning Research (03/2026)
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rmss with state expansion. In First Conference on Language Modeling, 2024.
Qwen Team. Qwen3-next: Towards ultimate training & inference efficiency. https://quen. www..ee....bloliiiii. 68443808866606555771111777777777
2025. Qwen Research Blog.
Samyam Rajbhandari, Jeff Rasicy, Olatunji Ruwase, and Yuxiong Hc. Zero: Mcmory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Compuling, Nettwrkingg Storage and Analysis, pp. 1-16. IEEE, 2020.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power cext-generation ai scale. In International conference on machine learning, pp. 18332-18346. PMLR, 2022.
Liliang Ren, Yang Liu, Yadong Lu, Chen Liang, Weizhu Chen, et al. Samba: Simple hybrid state space models for efficient unlimited context language modcling. In The Thirteenth International Conference Learning Represenlations, 2024.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-611, 2019.
Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, J Parker Mitchell, Prasanna Date, and Bill Kay. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, (1):10-19, 19, 2022.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Calanzaro. Megatron-Ima Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1999. 099990909000... 2019.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.0990 09864, 2021. URL httpps///ap..smmmaticccccloaad wtw.aaccooaaaaorrorrrppsssl.l
233307138
Weigao Sun, Disen Lan, Yiran Zhong, Xiaoye Qu, and Yu Cheng. Lasp-2: Rethinking sequence parallelism for linear attention and its hybrid. ArXiu, abs/2502.07563, 2025. URL https://api tttps://app..smmaticcchoaaar semanticscholar
org/CorpusID. http/oorpuIID::22590011 276259019
Yulao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: successor to transformer for large language models. orXiv preprint arXiv:2307.086..
2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. Xr preprint arXiv:2408.00) 00118, 2024.
Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, al. Kimi linear: An expressive, efficient attention architecture. arXiv preprint 2Xii:2510.26694 2025.
Mistral AI team. Mistral 7b.
www.eiinnnniitttt---la. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint wxXi::2307.09288 2023.
Leslie G. Valiant. A bridging model for parallel computation. Commun. ACM, 3388):003-111, August 1990. ISSN 0001-0782. doi: 0..114/79113..71111 URL htttp:://abi.orgg11.114//////11
29

---

Published in Transactions on Machine Learning Research (03/2026)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Roger Waleffe, Wonmin Bycon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba- based language models. arXiv preprint arXiv:2406.077 00.808877 2024.
Junxiong Wang, Daniele Paliotta, Avner May, Alexander Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37: 62432 62457, 2024.
Kexin Wang, Yuhong Chou, Di Shang, Shijie Mei, Jiahong Zhang, Yanbin Huang, Man Yao, Bo Xu, and Guoqi Li. MMDEND: Dendrite-inspired multi-branch multi-compartment parallel spiking neuron for sequence modeling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63vd Annual Meeling of the Associalion for Computational Linguislies (Volume 1: Long Papers), pp. 27459 27470, Vienna, Austria, July 2025. Association for Computational Linguistics. 1332. URL httpss//acclaaatooooo.oooooe///e 1332.
ISBN 979-8-89176-251-00 doi: 10.18653/vi /0/////////// 1332//
acl-long 1332/
Qinlong Wang, Bo Sang, Haitao Zhang, Mingjie Tang, and Ke Zhang. Dlrover: An elastic deep training extension with auto job resource recommendation. CoRR, 2023.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. Xixiv preprint arXiv:2006.0 04..0477688 2020.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint XrXi::2309, 1 17453, 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Ke- Yany Chen, Kexin Yang, Mei Li, Min Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yunyang Wan, Yunfei Chu, Zeyu Cui, Zhenru Zhang, and Zhi-Wei Fan. Qwen2 technical report. ArXiv, abs/2407.10671, 2024a. URL http:://api. commanticccoole.e
httg//Copus::::::: 27:21122300
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. ar preprint arXiv:2412.15115 2024b.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In International Conference on Machine Learning, pp. 56501-56523. PMLR, 2024c.
Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. In Advances in Neural Information Processing Systems, volume 36, pp. 64043-64058, 2023.
Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo XU, and Guoqi Li. Spike- driven transformer v2: Meta spiking neural network architecture inspiring the design of next-gencration neuromorphic chips. In The Twelfth International Conference on Learning Representations, 2024a.
Man Yao, Ole Richter, Guangshe Zhao, Ning Qiao, Yannan Xing, Dingheng Wang, Tianxiang Hu, Wei Fang. Tugha Demirci, Michele De Marchi, Lei Deng, Tianyi Yan, Carsten Niclsen, Sadique Sheik, Chenxi Wu, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-based dynamic computing with asynchronous sensing- computing neuromorphic chip. Nature Communications, 15(1):4464, 2024b. ISSN 2041-1723.
30

---

Published in Transactions on Machine Learning Research (03/2026)
Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling spike-driven transformer with efficient spike firing approximation training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4))2973 2990, 2025. doi: 10.1109/ 2P2MMI.005.3530244
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago Outanon, Philip Pham, Anirudh Ravula, Qilan Wang, Li Yang, el al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283 -12997, 2020.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint ar trXit:1905.08830 2019.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang. Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2405. 19327, 2024a.
Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, and Christopher Ré. Lolcats: On low-rank linearizing of large language models. arXiv preprint 2wXiu:2410.10220 2024b.
Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient finar-time sequence modeling. Advances in Neural Information Processing Systems, 37:116870-116998 2024c.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou, intructioonfollowinng evaluation for large language models. ar preprint arX222331.00711, 2023.
Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, llyas Chabed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint or arXiv:2410.0533 2024.
Jingwei Zuo, Maksim Velikanov, lyas Chahed, Youncs Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, et al. Falcon-h1: A family of hybrid-head language models redefining efficiency and performance. ar preprint 2xxv:2507.22444 2025.
31

---

Published in Transactions on Machine Learning Research (03/2026)
A Experiments
A.1 Benchmarks
In selecting evaluation metrics, we place greater emphasis on pretraining-orienter general-purpose benchmarks: MMLU [Hendrycks et al. 2020), CMMLU (Li et al. 2023a), C-Eval (Huang et al., 2023). ARC-C (Clark et al. 2018); and HellaSwag (Zelless al. 2019), as these better indicate whether our models trained with fewer than 200B tokens during efficient conversion- can inherit the knowlodge and modcling capacity of the base model. Moreover, these benchmarks reveal whether the converted models preserve strong generalization while demonstrating significant
efficiency advantages, without being confounded by data quality factors such as alignment performance after SFT.
For SpikingBrain chat. models after three-stage SFT, we evaluate general knowledge, long-sequence modeling, and instruction following, additionally including NQ (Kwiatkowski et al. 2019), TrQ (Joshi et al. 2017);
QuALITY (Pang al. 2022), and IFEval (Zhou et al. 2023 To assess real-world long-context modcling and understanding capability, also evaluate six popular tasks on LongBench (Bai et al. 2024).
A.2 Models
We compare SpikingBrain models against open-source baselines of comparable scale spanning Hybrid, Transformer, and MoE architectures:
Linear-complexity models: Falcon-Mamba-7B (Zuo et. al.
2024
Mistral--B (Leam
2023
Hybrid linear-complexity models: Zamba-vl-7B Gllorosso et al.
2024
Jamba-522 520--223---- A12B
Lieber
al.
2024
Quadratic-complexity models: Llama3.1-88 1-BB (Dubey et al. 2024).
Qwen2.5-7B Yang et al.
2024b)
Mixtral-47B-A131 (Jiang al. 2024 LJamm2-70b, (Touvron el al. 2023), Gemma2-27B
(Team
ct al.
2024
A.3 Evaluation of Instruct Models
We apply three-stage SFT the converted models to add instruction following. As shown in Table 4 the aligned models match similarly sized open-source chat basclines across general knowledge, long-sequence mod- eling, and instruction following, while avoiding overfitting and preserving pretrained capabilities indicating that (hybrid) linear architectures remain stable and scalable under alignment.
Table 4: Performance Evaluation of SpikingBrain Instruct Models. All models tested with the vLLM framework and evaluated using generation-basel method. For QuALITY and IFEvll, report results from the non-CoT model (after SFT stage 2) to avoid chain-of-thouge interference.
SpikingBrain-7B
SpikingBrain-76B
Llama3
Qwen2.5
Mixtral
Params Complexity Type
7B
12B/76B Hybrid
8B
7B
13B/478
Linear
Quadratic
Quadratic
Quadratic
Benchmarks
MMLU
65.57
73.71
68.69
75.17
71.03
CMMLU
68.76
77.41
55.17
79.14
51.03
HellaSwag
68.95
86.63
76.80
85.39
75.63
Ceval
69.07
76.32
55.01
77.93
50.88
NQ
21.47
21.55
30.97
17.67
28.48
TrQ
57.03
55.13
65.78
55.72
71.00
QuALITY
60.12
69.56
66.25
73.63
51.34
IFEval
42.70
49.72
73.01
73.20
48.06
32

---

Published in Transactions on Machine Learning Research (03/2026)
A.4 Evaluation LongBench
To assess real-world long-context understandingg, we evaluate SpikingBBain--66 (after stage-2 SFT) on six LongBench tasks (Bai al., 2024), inchuding HotpotQA, NarrativeQA, TriviaQA, TREC, MultiNews and Qasper. Given the baseline's maximum positional embeddings, we truncale prompls to 32k. As shown in Table model is competitive with other hybrid and Transformer baselines while using only four global softmax attention layers.
Table 5: Performance Evaluation of SpikingBBainn-66B on six LongBench tasks (32k).
SpikingBBran-766B
Jamba
Qwen2.5
Mixtral
Params Complexity Type
12B/76B
12B/52B Hybrid
7B
13B/47B
Hybrid
Quadratic
Quadratic
Benchmarks
HotpotQA
50.00
11.38
57.62
36.42
NarrativeQA
23.92
3.73
27.96
7.44
TriviaQA
90.84
92.63
85.92
85.97
TREC
67.00
77.00
68.50
19.25
MultiNews
25.89
8.44
23.79
26.52
Qasper
39.59
10.67
45.13
28.37
To strengthen the comparison, we extend the Qwen2.5 baseline 128k context length using YaRN and evaluate the same six LongBench tasks with prompts truncated 128k tokens. As shown in Table the results demonstrate that SpikingBrainn766B remains competitive at longer context lengths, even with relatively high hybrid ratio of 1:6.
Table 6: Performance Evaluation of SpikingBrain-761 on six LongBench tasks (128k).
SpikingBrain-76B3
Qwen2.5
Params Complexity Type
12B/76B Hybrid
7B
Quadratic
Benchmarks
HotpotQA
48.42
57.96
NarrativeQA
23.35
27.10
TriviaQA
91.24
88.25
TRFC
72.00
62.50
MultiNews
26.51
23.51
Qasper
38.85
44.43
Discussion of Recall. Tt important note that LongBench primarily evaluates long-context modeling and understanding capabilitics, and docs not directly measure long-context recall performance, Lincar and subquadratic (hybrid) altention models exhibit known limitations in recall due to their reduced effective state size compared to quadratic Transformers (Waleffe et al. 2024 Arora et al. 2024ah. We empirically validate
this recall gap in Appendix B.1
This limitation can be mitigated through improved training data quality and scale. Prior industrial stud- ies (Qwen Team 2025 Team et al. 2025) have shown that hybrid attention architectures can achieve retrieval performance comparable to Transformers when trained with sufficient data and infrastructure. These findings suggest that recall limitations are intrinsic barriers, but can be reduced through appropriate training and system design.
33

---

Published in Transactions on Machine Learning Research (03/2026)
A.5 Long-context Efficiency
Table 7: Inference speed comparison of SpikingBrain-77 under sequence parallclism (SP). The metric TTFT (ms), defined the latency lo complete prefill and generale the first loken for a given prompl length. In SP configuration, our 7B model uses ZeCO for the linear attention module and P2P commmicattin for the SWA module, while the Qwen2.5 bascline employs A2A communication. All mcasurements are conducted on NVIDIA H100 GPUs and averaged over 10 runs. indicates infeasible measurement due to resource constraints.
Sequence Length
256k
512k
1M
2M
4M
GPU Count
8
16
32
64
128
SpikingBrainn77
1015
1037
1054
1070
1073
Qwen2.5
7419
14398
27929
Table 8: Inference latency (ms) comparison across models, frameworks, and sequence lengths. Tests are conducted on HuggingFace single-GPU deployment with MetaX C550 GPUs and on vLLM with both single- and multi-GPU settings. Latency is measured as the time to process the input sequence and generate 128 output tokens,
SpikingBrain
Mistral
Qwen2.5
SpikingBrain
Mistral
Qwen2.5 -7B
SpikingBrain -76B
Mixtral 8x7H v0.1
Llama -2. 2.70b
v0.
-7B
GPUs
Prameworks
HugeingFare
vlhm
vllm
Experi Parallel
Non MoB
Enable
Disable
Mon-MoE
32k
8452
10202
7969
4369
4570
6421
9451 / 5.305
10477 / 3209
15881
64k
11685
16467
16257
7399
7084
15093
14077
8142
19062 / 4840
32953
128k
15974
28382
38487
12048
11867
45141
27106 / 14109
34983 / 8046
86120
A.6 CPU-side Inference
We demonstrate the deployment of compressed 1B-scale models, including the SpikingBrain linear model with SWA and GLA, well as Llama3.2 (Dubey et al. 2024), CPU-based mobile inference framework. The deployment leverages llama.cpp as the inference backend. As illustrated in Figure g the workflow consists of four main steps:
Weight conversion and quantization. The trained weights are first converted into the GGUF format with an appropriate quantization strategy. Unlike generic training formats, GGUF emphasizes compact encoding and cross-platform consistency, yielding more predictable load latency and memory footprint during CPU inference. Floating-por point weights are compressed into fow-bit integers (e.g., Q4, Q5). Combined with efficient decoding and optimized matrix- matrix- kernels, this greatly improves throughput while reducing memory usage.
Model registration and tensor mapping. A dedicated computation graph constructed by registering architecture components and mapping tensors within the inference engine. Each tensor in the weight file is directly linked to its corresponding graph structure, with associated metadata such as shape and precision format (e.g., float32, float64).
Graph and operator optimization. Following the forward propagation flow, implement key operators including attention, rotary position embedding (RoPE (Su et al., (021)), and residual connections. Two major oplimizations are applied: i) Operator fusion: combining matrix multipli- cation, bias addition, and nonlinear activation into a single kernel to reduce intermediate storage and memory access. ii) Graph-level optimization: decomposing and reorganizing operations such as RMSNorm, Softmax, and RoPE to enable fusion with matrix multiplication or low-level kernels, minimizing redundant computation.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Quantized inference. Finally, the quantized weights are loaded, and forward computation is performed in low precision. By default, Q4 quantization is applied to the weights to balance computational throughput with memory usage. In addition, we use compact single-step integer activations rather than expanded spike trains, ensuring compatibility with current CPU hardware.
A key innovation of the SpikingBrain-11 linear model lies in its inter-layer hybridization of linear attention and SWA, which avoids full computation over all historical Keys and Values. This design not only reduces KV-cache memory consumption but also significantly lowers inference latency. Compared with conventional Transformer- based models, our computation graph explicitly integrates these mechanisms to further optimize KV-cache management. We evaluate decoding speed at different output lengths using a 1k input. The hardware setup includes an Intel Core i5- 15-12K0K CPU, 64 GB of memory, and Ubuntu 22.04.4 LTS, with the project compiled using CMake 3.28.3. As shown in Figure 7 the SpikingBrain-1D B model maintains constant computation and memory overhead during decoding, resulting in stable throughput the output sequence length increases. In contrast, the Llama3.2 2-1B baseline shows sharp decline in decoding speed due to full KV-cache computation. Overall, SpikingBrain- 1B achieves speedups of 4.04x, 7.52x, and x at sequence lengths of 64k, 128k, and 256k, respectively.
This workflow allows the inference engine to seamlessly adapt brain-inspired models for CPU deployment, achieving substantial gains in computational efficiency, memory utilization, and hardware adaptability, It provides practical solution for running large models efficiently in resourco-constrained environments.
A.7 MetaX Performance Metrics Relative to NVIDIA
Beyond the MFU comparison with NVIDIA A800, we conduct additional experiments both NVIDIA A100- SXM4-80GB and MetaX C550 (64GB HBM), focusing on end-to-end model performance and operator-level behavior. These experiments aim to demonstrate the cross-platform adaptability of SpikingBrain and enable direct performance comparison under controlled conditions.
Overall model speed comparison. We benchmark forward-only execution of SpikingBrain-771 with batch size (bs) 1 and sequence lengths ranging from 1k to 16k. For each configuration, throughput (tokens/s) is computed as bs seqlen/wall /wall-llock time. Identical runtime settings and mcasurement protocols are used both platforms, including warm-up runs followed by averaged timed measurements.
As shown in Table 9 MetaX achieves approximately
the throughput of NVIDIA A100 in this forward benchmark. Notably, the performance gap narrows as sequence length increases, indicating improvod hardware utilization on MetaX in long-context regimes.
Table 9: Forward throughput comparison of SpikingBrain-7B on NVIDIA A100 and MetaX C550. Throughput is measured in tokens/s with batch size = 1.
Sequence Length
NVIDIA (tokens/s) 1
MetaX (tokens/s /s)
Ratio
1024
11140.93
8265.79
0.769
2018
12316.52
10440.11
0.848
1096
12524.46
10897.70
0.870
8192
12621.84
11392.21
0.903
16384
12768.80
11847.01
0.928
Operator-level comparison. To further analyze hardware efficiency, we profile an inter-layer hybrid block and measure per-operator latency using GPU event timing. To enstre fair shape comparison, we fix the total token count (bs x seglen = 65536) while varying batch size and sequence length. All latency values are reported in milliseconds.
As shown in Table 10 the end-to-end hybrid block latency remains within +2% between MetaX C550 and NVIDIA A100 across all testod configurations. At operator gramularity, MctaX is consistently faster on FFN and normalization layers, largely comparable on SWA, while linear attention (LA) remains the primary source of latency overhead on MetaX. However, this gap decreases as sequence length increases.
35

---

Published in Transactions on Machine Learning Research (03/2026)
Table 10: Operator-level latency comparison of SpikingBrain inter-layer hybrid blocks on NVIDIA
A and MetaX C550. All values in milliscconds.
bs
seqlen
NVIDIA (ms) + MetaX (ms) 4
A total
A FFN
A LA
A SWA
A Norm
16
4096
7051
7188
+1.9%
-6.2%
+54.6%
-4.2%
-2.9%
8192
7220
7264
+0.6%
-6.0%
+45.8%
-6.0%
-2.9%
4
16384
7284
7236
-0.7%
-5.8%
+33.7%
-5.1%
-2.7%
32768
7303
7284
-0.3%
-5.7%
+33.6%
-3.2%
-2.7%
65536
7291
7258
-0.5%
-6.1%
+23.9%
+4.2%
-2.9%
These results support several important observations. First, SpikingBrain demonstrates strong cross-platform portability, achieving comparable end-to-end performance both NVIDIA A100 and MetaX C550 using the same implementation. The hybrid block latency remains within +2% across wide range of tensor shapes. Second, the throughput gap between MelaX and NVIDIA decreases as sequence length increases, with MetaX reaching 0.928x of NVIDIA throughput at 16k sequence length, compared to 0.769x at 1k, indicating favorable scaling in long-context scttings. Finally, operator-level analysis reveals that MetaX outperforms NVIDIA on FFN and normalization layers, while the remaining performance gap is primarily attributable to linear attention. This suggests that further optimization of linear attention kernels represents the most direct path closing the remaining performance gap.
A.8 Effect of Adaptive Threshold Hyperparameter
The adaptive threshold hyperparameter k controls the scaling of the firing threshold and thereby determines the spike count distribution. Specifically, the adaptive threshold is defined as
Vih(x) 1 mean(abs(x)),
(25)
such that larger values of & correspond to lower thresholds, resulting in increased spike activity (i.e., reduced sparsity). While lower sparsity generally improves accuracy by preserving more activation information, also reduces the potential efficiency benefits of spike-based computation. This reflects an inherent accuracy sparsity trade-off in spike coding.
We empirically evaluate this tradc-off using the SpikingBrain-7B model across multiple benchmark tasks. As shown in Table 11. increasing & consistently improves overall accuracy while decreasing sparsity, confirming the expected relshionship between threshold scaling, spike activity, and model performance. Based on this trade-off, we select as the default setting, which provides a favorable balance between accuracy and sparsity.
Table 11: Effect of adaptive threshold hyperparameter on accuracy and sparsity in SpikingBrain- 7B. Larger values reduce sparsity and improve accuracy, demonstrating the accuracy-sparsity trade-off.
Configuration
WG
ARC-E
ARC-C
HellaSwag
PIQA
MMLU
CMMLU
Avg.
Sparsity
Values
BF16
0.6992
0.8047
0.3566
0.6777
0.7949
0.6751
0.6004
0.6998
W8ASpike (k
0.6835
0.7887
0.5213
0.6931
0.7927
0.6012
0.6339
0.6735
0.730
0.980
WSASpike (R= = 2.5)
0.6780
0.7934
0.5316
0.7021
0.7894
0.6141
0.6560
0.6807
0.709
0.978
WSASpike (N- = 3)
0.6895
0.7861
0.5410
0.6758
0.7979
0.65-46
0.6677
0.6875
0.892
0.951
WSASpike (R = 3.5)
0.6851
0.8035
0.5427
0.7012
0.7933
0.6305
0.6723
0.6898
0.677
0.916
W8ASpike
0.6946
0.8056
0.5452
0.7046
0.7933
0.6331
0.6817
0.6940
A.9 Spike Visualization Interface
We provide a unified visualization interface to examine the spiking activity of neurons under different coding strategics. As shown in Figure [0] we compare Binary, Ternary, and Bitwise Spike Coding using two- dimensional Line-neuron firing maps. This tool provides an intuitive means for analyzing spiking behaviors across different spike coding strategies.
36

---

Published in Transactions on Machine Learning Research (03/2026)
BinaryLif (Tokens= 20 TimeStep=11 15 FiringRate=0.50 0.50)
28
24
20
Neure
16
12
40
80
120
160
200
240
280
Time
Ternary Lif (Tokens=20 TimeStep=8 FiringRate=
28
t.....
24
20
Neuroh
16
12
20
60
100
120
140
Time
Bitwise Lif (Tokens= 20 TimeStep=3 Faring 0.27 Bidirectional)
28
24
Neuron
16
12
0
8
16
24
32
40
48
56
Time
Bitwise Lif (Tokens=20 TimeStep=4 Firing 00.42 Complement)
28
24
20
Neuron
16
12
10
30
40
60
80
Time
Figure 10: Time neuron firing maps under different spike coding schemes. The figure shows the spike firing distributions for the same input under different coding strategies, including Binary, Ternary, and two variants of Bitwise spike coding. The horizontal axis represents time (Time), defined as token timesteps x expanded timesteps; the vertical axis represents neuron index (Neuron). Black dots indicate spike events of the corresponding neuron at that time.
37

---

Published in Transactions on Machine Learning Research (03/2026)
B
Ablation Studies
To enable controlled comparisons, we conduct strict data-matched conversion ablation experiments using continual pre-training (CPT) from Qwen3-1.7B with 5B tokens. The training configuration uses sequence length 8k, batch size 32, and learning rate 3 x 10-5.
B.1 Attention Comparison under Identical Conversion Data Budgets
We compare the inter-layer hybrid attention paradigm used in SpikingBrain-7B (LA:SWA = 1:1), pure linear attention (LA, similar to Falcon-Mamba), and pure sliding-window attention (SWA, similar to Mistral) under identical conversion settings. For SWA training, we use a window size of 4k and report the equivalent window size for all models. For fair comparison, we additionally evaluate a pure SWA variant with the window size fixed to 128.
Table 12: Attention comparison under identical conversion data budgets.
Equivalent Window Size
Model
HellaSwag
MMLU
SNIAHI-4k
SNIAHI-SK
FA Baseline
x
60.21
61.78
100.0
98.41
Pure SWA (4k)
4096
60.12
61.74
100.0
47.60
Pure SWA (128)
128
1.87
0.00
Pure LA
64
54.72
39.87
36.16
39.40
Hybrid
2080
57.88
56.08
96.83
44.44
The results in Table 12 reveal several important trends:
General modeling performance strongly influenced by the degree of architectural deviation from the original Transformer. Pure SWA achieves performance closest to the full- attention baseline on HellaSwag and MMLU, indicating that smaller architectural changes facilitate faster recovery of language modeling capability under limited conversion data.
Recall performance primarily governed by effective state size. While SWA-4k achieves optimal recall within its attention window, pure linear attention demonstrates substantially stronger recall than SWA-128 despite having comparable state size, reflecting its ability to accumulate information beyond fixed window constraints.
Hybrid attention provides the best overall trade-off between performance, recall capa- bility, and efficiency. By combining SWA and linear attention, the hybrid design achieves strong recall performance while maintaining favorable computational efficiency and memory usage, making it the most effective architecture choice under constrained conversion settings.
B.2 Isolated Impact of Hybrid Attention, MoE, and Spike Coding
SpikingBrrain consists of three core components: hybrid efficient attention (H), MoE modules (M), and spike encoding (S). Together, these components enable compelitive performance, improved long-contexl efficiency, hardware adaptability, and potential energy-effiiencc benefits.
For comprchensive ablation analysis, we evaluate seven possible combinations of these components: H, M, S, H+M, H+S, M+S, and H+M+S. The components are defined as follows:
Hybrid efficient attention (H): inter-layer hybridization of linear attention and sliding- window attention with a 1:3 ratio.
MoE modules (M): sparse mixture-of-experls with 4 routed experts (top-1 routing) and one shared expert.
Spike encoding (S): adaptive-threshold spike encoding applied to activations.
38

---

Published in Transactions on Machine Learning Research (03/2026)
H+M+S: the full SpikingBrain model.
We evaluate both model performance and long-context efficiency for all configurations, together with the Qwen3-1.7B baseline.
Table 13: Component ablation study of SpikingBrain. Note: Values in perentheses indicate configura- tions where spike encoding uses integer activations on GPUs rather than neuromorphic hardware. In this setting, TTFT identical to the corresponding non-spike configuration.
Model
HellaSwag
MMLU TTFT-64k (ms) TTFT-128k (ms)
Baseline
60.21
61.78
3805.6
12553.3
H
57.88
56.08
2239.7
4729.4
M
60.25
61.88
4880.7
14230.1
59.63
61.17
(3805.6)
(12553.3)
I++M
57.93
56.05
2558.2
5055.2
H+S
57.37
54.87
(2239.7)
(4729.4)
M+S
59.66
59.22
(4880.7)
(14230.1)
H+M+S
57.77
55.02
(2558.2)
(5055.2)
The results in Table 13 highlight the distinct contributions of each component:
Hybrid efficient attention is the dominant contributor to long-context efficiency gains, reducing TTFT by more than 2x, albeit with some performance degradation.
MoE modules improve model accuracy with relatively modest efficiency impact. this ablation setting, the performance gains from MoE are moderate due to the limited conversion data, but larger gains are expected with increased training scale.
Spike encoding consistently incurs only limited performance degradation across configurations, while offering potential energy-effciency benefits under appropriate hardware assumptions.
Overall, these results suggest a modular adoption strategy: Hybrid attention is the primary choice when long-context elficiency critical; MoE can be incorporated to further improve model performance; and spike encoding particularly suitable for deployments targeting event-driven or asynchronous hardware.
C Hardware Implementation of Spike Coding
As described Section 3.3 the proposed spike coding scheme converts continuous-valuue activations into two oquivalent representations: integer spike counts and expanded sparse spike trains. The integer formulation enables efficient execution conventional GPU hardware, while the spike-train formulation naturally supports event-driven execution paradigms.
When used purely for numerical simulation, this representation requires no specialized hardware support and serves primarily for algorithmic validation. However, achieving practical performance and energy- efficiency benefits requires careful hardware-aware implementation. this section, we discuss implementation considerations for both GPU platforms and asynchronous neuromorphic hardware.
C.1 GPU-based Implementation
On GPUs, acceleration is achieved by leveraging INT-type quantized kernels. In SpikingBrain, this is enabled by INTS quantization of weights (Section 5.5). allowing spike-count-basee integer operations to be exccuted efficiently using existing GPU kernel optimizations. This formulation preserves compatibility with conventional deep learning software stacks while enabling efficient computation through integer arithmetic and reduced memory bandwidth.
39

---

Published in Transactions on Machine Learning Research (03/2026)
C.2 Asynchronous Neuromorphic Hardware Implementation
The energy-elficiency estimates presented in Section 5.5 are based on idealized assumplions regarding asynchronous hardware behavior. In practice, the realized efficiency depends on hardware architecture, memory organization, and control mechanisms. Achioving effective energy savings therefore roquires careful algorithm algortthm-hardware hardware co-design. Below, we discuss key hardware challenges and possible implementation strategies.
Hardware challenges underlying the theoretical estimates:
Memory access granularity. The analysis assumes channel-level skipping. In practice, DRAM/SRAM operate on block granularity; active and inactive channel weights are co-located, memory fetches cannot be avoided, reducing effective savings.
Asynchronous control overhead. The estimation excludes the energy and area cost of asyn- chronous control logic (c.g,, handshake protocols, spike detection, event routing), which introduces additional signal switching absent synchronous designs.
Dynamic spike sparsity. The average spike count 1.13 is dalaset- and layer-dependent. Variations in sparsity across inputs and layers may reduce efficiency relative to the estimated average. Hardware designed for fixed average sparsity will lose energy efficiency low sparsity scenarios.
Directions toward practical realization:
Algorithm-hardware co-design for memory access. Conduct offline profiling to identify sparsity pattern of SpikingBrain. On the software side, algorithn-level weight rearrangement and wight compression (e-g., run-length encoding) could be performed; on the hardware side, channel-aware memory controller could be designed and channel-level memory partitioning could be implemented to isolate weights of active and inactive channels into separate memory blocks, enabling effective skipping of inactive channels.
Low-overhead asynchronous control. Adopt lightweight asynchronous protocols reduce control overhead. For instance, click controllers and two-phase handshake protocols can minimize unnecessary signal switching. Additionally, an adaptive scheduler could be implemented to monitor spike sparsity in real time. The scheduler dynamically decides whether to merge low-activity channels or expand computing resources, further optimizing energy consumption.
End-to-end hardware modeling and optimization. In hardware development, the theoretical estimation model is gradually revised by incorporating previously omitted overheads derived from hardware simulation into the model. The revised model can be used to guide hardware design and algorithm optimization, ensuring that the balance power, performance and area achieved, energy efficiency claims are consistent with practical implementation results, and a complete estimation of end-to-end energy consumption is realized.
40


Reviews:
[{"review_id": "MwHdMjWBjW", "reviewer": "Reviewer_WxpP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission introduces SpikingBrain, a family of LLMs designed to address efficiency bottlenecks in traditional Transformers, particularly regarding long-context processing and deployment on power-constrained hardware. The authors propose a \"conversion-based\" training pipeline that adapts a pre-trained Transformer (Qwen2.5-7B) into a brain-inspired architecture.\n\nStrengths\n- The conversion-based training pipeline, even though not entirely novel, aligns with the trend of recent training recipes, and is a practical path to adapt pretrained full attention models to subquadratic variants rapidly.\n- The implementation on a non-NVIDIA hardware platform is interesting. However, most of the techniques presented don't seem novel. It would be great if the authors could provide a clearer specification of which techniques are merely implementations of existing work and which are the innovations presented in this paper. The MetaX GPUs and MACA frameworks are under-discussed, which makes it hard to appreciate the authors' engineering efforts (e.g., in reimplementing operators in MACA).\n\nWeaknesses\n- Experimental rigor. Accuracy comparisons are not data-matched, making the takeaways unconvincing. Important evaluations (e.g., recall) are missing.\n- Clarity and presentation issues. Table 1's presentation is dangerously misleading. Multiple typos throughout the paper.\n- Limited novelty. The current writing doesn't do justice in highlighting the authors' novel contributions, and reads more like a technical report on the implementation efforts in training what is essentially a Hybrid + MoE + spike coding model.", "claims_and_evidence_yn": "No", "claims_explanation": "- When comparing model architectures, it's important to ensure a fair comparison by ensuring that evaluations are data-matched [1,3]. However, Table 1 is an inherently unfair comparison. For example, Falcon-Mamba is a pretrained model, whereas SpikingBrain-7B is adapted from an existing model, therefore adopting a completely different training pipeline. \n  + Similarly, \"...competitive performance with only ∼2% of the training data typical for mainstream LLMs\" is also a slightly exaggerated claim, since SpikingBrain's training is converting an existing base LLM, not pre-training from scratch. I recommend that the authors address this framing.\n- A known issue with subquadratic models is their relatively weak recall capabilities [1,2], which are not present in this paper. I acknowledge that the authors evaluate on LongBench, but the dataset is more for long-context understanding and not designed particularly to evaluate on recall.\n- Subquadratic attention and hybrid models have extensive prior art (e.g., Qwen3-Next, Kimi-Linear, Nemotron-H) that is neither mentioned nor quantitatively compared with.\n- Long-context evaluations (e.g., 4M sequence length with 128 GPUs) should include more details on experiment setups (e.g., hardware details, interconnect details).", "audience_yn": "Yes", "audience_explanation": "- The choice of integrating subquadratic attention both inter- and intra-layer is well-motivated to address long-context efficiency.\n- The conversion-based training pipeline, even though not entirely novel, aligns with the trend of recent training recipes, and is a practical path to adapt pretrained full attention models to subquadratic variants rapidly.\n- The paper makes an effort in presenting a thorough evaluation (e.g., stress-testing on 4M context).", "requested_changes": "- [Critical] Table 1 is dangerously misleading. The bolded numbers make it seem that SpikingBrain-7B achieves the best performance across benchmarks. However, upon close examination, it is inferior on many tasks (e.g., Qwen2.5 is better across all workloads!). I recommend that the authors fix it by bolding the numbers that represent the best performance.\n- [Critical] Include additional evaluations on the recall capabilities.\n- [Critical] The abstract claims stable training with MFU at expected levels. However, section 5.4 only reports the MFU of 23.4% on the MetaX cluster, with no mention of the \"expected MFU\" on NVIDIA equivalents. It would be great to add quantitative comparisons.\n- [Critical] Writing changes that address my prior concerns.\n- [Critical] Address these typos and presentation issues:\n  + Typos: There are multiple places where \"Hybrid\" is spelled as \"Hybird\".\n  + Nit: HellaSwag is abbreviated as \"HS\", which is uncommon.\n- I think the paper would be greatly strengthened if the authors could include data-matched evaluations to make the takeaways more convincing.\n- The adaptive spiking mechanism is interesting. However, its associated implementation efforts (e.g., specialized kernels) are under-discussed. It would be great if the authors could expand Section 4 so that it not only discusses the training details on the MetaX cluster.", "broader_impact_concerns": "The work doesn't present notable ethical implications.", "additional_comments": "- I acknowledge that Qwen2.5 uses full attention and therefore is somewhat of an \"accuracy bar\" for linear/hybrid models -- and I appreciate the authors acknowledging the performance gap in the evaluation section -- but still, the performance gap is significant, especially when compared to other linear/hybrid models [1].\n- One other aspect is to add more details on what didn't work during the adaptation on non-NVIDIA hardware. Right now, the report reads as \"what ultimately worked\", and detailing what didn't would be helpful to the community.\n\n\nReferences\n1. Waleffe, Roger, et al. \"An empirical study of mamba-based language models.\" arXiv preprint arXiv:2406.07887 (2024).\n2. Arora, Simran, et al. \"Just read twice: closing the recall gap for recurrent language models.\" arXiv preprint arXiv:2407.05483 (2024).\n3. Allen-Zhu, Zeyuan, and Yuanzhi Li. \"Physics of language models: Part 3.1, knowledge storage and extraction.\" arXiv preprint arXiv:2309.14316 (2023).", "extra_scores": {}, "extra_text": {}}, {"review_id": "bycyq0NFtA", "reviewer": "Reviewer_gn98", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents SpikingBrain, a suite of large language models that incorporate hybrid efficient attention mechanisms (linear attention combined with sliding-window attention), MoE upcycling, and a novel adaptive-threshold spiking technique to achieve activation sparsity. A core contribution is the demonstration that both a 7B model and a 76B-parameter MoE model can be trained and deployed on MetaX, a non-NVIDIA GPU platform, using a conversion-based training pipeline that repurposes existing Transformer checkpoints. The authors report strong long-context efficiency, most notably >100× TTFT speedups at 4M-token contexts and near-constant memory inference, along with preservation of roughly 90% of the base model's capability. The work is technically ambitious with strong systems engineering depth, covering architecture design, distributed training optimizations, operator adaptations, and multi-platform deployment. However, several scientific claims require further empirical support, and some framing, particularly around biological analogies, should be moderated to better reflect the engineering nature of the contributions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Partially.\n\nThe submission provides convincing evidence for its systems-level and efficiency-related claims, but the support for several scientific and architectural claims is incomplete or insufficiently validated.", "audience_yn": "Yes", "audience_explanation": "Yes. \n\nAt least some individuals in TMLR’s audience would likely be very interested in the findings of this paper, for systems and engineering contributions, long-context LLM efficiency, spiking / neuromorphic inspiration", "requested_changes": "1. Provide Comprehensive Ablation Studies\n\nThe paper combines three distinct techniques (hybrid attention, MoE, and spiking) but does not disentangle their individual contributions. Please provide ablation experiments that isolate the impact of:\n\n• Hybrid linear/SWA attention alone\n\n• MoE upcycling alone\n\n• Spiking activation alone\n\n• Pairwise combinations of the above\n\nFor each configuration, report both performance metrics (MMLU, CMMLU, etc.) and efficiency metrics (TTFT, memory usage, throughput). This will allow readers to understand which component drives which benefit and guide practitioners on which techniques to adopt for their specific use cases.\n\n2. Clarification on “Training From Scratch”\n\nThe conversion framework claims ~2% of data is sufficient, but only because the model starts from a full Transformer checkpoint. Could the proposed hybrid architecture converge if trained from scratch with this budget?\n\n3. Moderate Biological Claims and Reframe as Engineering Contributions\n\nThe extensive emphasis on \"brain-inspired\" mechanisms and biological plausibility is not well-supported by the technical content. Claims such as \"linear attention models dendritic dynamics\" or \"adaptive threshold preserves biological plausibility\" are loose analogies rather than mechanistically accurate descriptions. The adaptive threshold V_th = mean(abs(x))/k is a practical statistical heuristic, not derived from biophysical principles. The neuroscience metaphors should be moderated or explicitly reframed as inspirational analogies, not mechanistic claims. There is no neuroscientific validation, and overstated claims may detract from the genuine engineering contributions.\n\n4. Provide MetaX performance metrics relative to NVIDIA\n\nSince MetaX operators are a key contribution, it would be helpful to include Direct performance comparison: MetaX C550 vs NVIDIA A100/H100 on identical model configurations, operator-level breakdowns showing where MetaX excels or lags. This will help readers assess the generalizability beyond MetaX-specific environments.\n\n5. Clarify near-term utility of spiking on today’s hardware\n\nGiven that spiking is simulated and introduces accuracy tradeoffs: What is the immediate value proposition for practitioners using current GPU stacks?\nA clearer statement of practical implications would strengthen the contribution.\n\n6. Long-context evaluation at 4M tokens\n\nBeyond TTFT throughput, please include qualitative or accuracy-based evaluations at million-token scales (e.g., Needle-in-a-Haystack, long-range QA) to confirm that the speedups do not come at the cost of severe retrieval degradation.\n\nMinor Recommendation\n\n1. Consider code and model release for reproducibility\n2. Consider adding a section on things that did not work.\n\nVerdict: Major Revision", "broader_impact_concerns": "No Concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zUoOfmMNNd", "reviewer": "Reviewer_WkSJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SpikingBrain, a novel brain-inspired LLM architecture and its training recipe on MetaX GPUs. The model architecture involves hybrid-linear attention with adaptive spiking neurons for efficient long-context training and inference. The new architecture is compatible with the standard transformer architecture and the architectures of many existing pre-trained LLMs: one can convert an existing pre-trained open-weight LLM into a SpikingBrain model with very little amount of fine-tuning. The authors also released their customized training and inference frameworks for LLMs on MetaX GPUs. The resulting SpikingBrain-7B model significantly improves on long-context efficiency over existing open-weight pre-trained LLMs while retaining most of the performance.\n\nKey strengths:\n- Proposes a novel brain-inspired LLM architecture that is compatible with existing mainstream pre-trained LLMs and therefore does not require pre-training from scratch which is computationally prohibitive for most academic researchers.\n- The proposed architecture enables efficient long-context training and inference. This is supported with strong empirical evidence.\n\nKey weaknesses:\n- Some descriptions about the training and evaluation recipe are somewhat vague and non-concrete, such as the bullet points on pages 8-9.\n- Some headline claims rely on indirect or hypothetical evidence. For example, the 100x TTFT speedup for 4M-token is estimated with extrapolated results on the baseline Qwen model. The energy efficiency claims on asynchronous hardware are also based on hypothetical and highly idealized estimations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core results and claims of the paper are mostly well-supported by convincing evidence. \n- The paper reports benchmark results for SpikingBrain models against a range of baselines, and describes the multi-stage CPT+SFT pipeline used to adapt from a Transformer checkpoint with substantially fewer tokens than training-from-scratch regimes.\n- The reported spike distribution characteristics supports the claim that the encoding yields meaningful sparsity.\n- Results in table 3 supports the claim that applying the spiking scheme retains most of the performance on benchmarking tasks.", "audience_yn": "Yes", "audience_explanation": "The findings of the paper is highly relevant to the machine learning community, especially researchers working with LLMs. The paper proposes a new type of LLM architecture with efficient conversion from existing open-weight pre-trained models, and introduces effective fine-tuning recipes of LLMs (with MoE and spike coding) on non-NVIDIA hardware. These contributions provides new insights for researchers to explore novel LLM architecture designs, and for practitioners to train LLMs with MoE more effectively.", "requested_changes": "Critical changes:\n- Provide more concrete details about the training and evaluation recipe. This is most applicable for the bullet points in pages 8-9, but also applicable in some other cases (like the bullet points on pages 15-16). For example, for \"perform[ing] long-context extension during conversion,\" how is it exactly done? When \"restrict[ing] the training length and resource usage of the original quadratic attention model,\" what is the exact amount of  compute/memory that you restrict the model to?\n- For Table 1 and Table 2, what does the bolded numbers refer to? I believe conventionally they refer to the best result in each row/column, but it does not seem to be the case here, and can be a bit misleading to the readers.\n\nNon-critical changes that strengthen the work:\n- Adding more ablations experiments that isolate contributions. E.g. performance of model converted without spike coding vs. with spike coding; effect of adaptive threshold hyperparameter k on sparsity/accuracy.\n- More details on MetaX training stability/utilization. How is the MFU (Model FLOPs Utilization) defined? What are the failure/restart rates across all the training runs?", "broader_impact_concerns": "The methods introduced by the paper enables more efficient training and inference of LLMs on non-NVIDIA GPUs and potentially new types of hardware. While it broadens the access of LLMs, it can also potentially increase risk by supporting more widespread deployment of LLMs without safety controls. Although this risk is not unique to this work, I would encourage the authors to add a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)