SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
Authors: Xingrun Xing
Boyan Gao
Zheng Liu
David A. Clifton
Shitao Xiao
Wanpeng Zhang
Li Du
Zheng Zhang
Guoqi Li
Jiajun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: spiking neural network
energy efficient language model
brain-inspired computing
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant energy and computational resources. In contrast, the human brain, with approximately 86 billion neurons, is much more energy-efficient than LLMs with similar parameters. Inspired by this, we redesign 7$\sim$70 billion parameter LLMs using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain. We propose the first spiking large language model, SpikeLLM. Coupled with the proposed model, two essential approaches are proposed to improve spike training efficiency: Generalized Integrate-and-Fire (GIF) neurons to compress spike length from $T$ to $\frac{T}{L} \log_2 L$ bits, and an Optimal Brain Spiking framework to divide outlier channels and allocate different $T$ for GIF neurons, which further compresses spike length to approximate $log_2T$ bits. The necessity of spike-driven LLM is proved by comparison with quantized LLMs with similar operations. In the OmniQuant pipeline, SpikeLLM reduces 11.01\% WikiText2 perplexity and improves 2.55\% accuracy of common scene reasoning on a LLAMA-7B W4A4 model. In the GPTQ pipeline, SpikeLLM achieves direct additive in linear layers, significantly exceeding PB-LLMs. Our code is publicly available at https://github.com/Xingrun-Xing2/SpikeLLM.

Published as a conference paper at ICLR 2025
SPIKELLM: SCALING UP SPIKING NEURAL NETWORK TO LARGE LANGUAGE MODELS VIA SALIENCY-BASED SPIKING
Xingrun Xingl22, Boyan Gao Zheng Liu"; David A. Clifton', Shitao Xiao*,
Wanpeng Zhang Li Du', Zheng Zhang', Guoqi Li Zhang
School of Artificial Intelligence, University of Chinese Academy of Sciences
2 Institute of Automation, Chinese Academy of Sciences
Beijing Academy of Artificial Intelligence Uniieesityy of Oxford Peking University zhengl iul 0266 6 gmaa i 1. com, guoqi 1i@ waa.ccccoom cn, jjzhang@n pr. par.ia.acc. a.aacoom on
ABSTRACT
Recent advancements in large language models (LLMs) with billions of parame- ters have improved performance various applications, but their inference pro- cesses demand sigmificant energy and computational resources. In contrast, the hu- man brain, with approximately billion neurons, is much more energy-efficient than LLMs with similar parameters. Inspired by this, we redesign 7~70 billion parameter L.Mss using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain. We propose the first spiking large language model, SpikeLLM. Coupled with the proposed model, two essential approaches are pro- posed to improve spike training efficiency: Generalized Integrate-and and-e-- (GIF) neurons to compress spike length from to I log, L bits, and an Optimal Brain Spiking framework to divide outlier channels and allocate different T for GIF neurons, which further compresses spike length to approximate log2T bits. The necessity of spike-driven LLM is proved by comparison with quantized LLMs with similar operations. In the OmniQuant pipeline. SpikeLLM reduces 11.01% % WikiText2 perplexity and improves 2.55% accuracy of common scene reasoning on a LLAMA-7B W4A4 model. In the GPTQ pipeline, SpikeLLM achieves direct additive in linear layers, significantly exceeding PB-LLMs. Our code is publicly available at https: / gi thub. com
wwt.ccoom/////eeee Xiig/nnngnunu Xi wingg///nnae /
INTRODUCTION
Recent Artificial Neural Networks (ANNs) have shown scaling up Large Language Models (LLMs) (Brown et al. 2020 Touvron et al.) 2023b Zhang et al. 2022a Le Scao et al. 2023) can be one of the most potential techniques to access Artificial General Intelligecee However, despite these unprecedented and promising achievements, steering LLMs imposes tremendous burden in terms of energy costs and computational requirements. For instance, running inference on the LLAMA- 2-70B model requires 2x A100-80 GPUs, and each energy consumption is 400W. This creates significant obstacle for deploying LLMs to real-world applications, especially where limited battery capacity and memory size are critical, such as in mobile devices (Xing al. (025). To lower these barriers and broaden the applications of LLMs, we focus on energy-efficient artificial intelligence.
Compared with ANN-based L.Mss human brain nervous systems achieve superior intelligence with much less energy consumption (Gerstner et al./ [2014] Izhikevich] 2003) and a comparable number of neurons, approximately 86 billion. For several decades, the brain-inspired computing (BIC) field (Mehonic & Kenyon 2022 Zhang et al. 2020) focuses on mimicking the biological nature of the human brain to develop more efficient and general Al algorithms (Maass) 1997) and physical platforms (Schuman et al.)
2022
Roy et al.) 2019, Pei et al., 2019). Among these, spiking neural Gerstner et al. 2014) are particularly notable for their biological
networks (SNNs) (Mass
1997:
Corresponding author

---

Published as a conference paper at ICLR 2025
Spiking Netivations
Quantited Activations
Mix Prodsion Quast.
Quantied Weights
Singlo Precision Quant.
Quastized Weighes
N lugT Bis
B-bit 4-bit
(N/T-11) logTT Bis 4 lagt Biss
Capy Clunch
B1
logT Bits
Spikk
Quenticanoon
Mix-grezisio Quem
Spika
Figure I: Different encoding methods. In (a, b), the activation has N channels; each value has T quantization levels. Given salient channels (in blue), mix mixxprrecssion methods (c) are deployment unfriendly. In spike-driven methods (d), we expand salient channels by spiking dynamics to realize single precision quantization, where T is spiking steps in salient channels.
Table I: Comparason of encoding efficiency. L indicates quantization levels in each step, where L E [1, The accuracy and perplexity are evaluated in the common scene reasoning and WikiText with the LLAMA-7B, W4A4 setting (Appendix A.2).
method
Steps
Bits/Step
Bits
Quant- Levels
Acc. 1
PPL L
IF-SNN
T
T
T
GIF-SNN
T
log2L
T 1, Flog2IL log2T -log2T
T
I.
Quant-ANN
I
log2T log2T
T
49.93
11.85
SpikeLLM
1
2 T
52.48
9.98
plausibility and binary event-driven efficiency (Yin et al.
2021| Schuman et al. 2022 Despite
their potential efficiency advantage, recent SNNs face two significant bottlenecks: (i) Firing Rate
Encoding (Bu et al.) 2023 Li et al. 2021 Deng & Gu 2021) in existing SNNs is inefficient in
capturing adequate semantic information. As shown in Table and Fig. [ (a. b). to express T
quantization levels, a typical Integrate- and-Fere (IF) (Liu & Wang 2001 Barbi et al. 2003) neuron
requires T time steps to encode full information, while quantization only requires logoT bit digits in one step. (ii) Inefficient Optimization. Direct training (Neftci al. 2019 Wu et al. 2018) SNNs need gradient estimation in BackPropagation Through Time (BPTT) because of non-differentiable
spiking dynamic; the ANN-SNN conversion (Han al.. 2020 Hao et al. 2023 Bu et al. 2023
Li et al., (02T) often requires much more inference steps to simulate ANNs, both of which are impractical for scaling up to LLMs. These challenges have kept SNNs relatively small (under B parameters) and hinder their ability to scale to the complexity of the human brain.
This work aims to scale up SNNs large language models or even part of the human brain nervous system with 86B neurons. Towards this goal, we propose both micro spiking neuron design to improve spike efficiency and macro saliency-based spiking design drive spiking neurons. Cur- rently, the primary approach to encode real-valued features to limited binary digits is model quan- tization. Given T quantization levels, quantization functions can efficiently encode features into log2T binary digits in step but encounter significant quantization errors for outliers salient values in low-bit conditions (Xiao et al.) 2023] Lin et al.) (2023b). Inspired by both quantized ANNs and SNNs, introduce a hybrid encoding method of quantization and spike firing termed Generalized integrate-and-Fin neurons and encode as Log2L bits as quantization; across different steps, keep the recursive spik- ing dynamics. Compared with IF neurons, GIF compresses spike length from I bits to Flogz L bits, where is the quantization levels in each step. Compared with quantization, GIF neurons main- tain recursive encoding advantages to accurately quantize salient channels with multisteps, which gives the potential to exceed quantization. To further determine channel-wis wise spiking steps in GIF neurons, as shown in Fig. [ (d), a saliency-based spiking mechanism is proposed to divide and conquer salient channels and almost non-salient ones. And then, we allocates multistep spiking to encode salient channels and one-step spiking for others, which further compresses to approximate log2T' bits over all channels. Unlike mix precisio quantization in Fig. (c), saliency-based spiking expands salient channels with GIF neurons and equals to single-precision quantization in Fig. (d).
encoding. (GIF) neurons. In each spiking step, we merge L. steps in IF
To detect salient channels, we propose a framework named Optimal Brain Spiking (OBSpiking), which is weight-activation generalization of the classic Optimal Brain Surgeon (OBS) (Hassibi & Stork [1992) for model pruning. The key concept of our method is distinguished approximations of weight and activation Taylor expansion calculate their saliency. In detail, the first-order gradient
2

---

Published as a conference paper at ICLR 2025
and second-order Hessian metrics are leveraged for activations and weights respectively. For every matrix multiplication in LLMs, GIF neurons can be viewed as generalized quantizers to quantize salient and other channels based on the saliency rank generated by the OBSpiking framework.
To evaluate the necessity of the Spiking LLMs, we integrate the proposed spiking mechanism with the most classic LLM quantization pipelines including the Omniquant (Shao al. 2023) and GPTQ, named SpikeLLMM. For weight-activation quantization (Shao et al.] (023)). we observe significant performance improvements in generation and common scene reasoning. For weight-only quantiza-
tion, GIF neurons further quantize temary weights with the GPTQ (Frantar et al. 2022) pipeline,
achieving direct addition networks. SpikeLLM exceeds PB-LLM (Shang et al. 2023) dramatically.
Our contributions are summarised as follows:
We first scale up spiking neuronal dynamics to 7~70 billion parameters, promoting SNNs to the era of LLMs. The necessity of introducing a spiking mechanism to LLMs is proved by the comparison with quantization methods.
We propose Generalized Integrate-and-Fiig neuron and Optimal Brain Spiking framework as a general alternative to traditional quantizers. The efficiency issue of firing rate encoding in SNNs is addressed by spike length compression.
The proposed GIF neuron and OBSpiking can be implemented by both general matrix multiplication (GEMM) and binary event-driven operations, providing the first practical design of SNNs with more than tens of billion parameters.
RELATED WORKS
Irain-Inspired Computing. The Brain-Inspired Computing (BIC) (Mehonic & Kenyon 2022 Zhang et al.] 2020) field focuses on building up bio-plausible and general fundamental theories, algorithms, software (Fang et al. 2023), and hardware platforms (Schuman et al. 2022 Roy et al. 2019 Pei et al. 2019) inspired by the structures and functions of biological nervous systems like the human brain. Spiking neural network (SNN) (Roy et al., 2019 Maass 1997 is one of the most popular BIC algorithms which embeds biological spiking neural dynamics in each single neuron
Yinn et al. 2021 Schuman et al. 2022). Promoted by the development of both deep learning and advanced biological neuron science, recent SNNs focus on the deep residual learning (Fang et al.
2021). self-attention (Yao et al.) 2023 Zhou et al. 2023), normlization (Zheng et al., 2021). as well as biological learning rules (Payeur al. 1), structures (Pham al., 2021) and energy efficiency
(Schuman et al.] 2002). In optimization, recent SNNs apply ANN-SNN conversion (Han et al. 2020
Hao et al. [223]] Bu et al., 2023) Li et al., (02T) or direct training (Neftci et al. 2019 Wu et al,
2018) techniques. Most SNNs focus on the computation vision field; Einguage-oriented SNNs are
almost less than billion parameters, for example, SpikeLM (Xing et al., 2024b), SpikingBERT
(Bal& & Sengupta 2024 Lv et al., 2023 [Su et al. 2024), and SpikeGPT (Zhu et al.,
2023). How to
scale up bio bio-inspired spiking mechanisms to billions of parameters has become a valuable issue.
Model Quantization. Model quantization aims at reducing the bit-width of weights activations to accelerate network inference exemplified by the recent quantization works including Post-Training
Quantization (PTQ) (Xiao et al. 2023] Frantar et al.] (022). Quantization Aware Training (QAT)
(Liu et al. 2023b), and calibration training methods (Shao et al. 2023 For small neural networks,
QAT methods (Esser et al. 2019 Liu al. 2022 Xing et al. 2024a; 2022b a). benefit from the
specifically designed training protocol, achieving outstanding performance. However, these meth- ods are not practical in the LLM setting. In contrast. PTQ methods including GPTQ (Frantar al.) 2022). GPTQ-ada (Hco et al., (023). SpQR (Dettmers et al. 2023), OWQ (Lee et al., 2023), AWQ (Lin et al. 2023a), and PB-LLM (Shang et al. (0023) tailored for LLM are weight-only quantization; SmoothQuant (Xiao et al.) 2023) and RPTQ (uaa et al.) 2023) achieve weight-activation quanti- zation. Besides, LLM-QAT (Liu et al.) 2023b), QA-LORA (Xu et al. (023), and calibration based methods including Omniquant (Shao et al. 2023), QLLM (Liu et al. 2023a), AffineQuant (Ma et al.) 2024). and QuaRot (Ashkboos et al. 2024) achieve higher performance. However, current quantized LLMs encounter significant quantization errors in outlier salient channels as discussed in Section 3.2. Althrough more recent rotation quantization (Ashkboos et al. 2024) achieves outlier-free. there
needs additional matrix multiplications that can not be absorbed.
3

---

Published as a conference paper at ICLR 2025
3 PROBLEM FORMULATION
In this section, we first introduce bio-inspired Spiking Neural Networks (SNNs) and explore the potential of replacing traditional quantized Large Language Models (LLMs) with spiking LLMs. Additionally, we address the challenges of outlier and salient value quantization, which motivate the introduction of the spiking mechanism.
3.1 SPIKING NEURONAL DYNAMICS
SNNs can be considered ANNs by adding biological spiking neuronal dynamics in each neuron. Without loss of generality, the biological soma dynamics are approximately modeled by the first- or higher-order differential equations. The IF neuron is a first-order approximation of soma dynamics, combining the advantages of bio-plausibility and efficiency, which can be represented by:
v(t) v(t 1) + [x (t) min (x(* (E]] s0
1)
S0, v(t < Via
8 ()) - i1, if 2 Vaa
(2)
x (1) (1) + min (x((-1)), where the IF neuron encodes binary spike in each step for a duration of T. In Eq 1 the membrane potential v(t) accumulates current input (()=))(t) to the last time step v(t 1) to simulate the charging process in soma. subjection reset applied to subtract the spiking values from the membrane potential v(t). In Eq if v(t) exceeds a certain firing threshold Vin the neuron is fired and encodes the spike s (t) as I; otherwise, encodes as O. Thus, previous SNNs take the IF neuron as an activation quantizer, which encodes free-prrcision activation as 1-bit output per spiking step in Eq. 3 Different from SNNs with ReL.U activations, to encode the negative values in transformers, we encode x (t) min (x(-))) by spike firing rate, where min (x (((1)) is the min value of this activation token, and can be viewed as the zero-por vero-point (Shao et al. 2023 of the quantizer.
(3)
As shown in Table [ compared with the asymmetric uniform quantization (Appendix A.1) that encodes T levels via logzT' bits, IF neurons recursively encode spikes via total T bits. because firing rate encoding directly makes an average of spiking steps, which missing the numerical carry. In ANN-SNN conversion (Bu et al., [021] Li al. 202T), ; IF neuron equals uniform quantization, where SNNs expand T steps of their Tog2T T bit quantized ANN counterpart. By summation over spiking steps, the IF neuron encodes the input into logzI' bit integer values, where 5 is the spike firing rate; Clip and Round indicate the min-max clipping and floor functions:
INT = =clpp (Round Ts()) .0.T).
(4)
3.2 LIMITATIONS OF TRADITIONAL QUANTIZATION
Traditional quantization is an ill-posed problem between bit-width and quantization error, especially for post-training L.Mss. The performance drop is often caused by quantization errors of outliers (iiao et al. 2023) and salient values (Dettmers et al., 2023). In details, the traditional LLM quanti- zation faces the following challenges:
(i weight-activation quantization makes it hard to avoid quantization errors in outliers. AWQ (Lin et al. 2023a) uses per-channel quantization step sizes to smooth outlier channels; however, it can only applied to weight-only quantization, which is often insufficient for LLM compression. (ii) Per-channel quantization is unfriendly to deployment. Since matrix multiplication is calcu- lated per-token, per-channel quantization cannot be directly used accelerate. As mitigation, SmoothQuant (Xiao el al. 2023) and OmniQuant (Shao et al. 2023) rebalance the quantization difficulty of activations and weights; however. they do not directly eliminate the impact of outliers. (iii) Mix- precision quantization is hardware unfriendly. SpQR (Dettmers et al. (023) and PB-LLM (Shang et al. 2023) use mixed-precisee quantization avoid quantizing salient weights; however, mix-precision introduces difficulties in hardware deployment.
Additionally, recent QuaRot (Ashkboos et al.) 2024) achieves outlier free quantization by rotation. The Hadamard transformation in Quarot maps activations to roughly ball-shape Gaussian distribu- tion, where more accurate spike encoding for larger values still benefits performance in our exper- iments. Moreover, compared with traditional quantization, QuaRot adds 4 Hadamard matrixes in each bloch, and their multiplications could not be absorbed. Based on these challenges, there is a requirement to explore spike encoding methods which helps accurate quantization.
4

---

Published as a conference paper at ICLR 2025
4 SPIKE-DRIVEN QUANTIZATION
To avoid inefficient spike encoding in previous SNNS and significant quantization error in Quan- tized ANNs, we propose the first spiking large language model with two techniques: Generalized Bnegrate-and-File neurons for spike length comparison and Optimal Brain Spiking framework for accurate saliency-bai baseed spike encoding. Compared with IF-SNNs, SpikeLLM compresses binary code length from approximate log2 T and can infer with both the low-bit matrix multiplication workload on GPUs and the per-bit computation workload on neuromorphic chips.
4.1 GENERALIZED INTEGRATE-AND FID-FFF NEURON
Because of the inefficiency of IF neurons, recent SNNs are almost less than billion parameters and are hard train with long-timg backpropagation through time (BPTT). On the other hand, traditional quantization encodes all binary digits in one step, which makes it hard to quantize outliers. Based on both aspects, we make a balance between recursive steps and code length in each step. This achieved by merging L spiking steps and encoding each step low-bit digits with loge L bit length. We define this L-step merged neuron as the Generalized Integrate-and and-Fire (GIF) neuron:
Σ str((), scIr(t)= = {K. K, 1=1
L
kV S Lv(t') (k 1)V & 0, 1, if 2 Va
(GIr(!)
(5)
where there are L quantization levels each spiking step. After merging, the recursive steps T T. = [f.1 and each step has been encoded by log2L bit quantization.
According to 5 if simulating 32 (T = 32) quantization levels (or spiking steps), we can set the merged spiking step T = 2 and spiking level L 16 in each step. After merging, each step is quantized to 4 x {0.1) digits, and the total 32 quantization levels can be represented by 8 x {0, digits. Although GIF neuron still has longer code than traditional quantization, this gives us the freedom to choose spiking steps according to channel saliency in LLMs and achieve higher performance with similar costs.
Remark 1. Ternary Spike. The ternary spike ster(t) proposed by SpikeLM (Xing et al. 2024b) is a special case of GIF, which is formulated by merging a positive IF neuron $Tr (1) and a negative (t). Ternary spike not only increases quantization levels but also keeps additive in SNNs.
-1,, if v() < - Veh
(1) = (1)=(())((()) + (T)))), (te((1)
0. if v() ( Vah +Vae)
(6)
(1, vf + +Ven
choose different workflows for training and inference: (i) both multi-bit training and inference on GPUs: as shown in Figure (d), GIF neurons can be converted
On-Chip Inference. Due the equivalence before and after spike merging in Eq.
angle-precisioon quantization and apply the general matrix multiplication kernels for inference, which equals to quantization but improves performance in Table D (ii) Multi-bit training on GPUs and 1-bit inference on neuromorphic chips: during training, merged spikes are employed, while during inference, the merged spikes are equiva- lent to expand back into their 1-bit formulation before merging. For example, if the value is a in a step, it can be decoded
spikes occurred in L time. This workflow avoids long-distance BPTT during training and therefore enhances training accuracy. We detial the training and inference of GIF neurons in Appendix A.4.
4.2 SALIENCY-AWARE SPIKING STEPS
Given the unequal importance of channels in LLMs, we propose saliency-aware spiking mech- anism to quantify this concept guiding the subsequent quantization process. This is addressed by expanding salient channels in activations portant channels. Given the GIF neuron to quantize activations, we first detect salient channels C' and the other channels C, and then allocate 7" -tep spikes encode channels in C" and one-step for
weights with more spiking steps compared with unim-
5

---

Published as a conference paper at ICLR 2025
Weights
Activation
CO3
380
Por-Channel Spiking
Per- Per-Chunmel Copy Weights
Por-Channel Gradient Salkeney
Cache
KV Cache
Activation
QQQ
Attention
Map
Per-Channel Copy
Per-Channel Spiking
Per-Chamnel Hession Saliency
Figure 2: saliency-Aware spiking mechanisms in SpikeLLM. (Left) Spiking self-attention. Salient channels in the KV caches are encoded by multi-step spikes. (Right) Spiking activations or weights in linear layer, where saliency is detected by gradient or Hessian metric respectively.
others in C, which can be represented by:
VA
(0) + min(x((-1)
T
V
( (1)sEC + + min(x((-1)), T
where VA max (x) is per-token spiking thresholds, which confirms clipping values (as T" Eq As shown in Fig. [ this per-channel spiking mechanism can apply to KV-caches, activations, and weights. For weight-activation quantization, salient channels in KV-ccches and activations have I" spiking steps, while the other side of the matrix multiplication keeps one-step quantization, and copies corresponding channels for the same T" steps. For weight-only quantization, the salient chan- nels in weights have 7" spiking steps, while the corresponding channels in activations are copied. Following, it is essential to detect salient channels both weights and activations.
4.3 OPTIMAL BRAIN SPIKING
Our Optimal Brain Spiking is a weight-activation generalization of the classic Optimal Brain Sur- geon (OBS) framework (Hassibi & Stork 1992). Different from OBS which focuses on weight pruning with only the second-order differentiation. activations and weights via both first and second-order differentiation. Given post-training model well-optimized under loss function L, any weights or activations in the model can be expressed by second-orde -order Taylor expansion around its optimal value x":
focus detecting salient channels in both
[(x) ~ [(x*) + (x- x*) VL((*) + (xx x*) " Hxxx"xxx x*),
where V((')) and (xx))) is the first-order differentiation and the second-order Hessian matrixes under the final loss C and we define d((xx) = [(x) L(x"). Specifically, for linear layer with weights W and activations X, we donate the quantization function Q(.) and we have:
Theorem 1. Optimal Brain Spiking. Given the layerwise objective to minimize the squared error; argmin) WX Q(X)) the activation saliency X W TW.. and the weight saliency where the Hadamard product.
Proof. For activations, the gradient is zero in a well-optimized model in Eq. and we use the first-order Taylor expansion to approximate the effect of activation perturbations dx: 5L(xx) = 5x V(x"). Thus, the activation salient matrix is directly calculated according 5((x)):
Saliency = W dL
OWX X W
WXX
WX
(9)

---

Published as a conference paper at ICLR 2025
017
644
647
2000
2000
2000
1350
2003
3000
(a) Per-Token Gradient Saliency
(b) Per-Channel Gradient Saliency
(c) Per-Channel Hessian Saliency
Figure 3: Comparisons of different saliency metries in the first linear layer. (a) Insignificant per-
token gradient saliency in activations. (b) Significant per-channel gradient saliency in activations.
(c) Significant per-channel Hessian saliency weights. The horizontal axis represents each channel.
Spiking settings to simulate quantization. We set 10% or 5% salient channels for 2 or 4 spiking steps in OBSpiking. Attention, Act. and Weight indicate where to apply OBSpiking.
Table 2:
Quantizer Spike-Level
L
Step T"
Salient Channels
Attention
Act.
Weight
4bit 4bit
16
10%
16
4
5%
/
/
2bit
4 2
10%
2bit
4
4
5%
For weights, the gradient is zero because the optimizer directly optimizes weights to the local mini- mum after pretraining. Thus, the first-order term zero in Eq and 8L(8w) has approximate via the second-order term in Taylor expansion: ((((w) jōw He((" [ww),w, which is proved in OBS
(Hassibi Stork 1992). And we apply the same weight saliency metric as OBS-based methods
(Shang al.) 2023
Frantar al. 2022
Frantar & Alistarh 2022):
W
Saliency(W) W
H 2Xx
(10)
R
Per-Channel Spiking Mechanism. Given saliency matrixes Saliency(X) and Saliency(W) (W) from Optimal Brain Spiking, per-channel means of first- (or second-) order differentation- basdd saliency are significant enough to divide salient channels in Eg[7
Remark 2.
In implentation, the saliency matrix Saliency(X) and Saliency W) have the same shape with activa- tions X and weights W, which are inefficient to store. As shown in Fig 3 we calculate per-channel per-token means of the saliency matrix. We observe that both the per-channel saliency in acti- vations and weights are robust enough to detect salient channels while per-token insignificant. Based Remark 2, first compute per-channel saliency with calibration data and generate their rank to select the salient channels in Eq 7 Then, we store these lightweight masks for inference.
5 EXPERIMENTS
We evaluate the necessity to introduce SpikeLLM due to the inefficiency and incompatibility caused by the direct application of existing ANNs based quantisation strategies spike neural networks. This necessity is studied and verified in two main directions: (i) general weight-activation quantiza- tion in very low bits; (ii) ternary quantized LLM towards additive linear layers.
Training. As shown in Table 2 SpikeLLM can simulate W4A4 (4-bit activation, 4-bit weight), W2A8, or W2A16 quantization with different spike-level L in GIF neurons, and the ratio of saliency channels in OBSpiking. We follow main streams post-training guantization (PTQ) (Frantar et al., 2022) and calibration (Shao et al. 2023) pipelines. (i) For the low-bit quantization setting, our primary baseline is OmniQuant (Shao et al.
including the spiking step T' and
2023) and we report the detailed pipeline in Appendix A.2. We select LLAMA-1-7B (Touvron et al. full-precision models. (ii) For the 2022) framework and report training
2023a) and LLAMA-2-7B, 13B, 70B (Touvron et al. 2023b)
additive SpikeLLM setting, we follow the GPTQ (Frantar et al.
details in Appendix A.3. Both pipelines the same 128 Wike Text2
WekelText2 calibration data for continue

---

Published as a conference paper at ICLR 2025
Table 3: Comparisons with quantized LLMs. LpikeLLM, is defined in Table 2 where T is the spiking step in saliency channels, the same T Table 2 OmniQuant trained with the unified
settings in Appendix A.2. Results are evaluated by the old version Im-eval, the same as Omniquant.
Method
Saliency #Bits
ACEs
PlQAnum
ARC-caum
Are-Cuen
BoolQ
HellaSwag.om
Winogrande
Avg.
LLAMA-1-7B
FP16
1x
77.37
52.48
41.38
73.12
72.99
66.93
64,05
SmoothQuant
W4A4
0.0625x
49.80
30.40
25.80
49.10
27.40
48.00
38.41
I.MMQQAT
W444
0.0625x
51.50
27.90
23.90
61.30
31.10
51.90
41.27
LLM-QAT+SO
W4A4
0.0625x
55,90
35.50
26.40
62.40
47.80
50.60
46.43
OS+
W444
0,0625x
62.73
39.98
30.29
60.21
44.39
52.96
OmniQuant
W4A4
x
63,22
40.99
29,86
62.23
50.77
$2.49
49.93
,pikeLLL,
0.10
W4A4
0.0688
65.45
41.67
32.51
64.37
56.59
54.3
52.48
LLAMA-2-7B
1
$3.58
71.07
67.25
63.74
OmniQuan
W4A4
0,0625x
61.75
40.32
29.86
60.89
48.86
$2.17
48.98
SpikeLLM,,. 0.10
W4A4
0.0688x
62.35
41.41
29,95
58.87
54.27
50.20
49.51
Omni Quant
W2A8
0,0625x
50.71
27.74
25.17
38.26
26.16
50.36
36.40
SpikeLLM,
0.05
W2A8
x.075x
53.37
31.27
23.63
53.79
33,94
51.46
41.24
Omni Quant
W2A16
x
57.02
32.83
25.34
53.46
31.50
50.36
41.75
Spikel.I.M,zz 0.10
W2A16
0.138x
65.67
41.88
28.41
60.46
49.87
52.80
49.85
LLAMA-2-13B
FP16
1x
79.05
57.95
44,28
69.02
76.63
69.61
66.09
Omni Quant
W444
0,0625x
68.01
46.09
32.51
62.87
57.71
53.91
53.52
SpikeL
0.10
W4A4
0.0688x
66.0
45.71
33.19
64.07
61.38
54.54
54.15
QuantQuat
0.0625x
23.55
61.31
49.41
42.71
Spikel.LMM
0.05
W2A8
0.075x
64.69
41.46
28.16
62.29
50.5
$2.49
49.93
Omni Quant
W2A16
0.125x
40.15
29.52
62.45
53.12
Spikel.LM,,. 0.10
W2A16
0.138x
67.68
44,53
30,38
63.64
$8.02
57.22
53.58
LLAMA-2-70B
FP16
1x
80.85
59.72
47.95
76.7
80.85
77.03
70.52
OmniQuant
W2A16
0.125x
68.44
45.16
32.59
63.12
58.32
52.25
53.31
Spikel.LM,..
0.10
W2A16
0.138x
75.84
51.68
39.42
66.67
68.44
60.22
60.38
Table Comparisons with QuaRot. We follow evaluation metries of QuaRot and all the results are evaluated by Im-eval-v00.4.2 following recent works. Saliency indicates the ratio of salient channels, and we set spiking step, =2= in salient channels.
Method
Saliency #Bits
PIQA.
ARC-ca.
ARC-Crem
BoolQ HellaSwagowe
Winogrande
Avg.
LLAMA-2-78
FP16
78.84
74.54
46.33
77.74
75.97
69.22
70.44
QuaRot-RTN
W4A4
71.82
59.89
36.18
67.37
63.88
59.12
59.71
SpiKeLLM-RTN
0.2
W4A4
72.47
62.29
36.01
69.48
64.74
59.43
60.74
QuaRot-GPt
W4A4
75.95
68.43
39.76
72.54
72.23
64.72
65.61
SpikeLLM-GPTQ
0.2
W444
77.37
70.88
41.81
73.00
72.42
65.11
66.76
LLAMA-2-133 QuaRoi-RTN
FP16
80.63
77.48
49.23
80.73
79.37
71.74
80.69
W4A4
69.19
41.98
72.54
70.35
64.72
65.61
Spikel. SpikeT.IMMRTTS M-RTN
0.2
W4A4
75.79
69.53
41.21
74.31
71.51
65.51
66.31
QuaRoi-GTOO L-
W4A4
77.91
72.18
46.16
78.41
75.55 76.40
68.82 68.03
69.84
0.3
W444
78.45
73.74
46.84
78.75
70.37
layer-wise training based on full- precision LLMs. It takes around 3.5 hours to train a 7B model with OmniQuant pipeline on single A100-80 GPU.
Evaluation. We follow the same evaluation methods as the primary baselines, OmniQuant (Shao 2023). We evaluate the perplexity (PPL) of language generation in WikiText2 (Merity et al. 2016) and C4 (Raffel et al. (2020) benchmarks. We also evaluate zero-shot common scene reasoning tasks including PIQA (Bisk et al., 2020). ARC-easy Clark et al.] 2018), ARC-challenge (Clark et al.) 2018). BoolQ (Clark et al. 2019). HellaSwag
al. 2023) and PB-LLM (Shang
Clark et al. 2018), and Winogrande (Sakaguchi et 2021 datasets.
Operation Metric. (i) For GEMM kernels, make direct comparison of SpikeLLM and quan- tized ANNs by introducing the ACE metric (Zhang et al., (022b) (Appendix A. which is defined the number of binary operations, ACE MAC'ss bifweight bitact.. where witweeigtt and bitast. indicates the bit-width of weights and activations respectively. (ii) For inference on neuro- morphic chips, consider the event-driven sparsity in SNNs. We introduce Sparse ACE, defined as spar sparsily x ACE, which evaluated after expanding GIF neurons back to 1-bit spiking.
5.1 MAIN RESULTS
Comparison with General Quantization. As shown Table BaleeBana and A.3.I we compare SpikeLLM
with the most general weight-activation quantization methods including SmoothQuant, LLM-QAT, and OmniQuant, which shows SpikeLLM improves significantly based on OmniQuant with a few additional spikes enhance salient channels. When replacing activation and KV-cache quantiza-
8

---

Published as a conference paper at ICLR 2025
SplctL W444
Radden-5ylb-W2224
Rasker-yyka
Ave. Aoe. wft..ffo..mm.aa....o
02
(n)
Percrrtgg Sallan Acthration Channck (7))
(b)
Percantage Saller Actratten Chanedx (%)
(c)
Perceatggs Satien Weight Chandde
Sptet WwWAA Oppraaaa Controteed
Propocts (PPL WiBhh012
Bapplath (PPE) W
21.
115
(8)
Perondegg Salline Artivation Cheeneh
(b)
(0)
Figure 4: Ablation studies in LLAMA-2-7B. Average accuracy (not norm) is reported. (a) Compar- ison between SpikeLLM and Quantized-ANN with the same operations. (b) Ablations spiking salient channels in activations and KV-Caches. (c) Ablations on spiking salient channels weights.
tion with spiking neurons, SpikeLLM-W4A4 LLAMA-7B improves 2.55% accuracy and reduces 11.01% perplexity. When replacing weight quantization, improvements is more dramatic: for the LLAMM---77B- W2A8 and W2A16, SpikeLLM,.. and SpikeLLM, exceed 4.84% and 8.10% re- spectively; 89.58% and 61.89% perplexity are reduced. Notice that, SpikeLLM can plug in almost quantized LLMs beyond OmniQuant, which indicates the potential higher performance equipped with stronger quantization pipeline. In Appendix A.5, we further explore more training pipelines, such as QLoRA (Dettmers et al. 2024) and efficient QAT (Chen et al. 2024 settings.
Comparison with QuaRot. Recent QuaRot (Ashkboos et al. 2024 achieves outlier-free quanti- zation with the help of additional Hadamard transformations. After rotation, the activation become a roughly ball-shape Gaussian distribution. In SpikeLLMM framework, we select the more salient channels given Gaussian distribution, and allocate more steps. As shown in Table E SpikeLLM improves 1.03% and 1.15% when quantizing weights by RTN and GPTQ respectively in LLAMA- 2-7B, indicating the ability to cooperate with rotation-based quantization.
Operation Efficiency Compared with ANNs. In Table 5 we evaluate the multi-bit and 1- bit workflows proposed in Sec. 4.1. Com- pared to LLAMA-7B, SpikeLLM, for the W4A4 setting saves x10.79 ACEs or x6.38 sparse ACEs respectively. Because GIF neu- rons expand much longer in 1-bit inference, it achieves asynchronous computation for the fol- lowing linear layer at the cost of more operations than GEMM kernels.
Table 5:
Operations of SpikeLLM. ACE and Sparse ACE evaluate operations on the GEMM kernel and event-driven neuromorphic chips.
Models
ACE (10"3)
Sparse ACE (10'?)
LLAMA-1-7B
3886.22
3886.22
SpikeLLM
360.20
608.77 78
SpikeLLM
386.70
648.09
Ablation Studies on GIF neurons. We evaluate the efficiency of GIF neurons by comparison with OmniQuant-W444 with almost the same operations. This is achieved by applying GIF neurons in both weights and activations. It is known that GIF neurons and OBSpiking always introduce a few additional operations. To confirm the same operations, we accordingly decrease the spike length in non-salient channels in weights and maintain the total spike length of weights and activations the same as OmniQuant-W44 W4A4. As shown in Fig 4 (a), given different percentages of salient channels in activations, Spikel.LM always exceeds the Omniquant- W4A4 baseline shown as the red line.
Ablation Studies on Optimal Brain Spiking. To evaluate the efficiency of the OBSpiking frame- work, we compare SpikeLLM with equal-operation baselines with randomly selected spiking chan- nels, termed Random-Spike. In Fig. (b), we compare three settings: (i) SpikeLLM-W4A4: OB- Spiking in activations of linear layers alone, (ii) MAAkeLLL-K--WAAL OBSpiking in both acti- vations of linear layers and KV caches of self-attentions. and (iii) Random-Spike in activations of linear layers alone. We don't 't evaluate the random-spike spike in KV-caches because this setting is unsta- ble optimize. With different percentages of salient channels, OBSpiking in both KV chches and
9

---

Published as a conference paper at ICLR 2025
Table 6: Comparisons between Sp and PB-LLM in LLAMA-2-7B towards additive linear
layers. SpikeLLM7es: x:y:2 indicates using the ternary GIF neurons in Eq as weight quantizers,
where x:y:z is percentages of 1, 2, 4 spiking steps in weights (details in Appendix A.3).
Method
ACs
Equal Steps
PIQA
ARC-e Arc-c BoolQ
HellaSwag
Winogrande
Avg.
PEsL PB-LLMa90
80%
2,4
60.77
43.9
22.18
64.16
33.75
56.83
46.93
90%
1.7
54.03
27.9
19.37
$7.09
27.12
48,38
38.98
PB-LLM
95%
1.35
53.43
26.6
19.28
51.87
26.51
49.01
37.78
SpikeLLM, 705, 70:25:5
100%
1,4
65.83
51.89
25.17
68.47
40.48
60.77
52.10
SpikeLLM,
80:13:5
100%
1.3
60.88
42.26
24.23
68.65
34.02
54.38
47,40
SpikeLLM
Tes, 85:10:5
100%
1.25
55.44
31.99
20.99
61.83
30.02
52.01
42.05
SpikeLLM
100%
1.2
53.75
28.83
19.2
37.92
28.46
48.38
36.09
Tes, 90:5.5
ln 3
Pupplcatt (PPLINCC
(a)
Aurrage EqualSopsie Waigles
(b)
Anerage Expud Steps Weight
(c)
Average Exuai Strps Weight
Figure 5: Effiency comparisons of SpikeLLM,e and PB-LLM in Wikitext-2, C4 and 6 zero-shot benchmarks. We use the average of equal steps as the operation metric SNNs and BNNs.
activations helps to improve performance. In Fig. (c), we compare Random-Spiin and OBSpiking in weight quantization, which proves the effectiveness for weights similarly.
5.2 ADDITIVE SPIKING LLMS
To direct maintain the addition nature and event-driven sparsity of SNNs without expanding GIF neurons, we additionally build SpikeLLM,e- based on the ternary GIF neuron in ol the spiking weight quantizer. After weight quantization Eq linear layers can be implemented by ACcumu- lation (AC). The detailed training and evaluation settings are reported in Appendix A.3.
Comparison with binary LLM. We select PB-LLM for comparison since binary weight neural networks (BNNs) can be implemented by ACs. As shown in Table 6 Spikel.LM achieves full AC operations in linear layers by saliency- -based spiking neuronal dynamics, instead of the deployment- unfriendly sexed-precision quantization in PB-LLM.
Efficiency of additive LLi In Table by equal-steps. For PB-LLM, we view the average bit-width as the equal steps; for SpikeLLM, the equal steps are calculated by the average spiking steps of salient and other values. As shown in Table 6, SpikeLLM is able to exceed PB-LLM with similar etual-steps. Further, in Fig we evaluate the Pareto front about equal-steps and performance, which shows Spikel.LM exceeds PB-LLM (BNN) in both effectiveness and efficiency.
evaluate the AC operations in linear layer
CONCLUSION
We propose the first spiking large language models with 7~700 billion parameters, promoting SNNs to the era of L.MMs. This is achieved by significant improvement of spike encoding efficiency, where the GIF neurons compress the code length from To Loog I. bits; the OBSpiking further compresses to approximate logzT' bits. Unlike previous ANN-SNN conversions that rely quantization, this work exceeds quantization with the hybrid encoding of both SNNs and quantized-AAl ANNs.. Scaling up spiking neuronal dynamics in LLMs and developing training efficient spiking mechanisms for tens of billion parameters have become increasingly valuable issues.
Limitations A primary limitation of this paper is the performance gap between Spiking LLMs and ANN LLMs. This work applies ANN-SNN calibration with only 128 calibration data and finishes training a 7B model in 3.5 GPU hours. Continuous pretraining would boost the performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by National Science and Technology Major Project (20232D0121504) This work is partially supported by CAS Project for Young Scientists in Basic Research (YSBR-116), National Natural Science Foundation of China (62325603, 62236009), Beijing Science and Tech- nology Plan (Z241 (224110000222411),
REFERENCES
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated lims. arXiv preprint arXiv:2404.0044 2024.
Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11998-11006, 2024.
Michele Barbi, Santi Chillemi, Angelo Di Garbo, and Lara Reale. Stochastic resonance a sinu- soidally forced lif model with noisy threshold. Biosystems, (1(1-2):23-2) 2003.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 200..
Tong Bu, Wei Fang, Jianhao Ding, PENGLIN Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-aceuracy and ultra-low- low---aewwl spiking neural networks. In International Conference Learning Representations, 2021.
Tong Bu, Wei Fang, Jianhao Ding, Penglinn Ln Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-accuracy and lita--lowlatencc spiking neural networks. arXiv preprint arXiv:2303.04347, 2023.
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang. Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-awwe training for large language models. arXiv preprint arXiv:2407.11062, 2024.
Christopher Clark, Kenton Lee, Ming- Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905. 2019.
Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1rXhv:1803.05457. 2018.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.004470 2021.
Tim Dettmers, Ruslan Svirschevski. Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzumov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse- quanss-- repre- sentation for near-lossless IIm weight compression. arXiv preprint arXiv:2306.03077 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1ms. Advances in Neural Information Processing Systems, 36, 2024.
Steven K Esser, Jeffrey I. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Leamndd step size quantization. arXiv preprint arXiv: /902.099900000 08153, 2019.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069 2021.
11

---

Published as a conference paper at ICLR 2025
Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 940) 1480. 2023.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475- 4488, 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From
single neurons to networks and models of cognition. Cambridge University Press, 2014.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 13558- 1355--1367,, 2020.
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between anns and snns by calibrating offset spikes. arXiv preprint arXiv:2302.10685 2023.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 1992.
Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and Dong- soo Lee. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. arXiv preprint arXiv:2309, 10t.:111991111,,, 2023.
Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on neural networks, 1466):1569-1572, 2003.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhycok Park. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint 2rXiv:23066.2222.. 2023.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International conference on machine learning, pp. 63 6316-6325, PMLR, 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for I1m compression and acceleration. arXiv preprint arXiv:2306.00978 2023a.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq; Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00998, 2023b.
Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qlim: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041 2023a.
Ying-Hu Hui Liu and Xiao-Jing Wang. Spike- adaptation of a generalized leaky integrate- and-fire model neuron. Journal of computational neuroscience, 10:25 2001.
Zechun Liu, Kwang- Ting Cheng, Dong Huang, Eric Xing, and Zhiqiang Shen. Nonuniform-to uniform quantization: Towards accurate quantization via generalized ttraigh-tthoouee estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 4952-495, 2022.
12

---

Published as a conference paper at ICLR 2025
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint 3rXIv22305.77888 2023b.
Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer trained with two-stage knowledge distil- lation from bert. arXiv preprint arXiv:2308. 15122, 2023.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang. Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint wrwiv:2403....... 12544, 2024.
Wolfgang Maass, Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1671, 1997.
Adnan Mehonic and Anthony J Kenyon. Brain-inspired computing needs master plan. Nature, 60447905):555-26,, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0784/ 2016.
Emre Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-base optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 2019.
Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake Richards, and Richard Naud. Burst-dependent synaptic plasticity coordinate learning in hierarchical circuits. Nature neu- roscience, 24(7): 1010-109, 2021.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip 2019.
architecture. Nature, 572(7767):106-111
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-16144, 2021.
Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xian- glong Liu, and Michele Magno. Accurate lora-finetuning quantization of Ilms via information retention. arXiv preprint arXivv2402.05445. 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2 (111):4485-55)) 2020.
Kaushik Roy. Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-base machine intelligence with neuromorphic computing. Nature, 575(7784))607-617, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 106, 2021.
Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, J Parker Mitchell, Bill Kay, et al. Oppor- tunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2(1):10-19, 2022.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large lan- guage models. arXiv preprint arXiv:2310.00034 2023.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang. Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.1311 13133 2023.
Qiaoyi Su, Shijie Mei, Xingrun Xing, Man Yao, Jiajun Zhang, Bo Xu, and Guoqi Li. Snn- bert: Training-efficicnt spiking neural networks for 106630, 2024.
bert. Neural Networks, 180:
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXtv:2302.1397, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 009888, 2023b.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio- backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Xingrun Xing, Yalong Jiang, Baochang Zhang, Wenrui Ding, Yangguang Li, Hongguang Li, and Huan Peng. Binary dense predictors for human pose estimation based dynamic thresholds and filtering. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1705-1709. IEEE, 2022a.
Xingrun Xing, Yangguang Li, Wei Li, Wenrui Ding, Yalong Jiang, Yufeng Wang. Jing Shao, Chun- lei Liu, and Xianglong Liu. Towards accurate binary neural networks via modeling contextual dependencies. In European Conference on Computer Vision, pp. 536-552. Springer, 2022b.
Xingrun Xing, Li Du, Xinyuan Wang, Xianlin Zeng, Yequan Wang, Zheng Zhang, and Jiajun Zhang. Bipft: Binary pre-trained foundation transformer with low-rank estimation of binarization residual polynomials. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16094-16102, 2024a.
Xingrun Xing, Zheng Zhang, Ziyi Ni, Shitao Xiao, Yiming Ju, Siqi Fan, Yequan Wang, Jiajun Zhang, and Guoqi Li. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms. arXiv preprint arXiv:2406.03287, 2024b.
Xingrun Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Wanpeng Zhang, Haokun Lin, Guoqi Li, and Jiajun Zhang. Efficientllm: Scalable pruning-aware pretraining for architecture- agnostic edge language models. arXiv preprint arXiv:2502.06663 2025.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization aware low-rank adaptation of large language models. arXiv preprint 2rXhv:2309.14711 2023.
Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, XU Bo, and Guoqi Li. Spike-driven transformer. In seirth- Conference on Neural Information Processing Systems, 2023.
Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classi- fication with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10): 905-913 2021.
Zhihang Yuan, Lin Niu. Jiawei Liu, Wenyu Liu, Xinggang Wang. Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint wxXiv:234.00108) 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 2rXiv:2205.01066 2022a.
Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp- 12475-12485, 2022b.
Youhui Zhang, Peng Qu, Yu Ji. Weihao Zhang, Guangrong Gao, Guanrui Wang, Sen Song, Guoqi Li, Wenguang Chen. Weimin Zheng, et al. A system hierarchy for brain-inspired computing. Nature, 586(788)))3398788355 2020.
14

---

Published as a conference paper at ICLR 2025
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with sirectly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11062-11070, 2021.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-based spiking neural net- work. arXiv preprint arXiv:2304. 1..3004.19944 11954, 2023.
Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint 23Xiv:2302.139933 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LOW-BIT QUANTIZATION
Low-bit quantization typically maps full-precision value to fewer quantization levels. We first focus on the most widely used asymmetric uniform quantization. Given the full-precision value x, the quantizer first maps x INT value by linear translation and Round functions, and then maps the discrete INT value back to its original range, which the former translation can be expressed as:
FP16 min(x
A max x) min(x)
X INT Round
(11)
A
2 2
In linear layers, quantized weights w or activations a can be represented as binary digits in M or N-1
N bits: a² Zi=0 MM-1 a;2*, .w Z==0 w,2'. Therefore, the Multppy-ACcuumllte (MAC) can be implemented by bit level AND and PopCount operations:
M-1N-1
d w
Σ
2 PopCount(ANDiaa wj)]-
Σ
2
(12)
This indicates the complexity and energy consumption of MAC operations are proportional to M x N. Therefore, the number operations in a quantized model can be measured using the arithmetic computation effort (ACE) metric (Zhang et al., (022b),, which is defined as M N for MAC operation between the M-bit weight and N-bit activation. Recent LLMs contain more than 10B ~ 100B parameters, making it an essential requirement push not only weights but also activations to lower bit-width.
A.2 TRAINING DETAILS OF THE OMNIQUANT PIPELINE.
A.2.1 TRAINING DETAILS.
We use unified training config as shown all of our experiments. We train LLAMA-1 (7B), and LLAMA-2 (7B, 13B, 70B) for both OmniQuant baselines and SpikeLLMS according to this training scheme. Compared with the original OmniQuant, we do not apply loss augmentation methods except for 70B models for training stability and we set the quantization group number I in all of the experiments. Therefore, this scheme can be viewed as simplified version without bells and whistles to focus the influence of the quantization method itself.
In SpikeLLM, we compute the saliency metric for activations layer by layer during the OmniQuant pipeline. Different from activations, we compute the saliency metric for weights direct using the features from the first embedding layer. Because we find this can make the computation of the inverse Hessian matrix more stable compared with computing layer by layer. Table 7: Training settings the OmniQuant scheme. LET and LWC indicate learnable equivalent transformation and learnable weight clipping.
config LET
4W4A
2W8A
2W16A
True
True
False
LWC
True
True
True
learning rate of LET
0.001
0.001
N/A
learning rate of LWC
0.01
0.01
0.01
activation smooth batch size
0.75
N/A
N/A
1
I
I
loss augmentation epochs group
False
False
True
20
20
40
1
1
1
A.2.2 TRAINING DATA.
Following OmniQuant (Shao et al. 2023). we randomly select 128 calibration data from WikiText2, which are 2048-token chunks. We further investigate the influence of different training samples to confirm the robustness of the proposed methods. In the OmniQuant pipeline, training samples are randomly cropped from the Wikitext2 dataset. To compare the performance of Spikel.LM and the
16

---

Published as a conference paper at ICLR 2025
OmniQuant baseline, Fig. we sample data and train 16 times, using the same seed for both OmniQuant and SpikeLLM each time. SpikeLLM achieves higher average accuracy 6 zero-shot datasets and lower Wikitext2 or C4 perplexity at the same time, showing consistently better performance. For the same training data, SpikeI.L. always performs better. In other experiments, keep the random seed
set of random seeds sample different training data each time. In
0.52
Arc
0,53
0.500
0.40
0.400
0.75
10.30
20:25
10.50
10.75
11:20
11.25
11.30
12.5
134
13.3
16.2
14.5
15.0
Figure 6: Comparison of different training data for LLAMA-1-7b W4A4 models. In SpikeLLMs, 10% activation channels are set to 2 spiking steps. Loss augmentations applied as OmniQuant Shao et al. 2023). The average accuracy (not norm) is evaluated.
A.3 TRAINING DETAILS THE TERNARY SPIKELLM.
A.3.1 MODEL DEFINITION.
In additive SpikeLLM, we apply the ternary spike level {-1,0++1} to encode weights according to Eq. [ After weight ternarization, the matrix multiplication in the linear layer be implemented by full ACcumulation (AC). As shown in Table g we have 3 spiking-step settings. For example, allocate the most salient 5% values in weights with
for the SpikeLLMM rez, 70:25:5 model, spiking steps, the following 25% with 2 spiking steps, and the rest 70% with spiking steps. The unstructured spiking steps are applied, which is different from the experiment in the Main Result Section. As the following section, this is because of more accurate quantization in ultra-low bit quantization and fair comparison with PB-LLM (Shang et 2023).
Table Model settings additive SpikeLLM. 1,2,4-Steps indicate the percentages 1, 2, and spiking-step encoding in weights respectively.
Model
1-Step
2-Step
4-Step
Avg. Step
Spike-Level
SpikeLLM
25%
1.4
{-1,0++1}
SpikeLLM, B0:150
80%
15%
5%
1.3
(-1,0++1
SpikeLLM 10:185:10:5
85%
10%
5%
1.25
[-1,0,+1]
SpikeLLM 90:515
90%
5%
5%
1.2
{-1,0++1)
A.3.2 STRUCTURED vs. UNSTRUCTURED SPIKING IN WEIGHTS.
For weight quantization, shown in Table 9 unstructured spiking steps usually achieve higher performance compared with structured ones. Our per-channel spiking scheme can achieve higher performance by setting per-channel spiking steps elementwise spiking steps. However, unstruc- tured conditions need additional masks and are less friendly to deployment. Therefore, we keep structured settings in low-bit quantization. But for additive LLMs, the performance is more impor- tant in the extreme case, and we choose the unstructured settings. Moreover, the PB-LLM baselines also unstructured, that, it can also confirm the fair comparison.
Table 9: Comaprison between structured and unstructured weight quantization in the OmniQuan pipeline.
Method
Saliency
#Bits
ACEs
PIQA
ARC-e
Arc-c
BoolQ
HellaSwag
Winogrande
Avg.
LLAMA-2-7B
FP16
1x
78.45
69.32
40.02
71.07
56.69
67.25
63.80
OmniQuant
W2A16
0.125x
57.13
35.02
53.46
29.32
50.36
0.10
W2A16
0,138x
65.61
48.15
27.39
60.46
39.01
52.80
48,90
SpikeLLM, Unstructured
0.10
W2A16
0.138x
72.63
60.06
30.89
65.05
48.52
59.51
56.11
17

---

Published as a conference paper at ICLR 2025
A.3.3 TRAINING DETAILS.
The same as PB-LLM (Shang et al. 2023), we follow the GPT-Q Frantar al. 2022 pipeline for the post-training weight quantization. In quantization, we keep the same block size of 128. We apply the same 128 calibration data as PB-LLMs. Each data is randomly selected from WikiText2 and tokenized to formulate 2048-token chunk.
A.4
SPIKING NEURONAL DYNAMICS
A.4.1
STEP MERGING IN GIF NEURONS
In Fig. 7 illustrate the function Eq. (5). After merging multiple steps (L) of the IF neuron, the multi-step spike become a floor function, which can activate multi-level firing. The membrane potential of the GIF neuron will fire according to Eq. (5). Finally, the output of GIF neuron is according to Eq. (3): x((-
(t) + min (x).
s(t)
V ch
Vah
v(t)
1.
Figure The floor firing function of the GIF neuron.
A.4.2 COMPARISON OF INFERENCE
(i) Neuronal Dynamics of IF Neurons (Fig. (a)): Given real-valued input sequence, the mem- brane potential accumulates over time. When it exceeds the firing threshold, the IF neuron emits spike, and the membrane potential reset by subtracting the threshold value.
(ii) Neuronal Dynamics of Multi-Step GIF Neurons (Fig. 8 (b)): Similar to IF neurons, GIF neurons accumulate membrane potential over time steps. However, unlike IF neurons, GIF neurons first integrate inputs over a fixed number of steps L. Based on Eq. (5), GIF neurons can fire k discrete levels of spikes, where [0, L).
(iii) Neuronal Dynamics of Binary GIF Neurons (Fig. 8 (c)): The only difference between multi-bit and binary inference of GIF neurons lies in the firing threshold. GIF neurons in multi-bit inference have multiple thresholds corresponding to different firing levels, while binary inference uses single threshold, similar to neurons. As result, binary inference emits one bit of spike per time step, as result, multi-level spikes are effectively decoded as binary. Next, we demonstrate the equivalence of binary inferenced GIF neurons and IF neurons.
Equivalence Conditions for Spiking Neurons:
(i) Given the same real-valued input sequence with firing rate encoding, the neurons are considered equivalent if the spike firing rates are identical.
(ii) For finite-step spikes, the membrane potential at the end a given period must also be the same to ensure consistent firing rates in subsequent steps.
Based on these conditions, since the binary GIF and IF neurons share the same inputs and firing threshold over short time window, their firing rates and membrane potential values are identical, confirming their equivalence in binary inference.
A.4.3 COMPARISON OF TRAINING
As shown in Fig. 9, we compare the training dynamics of conventional spiking neural networks (SNNs) and GIF neurons. The discussions follows.
18

---

Published as a conference paper at ICLR 2025
Real-Vatnol Eoput
Σ
2
3
1
2
3
Membrane Potential
2V th
2V th
V th
final Ven
final Veh
final
2
4
3
1
3
4
Spikke
2V
Σ
Vin
V ch
1
2
3
3
Original Binary Spike Encoding
(b) 4-Step Merged Spike Enceding
Binary Spike
Figure 8: Neuronal dynamics of GIF neurons. We compare the neuron input, membrane potential, and output spike in three conditions: (a) the traditional IF neuron. (b) Multi-step merged GIF neuron. (c) Binary inference of GIF neuron.
(8) Directly Training SNNs
BPTT Training
Real-Valued Input
Binary Spilks
4
1
2
3
4
5
Input
Truining
Binury quantization accumulated through time
L-step real-valued input before training
Decode L-level spikes to binury spikes after training
Real-Valned Input
L=4
Quantization- Aware Training
||] R Round(R R)
1
2
3
2
3
4
5
Fire [S] times
Σ
2.3
Merge steps
Input
Smaller quantizution error
1
2
4
5
1
2
3
5
Figure 9: Training dynamics of GIF neurons. We compare the traditional backpropagation through time (BPTT), quantization-awar training in GIF neurons.
(i) Fig. 9 (a): Traditional backpropagation through time (BPTT) for IF (or LIF) neurons suffers from significant quantization errors due to binary quantization at each time step. These errors accumulate over time, making SNNs difficult to train.
(ii) Fig. (b): GIF neurons quantization-ane -ware training. By aggregating L time steps, GIF neurons perform multi-bi guantization, which eliminates inter-step enror accumulation. The multi- bit quantization improves gradient estimation (STE), leading more accurate training.
19

---

Published as a conference paper at ICLR 2025
A.5
MORE RESULTS
A.5.1
PERPLEXITY IN OMNIQUANT SETTINGS
Table 10:
Comparisons between SpikeLLMM and OmniQuant in the Wikitext2 and C4 PPL met- rics. The LLAMA-2 family is evaluated. We do not evaluate the W4A4 and W2A8 settings for
LLAMA-2-70BB because the grouped-query attention (GQA) makes training unstable in the Omni-
Quant pipeline.
LLAMA-2-7B
LLAMA-2-13B
LLAMA-2-70B
Method
Saliency
#Bits
Wikitext2
C4
Wikitext2
C4
Wikitex12
C4
OmniQuant
W4A4
15.25
19.35
12.40
15.87
SpikeLLM 0.10
W4A4
11.93
15.34
9.64
12.48
SpikeLLM,.u
0.05
W4A4
11.85
15.18
12.17
15.71
OmniQuant
W2A8
287.64
445.21
53.87
72.33
SpikeL.LMpee
0.05
W2A8
29.98
46.61
12.80
17.06
OmniQuant
W2A16
38.05
98.74
17.14
27.12
10.04
19.31
SpikeLLMM
0.10
W2A16
14.16
19.73
9,45
13.86
6.12
8.82
A.5.2
COMPARISON WITH QLORA METHODS
This line of work focuses on adding a low-rank branch finetune the low-bit quantized model. State-of-the-art methods including IR-QLoRA (Qin et al. 2024), QA-LoRA (Xu et al. 2023). QLoRA (Dettmers et al. 2024). Before training, we calibrate saliency and acquire 2-bit weight spiking model with the EfficientQAT (Chen et al. (024) setting. We follow the same settings of QLoRA and A Alpaca finetuning.
Table 11: Performance comparison in QLoRA Settings,
Method
Data
#Bit
Hums.
STEM
Social
Other
Avg.
LLAMA-7B
33.3
29.8
37.8
38.0
34.6
NormalFloat
24.2
28.9
31.1
25.0
26.9
QLoRA (w/ GPTQ)
Alpaca
23.4
26.2
26.4
28.4
25.8
QLoRA
Alpaca
24.0
27.0
27.5
26.7
26.2
QA-LoRA
Alpaca
27.3
26.1
26.1
30.3
27.5
IR QLORA
Alpaca
26.0
27.8
30.2
28.3
27.8
Spikel..MQQL.OAA
Alpaca
31.5
27.5
29.5
32.0
30.3
A.5.3 COMPARISON EFFICIENT QAT SETTINGS
For low-bit bit weight-only quantizations, we train and evaluate SpikeLLM following EfficientQAT
(Chen al. 2024) (including training data and evaluation metries, Im-eval-v0...2). Both the Ef- ficienQQAT and SpikeLLM are trained in 2048 length in two stages with the same condition. We compare the 2-bit performance follows.
Table 12: Performance comparison in Efficient QAT pipeline.
Method
#Bit
HellaSwag WinoGrande
ArcC
ArcE
PiQA
Avg.
I.I.MA2--7B
69.22
57.18
43.17
76.09
78.24
64.78
GPTQ
2
55.17
32.59
21.25
40.45
58.32
41.56
OmniQ
2
55,88
40.28
23.46
50.13
65.13
46.98
AutoRound
2
61.01
40.28
32.25
65.99
72.96
54.50
AQLM2x8
2
65.27
49,96
32.85
66.92
73.07
57.61
EfficientQAT
2
65.19
49.53
35.15
69.74
73.67
58.65
LpikeLLLM
65.35
50.90
36.01
70.54
74.54
59.47
20


Reviews:
[{"review_id": "VUizPtToz5", "reviewer": "Reviewer_wFix", "summary": "The authors propose SpikeLLM, an energy-efficient large language model designed to tackle the increasing energy demands of current LLMs. They highlight two approaches namely, Generalized Integrate-and-Fire (GIF) neurons  and Optimal Brain Spiking framework to compresses spike length. The experimental results show improvements compared to the baselines used in the paper.", "questions": "1) Since this approach assigns different values of T corresponding to saliency of particular channels, can you please explain the temporal dynamics of the model (preferably during the on chip 1-bit inference phase) with a schematic diagram?\n\n2) In Table 6, SpikeLLM uses  ternary GIF neurons as weight quantizers. What is the quantization level of the activations?\n\n3) The results shown in Table 3 for SpikeLLM variants, all have activation quantization levels >= 4. Can we really call the models spiking?\n\n4) Can the authors please compare their results to current sota quantized LLMs.\n\nPlease see weaknesses as well.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well motivated and addresses a genuine problem of the increasing energy cost of LLMs. Experimental results show improved performance compared to the used baselines.", "weaknesses": "1) The temporal dynamics of the model is not explained in details. There is very limited results on the activity sparsity of the model, energy-efficiency comparison with other efficient LLMs (Table 5 only compares with LLAMA-1-7B)\n2) The authors can use better baselines for comparison such as 1.58 bit LLM [1], IR-QLoRA[2]. \n3) There is no experimental results showing 1-bit inference on neuromorphic chips. Even simulation showing spiking activity and convergence to similar result (as the merged spike) will demonstrate that we can use the model for efficient on-chip inference. No code has been added as part of the experiment. It is hard to understand how one would run the network in a 1-bit setting. If 1-bit (activity) inference cannot be shown experimentally then we cannot actually call this method spiking.\n\nReferences\n[1] Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J. and Wei, F., 2024. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764.\n\n[2] Qin, Haotong, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno. \"Accurate lora-finetuning quantization of llms via information retention.\" arXiv preprint arXiv:2402.05445 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KwXc8wzu2K", "reviewer": "Reviewer_VrDb", "summary": "This paper is the first to introduce SNNs into LLMs, an innovation that deserves high commendation and demonstrates the broad applicability of SNNs. Additionally, the authors combine quantization techniques with SNNs to propose GIF neurons and an OBS framework. Overall, I consider this paper to make a significant contribution.", "questions": "1. Lack of Logical Structure in Related Work: The related work section primarily lists existing studies without a systematic analysis of the development and challenges of SNNs and quantization methods.\n2. Unclear Formula Description: The explanation of Equation 5 (lines 239-244) is not intuitive, making it difficult for readers to understand the specific meaning of the GLIF model.\n3. Insufficient Visualization: There is a lack of intuitive graphical illustrations to demonstrate how the GLIF model divides long binary sequences into smaller segments over multiple time steps.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This study is the first to apply SNNs to large-scale language models with up to 70 billion parameters, filling a gap in the field.\n2.: The authors provide a good analysis of issues in quantized LLMs and propose a Spike-Driven Quantization method that effectively addresses these problems.\n3. The proposed GIF neurons leverage the time steps of SNNs to mitigate outliers that may result from direct quantization.\n4. The method is thoroughly trained and tested on LLAMA-2-7B and LLAMA-2-70B models, validating its effectiveness. To my knowledge, this is the first time that SNNs have been tested on a model as large as LLAMA-2-70B.", "weaknesses": "1\\I strongly suggest that the author takes the time to offer a comprehensive and meticulous description of the precise experimental results. It is essential to highlight the pivotal discoveries and thoroughly explore their potential ramifications within the broader context of the study.\n2\\It would greatly enhance the transparency and accessibility of the research if the author decides to open source the code. This action would not only foster a collaborative environment but also enable other researchers to replicate and extend the findings, ultimately advancing the field.\n3\\To fully ascertain the efficacy and superiority of the proposed method, it is imperative that the author conducts a more extensive series of experiments. These experiments should rigorously compare the performance of the current approach with various artificial neural network (ANN) alternatives, providing a comprehensive analysis of their respective strengths and weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PrdUjNUag3", "reviewer": "Reviewer_JYwS", "summary": "The paper introduces a post-training quantization scheme for large language models (LLMs). The quantization function is computed by a spiking neural network. The contributions over prior works are a reduction of the requires number SNN simulation steps by grouping time-steps into low-bit integer operations, handling outliers by allowing certain channels to conduct multiple simulation steps, and detecting outliers in weights and activations offline. Results are compared to the OmniQuant baseline and show improved downstream task performance on a standard evaluation protocol.", "questions": "- Is it feasible to compute the Hessian for such large models? Are any approximations used to get the saliency from the Hessian?\n- A related work paragraph on ANN-SNN conversion in the context of transformer based models would add value to the paper\n- The reviewer suggests to reconsider the use of the term \"auto-regressive\" as a synonym for recurrent, recursive or sequential. Auto-regression is a particular processing paradigm for sequences that aim to regress themselves, and is usually not interchangeably used with terms like recurrent or recursive. \n- The reviewer suggests to reconsider the claims on \"necessity for introducing SpikeLLM\" e.g. in lines 25, 110, 120, 370 in face of works like QuaRot that show that one get's away with simply rotating the weights.\n- It seems like the character \"\\~\" is out of place between 7B~70B in multiple places. Perhaps use `\\sim` instead\n- Please incorporate QuaRot and SliM-LLM in both the related work and the comparision", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is technically sound, well written and clear in its presentation. Tables 3 and 4 meet the standards of evaluation of the LLM community. Figures 1 and 2 assist the reader in following the presented method. The extend of the material is on par with recent quantization works that have been published recently in top ML conferences (e.g. [1], [2]). The authors compare their method on the standard evaluation benchmarks for LLMs reporting both downstream task performance and language modeling performance of large scale models from LLAMA 7B to 70B.\n\nANN-SNN conversion as a research field is of relevance to the ICLR community. Prior methods were not evaluated at this large scale. It is expected that ANN-SNN conversion methods would face similar issues regarding outliers as classical LLMs do. The authors use the core component of SNNs, i.e. temporal processing, to counter outliers and to compute channels with potential outliers in higher precision through longer simulation time. It is worth pointing out that using higher precision for such channels has been prominently studied in the literature (e.g. [3]), and saliency-driven calibration has been studied for quantized LLM (e.g. in [4]). Also ANN-SNN conversion based on integrate-and-fire neurons that model increased precision by conducting multiple steps was presented in prior works (e.g. [5], [6]). The core contribution of this paper is therefore the combination of these concepts to demonstrate > 7B parameter LLM quantization by handling outliers with increased simulation time budgets for certain channels. The paper therefore allows the comparison of rate-coded spiking neural networks against classical quantization methods in the large scale domain of LLMs.\n\n1. [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2211.10438)\n2. [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://arxiv.org/abs/2308.13137)\n3. [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)\n4. [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2405.14917)\n5. [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://arxiv.org/abs/2406.03287)\n6. [SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://arxiv.org/abs/2406.03470)", "weaknesses": "The fields of ANN-SNN conversion and LLM quantization are quickly moving. It is hence important to keep track of the recent developments in these fields. A weakness of this paper is that it is missing comparisons against some closely related recent papers.\n\n[1. SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://arxiv.org/abs/2406.03470)\nRelated work that uses integrate-and-fire neurons to implement low-bit quantization in transformers. Arguably this work does not work with as large models as the present paper.\n\n[2. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2405.14917)\nImproves over OmniQuant with saliency based metrics, similar to the ones proposed here. The reviewer is missing a comparison in tables 3 and 4.\n\n[3. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://arxiv.org/abs/2404.00456)\nShows that outliers can mostly be eliminated by mathematically equivalent parameterizations of LLMs (through appropriate weight matrix rotations). This work challenges some key claims made by the presented paper. E.g. line 370 claims \"necessity to introduce SpikeLLM due to the inefficiency and incompatibility [...] of existing ANNs based quantisation strategies to spike neural networks\". If outliers can be eliminated in LLMs, it seems likely that basic ANN-SNN conversion methods such as SpikeZIP would work in low precision even for larger LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jr4ZgA0c5D", "reviewer": "Reviewer_2u4J", "summary": "This paper proposes the first large-scale spiking large language model that can scale the spiking neuron dynamics to handle models with 7 billion to 70 billion parameters. It has the potential to address the energy and computational resource limitations typically associated with running large language models. The main contributions proposed by the authors include:\n\n1. Generalized Integral-Fire (GIF) neurons: This model compresses the spiking length by encoding the pulse train more efficiently. These neurons reduce the spiking length from T bits to T/L*log_2L bits. This novel approach improves coding efficiency while retaining the necessary computational power.\n\n3. Optimal Brain Spiking (OBS) framework:a saliency-based spiking mechanism that dynamically allocates spiking steps based on the importance of channels within the network. Channels that are considered more important receive more spike steps to capture information, while less important channels receive fewer steps. This saliency-based approach enables more efficient quantization and information encoding, further improving energy efficiency.\n\n4. Compatibility with modern hardware: SpikeLLM's GIF neurons and OBS framework can be deployed using both general matrix multiplication (GEMM) of traditional GPUs and event-driven neuromorphic chip architectures. This increases the feasibility of this method for deployment on neuromorphic hardware.\n\n5. Experimental results: Through various benchmarks, SpikeLLM has significantly improved both performance and energy efficiency over traditional quantization methods used in LLM, such as OmniQuant and GPTQ pipelines. For example, it reduces the perplexity on WikiText2 by 24.85% and improves the common sense reasoning accuracy by 2.01% compared to the LLAMA2-7B model with standard quantization.", "questions": "See weakness. Listed below:\n1.Please explain/correct Eq3, it seems confusing to me.\n2.In eq5, please fix the summation over t\n3.Fig4(a) missing baseline label\n4.Please provide a more detailed demonstration of the advantages of the proposed method.\n5.For SASS and OBP,  Does the saliency calculation need to be performed both during training and inference? etc\n6.how scalable is the approach proposed by the authors? etc\n7.What are the limits of both ann quantization and spiking llm boost? etc", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper attempts to solve the energy efficiency problem of LLM through SNN. This is a widely concerned issue. The author proposed several SNN based methods to solve the problems faced in traditional model quantization. The research problem is important and attractive, and the methods and experiments are of practical significance.\n\n2.The author proposed three core methods: GLIF, SALIENCY-AWARE SPIKING STEPS (SASS), and OPTIMAL BRAIN SPIKING (OBP). The logic is clear and easy to follow:\n-GLIF tries to encode more information in a single spike time step by merging L spiking steps\n-SASS distinguishes the importance of different channels and assigns different quantization levels to different channels to balance the computational load and performance\n-OBS determines the importance of any weights and activations through gradient methods.\n\n3.According to the author's experiments, the proposed pipeline works well and outperforms the quantized ANN of the same level in terms of energy efficiency and performance. The authors conducted extensive testing on both the diversity of experimental datasets and model sizes.", "weaknesses": "1.Please explain/correct Eq3, it seems confusing to me.\n2.In eq5, please fix the summation over t\n3.Fig4(a) missing baseline label\n\nMy main concerns are about the computational complexity of the proposed method. The authors did not elaborate in detail in the article on the computational cost of the proposed GLIF, SASS, and OPTIMAL BRAIN SPIKING. Specifically:\n\n4. For GLIF, In eq5, merging IF neurons of multiple steps into one step does reduce the requirement for SNN running time, but increases the computational cost of a single neuron; on the other hand, the implementation of neuromorphic hardware with multiple quantization levels is also unclear. Therefore, it is not sufficient to measure only auto-regressive steps here. Please provide a more detailed demonstration of the advantages of the proposed method.\n\n5. For SASS and OBP,  Does the saliency calculation need to be performed both during training and inference? What is the additional computational cost? How can KV-caches be combined with the proposed method to further reduce the amount of computation? According to my understanding, eq10 only needs to be calculated once, but eq7 and 9 must be performed during inference. (point out if I am wrong) Please  elaborate on the additional computational cost.\n\nExperiment result:\n\n6. In table4, Most of the experimental results for 70b models are missing. The authors suggest that GQA makes training unstable. It would be good to have a more detailed analysis showing the key reasons and potential research directions here. In other words: how scalable is the approach proposed by the authors? What specific issues need to be addressed for stable scaling to 70b or larger models?\n\n7. In table3, Although the proposed method is better than the traditional quantization method, the performance gap with the original model is still more than 10%, which limits the practicality of the method. The author proposed in Limitations: Continuous pretraining would boost the performance. What are the limits of both ann quantization and spiking llm boost? What is the performance difference with sufficient training time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)