SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking

## Abstract

Abstract The recent advancements in large language models (LLMs) with billions of parameters have significantly boosted their performance across various real-world applications. However, the inference processes for these models require substantial energy and computational resources, presenting considerable deployment challenges.
In contrast, human brains, which contain approximately 86 billion biological neurons, exhibit significantly greater energy efficiency compared to LLMs with a similar number of parameters. Inspired by this, we redesign 7~70 billion parameter LLMs using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain.
We propose the first spiking large language model as recent LLMs termed SpikeLLM. Coupled with the proposed model, a novel spike-driven quantization framework named Optimal Brain Spiking is introduced to reduce the energy cost and accelerate inference speed via two essential approaches: first (second)-order differentiation-based salient channel detection, and per-channel salient outlier expansion with Generalized Integrate-and-Fire neurons.
Our proposed spike-driven quantization can plug in main streams of quantization training methods. In the OmniQuant pipeline, SpikeLLM significantly reduces 25.51% WikiText2 perplexity and improves 3.08% average accuracy of 6 zero-shot datasets on a LLAMA2-7B 4A4W model. In the GPTQ pipeline, SpikeLLM realizes a sparse ternary quantization, which achieves additive in all linear layers. Compared with PB-LLM with similar operations, SpikeLLM also exceeds significantly. We will release our code on GitHub.

## 1 Introduction

Recent Artificial Neural Networks (ANNs) have shown scaling up Large Language Models (LLMs) [, , , ] can be one of the most potential techniques to access Artificial General Intelligence.
However, despite these unprecedented and promising achievements, steering LLMs imposes a tremendous burden in terms of energy costs and computational requirements.
For instance, running inference on the LLAMA-2 70B model requires three A100-80 GPUs, and each energy consumption is 400W.
This creates a significant obstacle for generalizing LLMs to real-world applications, especially where limited battery capacity and memory size are critical, such as in mobile devices. To lower these barriers and broaden the applications of LLMs, we focus on energy-efficient artificial intelligence.
Besides the traditional model compression algorithms [, ], solutions still largely remain open.

Compared with ANN-based LLMs, human brain nervous systems achieve superior intelligence with much less energy consumption [, ] and a comparable number of neurons, approximately 86 billion.
For several decades, the brain-inspired computing (BIC) field [, ] focuses on mimicking the biological nature of the human brain to develop more efficient and general AI algorithms [] and physical platforms [, , ]. Among these, spiking neural networks (SNNs) [, ] are particularly notable for their biological plausibility and binary event-driven efficiency [, ].
SNNs can be viewed as ANNs that substitute spiking neuronal dynamics in each neuron.
Despite their potential efficiency advantage, recent SNNs face two significant bottlenecks: (i) Firing Rate Encoding [, , ]. The firing rate encoding in existing SNNs fails to capture adequate semantic information efficiently. This limitation can hinder the generalization ability of LLMs, which heavily rely on acquired knowledge. (ii) Optimization. In direct training [, ], spiking dynamics are non-differentiable, and gradient estimation is inaccurate in backpropagation through time (BPTT). In ANN-SNN conversion [, , , ], it often requires much more inference steps to simulate ANNs, which is impractical for scaling up to LLMs.
These challenges have kept SNNs relatively small (under 1 B parameters) and hinder their ability to scale to the complexity of the human brain.

This work aims to bridge the scale gap between recent SNNs and ANN-based LLMs or even part of the human brain nervous system with 86B neurons; and develop efficient LLMs as an alternative to traditional model compression. To achieve this goal, we start by comparing traditional quantization and spiking neuronal dynamics. Traditional quantization functions efficiently encode binary digits in one step but has built-in drawbacks for outliers or salient values in low-bit conditions [, ]. On the other hand, the most popular Integrate-and-Fire (IF) [, ] neurons can auto-regressively encode more semantic information by more spiking steps but are much more inefficient in the original rate encoding method. This work views the auto-regressive spiking neurons as generalized quantizers and as an alternative to quantizing activations, weights, and self-attentions (query and KV-caches) in LLMs. A Generalized Integrate-and-Fire (GIF) neuron is defined to auto-regressively quantize salient outliers and address the efficiency issue in IF.
Based on GIF, spiking neurons own the capacity to express adequate semantic information which is essential to LLMs.

To drive spiking neurons, we propose an Optimal Brain Spiking framework to achieve salient-based quantization, which is a weight-activation generalization of the classic Optimal Brain Surgeon (OBS) []. Different from OBS, we detect salient channels in both activations and weights and allocate more spiking steps in GIF neurons for salient ones. After quantization, one salient channel is expanded as T channels by T spikeing steps, while most channels maintain one-step (or fewer-step) quantization.
We divide and conquer activation and weight saliency [] detection by different approximations of their Taylor expansion. In detail, the first-order gradient and second-order Hessian metrics are leveraged for activations and weights respectively. The Optimal Brain Spiking framework can cooperate with main streams of post-training quantization or calibration frameworks and a series of spike-driven quantized LLMs are built up termed SpikeLLM. For instance, for low-bit weight-activation quantization [], we observe significant performance improvements in WikiText2 [], C4 [] and 6 zero-shot datasets. To further achieve fully additive linear layers like previous SNNs, we conduct weight-only quantization similar to GPTQ [] pipeline. Compared with PB-LLM [] in similar cost, SpikeLLM exceeds dramatically.

Our contributions are summarised as follows:

- •
We first scale up spiking neuronal dynamics to more than 10B parameters. Spiking large language models with 7~70 billion parameters are proposed, and could become the alternative to more bio-plausible and energy-efficient AI.
- •
We propose a Generalized Integrate-and-Fire (GIF) neuron as a general alternative to traditional quantizers. The salient outliers are quantized auto-regressively and accurately.
- •
We propose an Optimal Brain Spiking framework to drive the GIF quantizers. Per-channel saliency is efficiently detected with the first (second)-order differentiation-based metrics. Compared with state-of-the-art quantization frameworks, significant performance improvements are achieved with spiking neuronal dynamics.

## 2 Related Works

Brain-Inspired Computing. The Brain-Inspired Computing (BIC) [, ] field focuses on building up bio-plausible and general fundamental theories, algorithms, software [], and hardware platforms [, , ] inspired by the structures and functions of biological nervous systems like the human brain.
Spiking neural network (SNN) [, ] is one of the most popular BIC algorithms which embeds biological spiking neural dynamics in each single neuron [, ]. Promoted by the development of both deep learning and advanced biological neuron science, recent SNNs focus on the deep residual learning [], self-attention [, ], normlization [], as well as biological learning rules [], structures [] and energy efficiency []. In optimization, recent SNNs apply ANN-SNN conversion [, , , ] or directly training [, ] techniques. Most SNNs focus on the computation vision field; language-oriented SNNs are almost less than 1 billion parameters, for example, SpikeLM [], SpikingBERT [], SpikeBERT [], and SpikeGPT [].

Model Quantization. Model quantization aims at reducing the bit-width of weights or activations to accelerate network inference. Recent quantization includes Post-Training Quantization (PTQ) [, ], Quantization Aware Training (QAT) [], and calibration training methods []. For small models, QAT methods achieve higher performance because of training from scratch, for example, LSQ [], U2NQ []. For LLM quantization, PTQ methods including GPTQ [], GPTQ-ada [], SpQR [], OWQ [], AWQ [], and PB-LLM [] are weight-only quantization; SmoothQuant [] and RPTQ [] achieve weight-activation quantization. Besides, LLM-QAT [], QA-LORA [], and calibration based methods including Omniquant [], QLLM [], and AffineQuant [] achieve higher performance.

## 3 Problem Formulation

To acquire quantized large language models, we start with recent low-bit quantization methods. Although traditional quantization is straight-through to encode weights and activations into binary digits, it is too simple to adapt to very low-bit conditions, resulting in a significant performance drop in LLMs, especially for activation quantization. Towards accurate and bio-plausible quantized LLMs, we replace direct quantization with spiking neuronal dynamics and pursue the first spiking large languages model inspired by advanced biological neuron science.

### 3.1 Low-Bit Quantization

Low-bit quantization typically maps full-precision value to fewer quantization levels. We first focus on the most widely used asymmetric uniform quantization. Given the full-precision value $\mathbf{x}$, the quantizer first maps $\mathbf{x}$ to a INT value by linear translation and Round functions, and then maps the discrete INT value back to its original range, which the former translation can be expressed as:

$$ ${\mathbf{x}}^{\text{INT}}=\texttt{Round}\lceil\frac{\mathbf{x^{\text{FP16}}}- \min(\mathbf{x^{\text{FP16}}})}{\Delta}\rfloor,\quad\Delta=\frac{\max(\mathbf{ x})-\min(\mathbf{x})}{2^{N}-1}.$ (1) $$

In linear layers, quantized weights $w^{q}$ or activations $a^{q}$ can be represented as binary digits in M or N bits: $a^{q}=\sum_{i=0}^{M-1}\mathbf{a}_{i}2^{i}$ $,w^{q}=\sum_{j=0}^{N-1}\mathbf{w}_{j}2^{j}$. Therefore, the Multiply-ACcumulate (MAC) can be implemented by bit level AND and PopCount operations:

$$ $a^{q}\cdot w^{q}=\sum_{i=0}^{M-1}\sum_{j=0}^{N-1}2^{i+j}\texttt{PopCount}[ \texttt{AND}(\mathbf{a}_{i},\mathbf{w}_{j})].$ (2) $$

This indicates the complexity and energy consumption of MAC operations are proportional to $M\times N$. Therefore, the number of operations in a quantized model can be measured using the arithmetic computation effort (ACE) metric [], which is defined as $M\times N$ for a MAC operation between the M-bit weight and N-bit activation. Recent LLMs contain more than 10B ~100B parameters, making it an essential requirement to push not only weights but also activations to lower bit-width.

### 3.2 Limitations of Traditional Quantization

Traditional quantization is an ill-posed problem between bit-width and quantization error, especially for post-training LLMs. In low-bit cases, the performance drop is often caused by quantization errors of outliers and salient values. Previous work has shown that outliers in activations have magnitudes over 100 $\times$ larger than most values, and salient values in weight matrices significantly impact the results. To more precisely quantize these values, previous methods such as AWQ [], SpQR [], SmoothQuant [], and Omniquant [] have proposed corresponding mitigation strategies. However, these methods are constrained by the limitations of traditional quantization frameworks:
(i) weight-activation quantization makes it hard to avoid quantization errors in outliers. AWQ [] uses per-channel quantization step sizes to smooth outlier channels; however, it can only be applied to weight-only quantization, which is often not enough for LLM compression.
(ii) Per-channel quantization is unfriendly to deployment. Since matrix multiplication is calculated per-token, per-channel quantization cannot be directly used to accelerate. As mitigation, SmoothQuant [] and OmniQuant [] rebalance the quantization difficulty of activations and weights; however, they do not directly eliminate the impact of outliers.
(iii) Mix-precision quantization is hardware unfriendly. SpQR [] and PB-LLM [] use mixed-precision quantization to avoid quantizing salient weights; however, mix-precision introduces difficulties in hardware deployment.
Based on these observations, there is an urgent requirement to explore weight-activation quantized LLMs and avoid the drawbacks of traditional quantization caused by outlier and salient values.

### 3.3 Spiking Neuronal Dynamics for Quantization

In this section, we first introduce the bio-inspired SNNs and then discuss the possibility of alternating traditional quantized LLMs with spiking LLMs.
SNNs can be considered ANNs with the addition of biological spiking neuronal dynamics in each single neuron. Without loss of generality, the biological soma dynamics can be approximately modeled using the first- or higher-order differential equations. The Integrate-and-Fire (IF) neuron is a first-order approximation of soma dynamics, combining the advantages of bio-plausibility and efficiency, and can be represented as follows:

$$ $\mathbf{v}(t)=\mathbf{v}(t-1)+\mathbf{x}^{(\ell-1)}(t)-\mathbf{s}^{(\ell)}(t)V _{th},$ (3) $$

$$ $\mathbf{s}^{(\ell)}(t)=\begin{cases}0,&\text{ if }\mathbf{v}(t)<V_{th}\\ 1,&\text{ if }\mathbf{v}(t)\geq V_{th}\end{cases},$ (4) $$

$$ $\mathbf{x}^{(\ell)}(t)=\mathbf{W}\mathbf{s}^{(\ell)\top}(t)V_{th}+\mathbf{W} \min{(\mathbf{x}^{(\ell-1)})}^{\top}+{b},$ (5) $$

where the IF neuron encodes a binary spike in each step $t$ for a duration of $T$.
In Eq.3, the membrane potential $\mathbf{v}(t)$ accumulates current inputs $\mathbf{x}^{(\ell-1)}(t)$ to the last time step $\mathbf{v}(t-1)$ to simulate the charging process in soma. A subjection reset is applied to subsect the spiking values from the membrane potential $\mathbf{v}(t)$.
In Eq.4, if $\mathbf{v}(t)$ exceeds a certain firing threshold $V_{th}$, the neuron is fired and encodes the spike $\mathbf{s}^{(\ell)}(t)$ as 1; otherwise, encodes as 0. Previous SNNs take the IF neuron as the activation quantizer. If we embed spiking neurons before a linear layer, this linear layer are derived as Eq.5 and the IF neurons perform asymmetric quantization. In Eq. 5, $\mathbf{W}\min{(\mathbf{x}^{(\ell-1)})}^{\top}+{b}$ formulates the bias of this linear layer.

Compared with the asymmetric uniform quantization in Eq.1 that encodes all binary digits at the same time, IF neurons auto-regressively encode per spike and can be viewed as a sequential expansion of the quantization function. In fact, IF neuron equals uniform quantization in some of the ANN-SNN conversion methods [, ]. In addition to previous ReLU activation-based conversion, we derive the equivalent forms of spiking linear towards asymmetric uniform quantization in Eq 5 and the quantized input of this layer can be represented as follows, where $\bar{\mathbf{s}}$ is the average of ${\mathbf{s}}(t)$:

$$ $\mathbf{x}^{\text{INT}}=\texttt{Clip}\left(\texttt{Round}\left\lceil{T\bar{ \mathbf{s}}^{(\ell)}}\right\rfloor,0,T\right).$ (6) $$

## 4 Spike-Driven Quantization

Towards accurate quantized LLMs, we conquer outlier and salient values via an Optimal Brain Spiking framework. Before that, we first define a Generalized Integrate-and-Fire (GIF) neuron as an alternative to a traditional quantizer, which is equivalent to inferring in binary event-driven or low-bit workloads. SNNs with 7~70B parameters are efficiently built up based on Optimal Brain Spiking.

### 4.1 Generalized Integrate-and-Fire Neuron

As shown in Eq.3, 4, the original IF neuron encodes one binary spike each step. To fully express FP16 values by the firing rate encoding, $2^{16}$ spiking steps are required, which is much more inefficient compared with 16-bit quantization.
Based on the inefficiency of IF spike encoding, recent SNNs are almost less than 1 billion parameters and are hard to train with long-time backpropagation through time (BPTT). On the other hand, traditional quantization encodes all binary digits in one step, which makes it hard to quantize outliers.
Based on both two aspects, we make a balance between auto-regressive steps and encoding length in each step. This is achieved by merging $L$ spiking steps and encoding each step as low-bit digits with $log_{2}L$ length. We define this L-step merged IF neuron as the Generalized Integrate-and-Fire (GIF) neuron:

$$ $\mathbf{s}_{GIF}(t)=\frac{1}{L}\sum_{t=1}^{L}\mathbf{s}_{IF}(t),\>\mathbf{s}_{ GIF}(t)=\begin{cases}\frac{k}{L},&\text{ if }kV_{th}\leq L\mathbf{v}(t)<(k+1)V _{th},k=0,1,...,L-1\\ 1,&\text{ if }\mathbf{v}(t)\geq V_{th}\end{cases}.$ (7) $$

In each spiking step, there are L quantization levels in the low-bit workloads, while GIF can also expand back to binary workloads as IF to become event-driven. In general, for larger merging-step $L$, low-bit workloads are more efficient when measured by ACE [], while for smaller, binary workloads are more efficient caused by event-driven sparsity.

Remark 1. Ternary Spike. The ternary spike $\mathbf{s}_{Ter}(t)$ proposed by SpikeLM [] is a special case of GIF, which is formulated by merging a positive IF neuron $\mathbf{s}^{+}_{IF}(t)$ and a negative $\mathbf{s}^{-}_{IF}(t)$. Ternary spike not only increases quantization levels but also keeps additive in SNNs.

$$ $\mathbf{s}_{Ter}(t)=\mathbf{s}^{+}_{IF}(t)+\mathbf{s}^{-}_{IF}(t),\quad\mathbf {s}_{Ter}(t)=\begin{cases}-1,&\text{ if }\mathbf{v}(t)<-V_{th}\\ 0,&\text{ if }\mathbf{v}(t)\in(-V_{th},+V_{th})\\ +1,&\text{ if }\mathbf{v}(t)>+V_{th}\end{cases}.$ (8) $$

Figure: Figure 1: Per-channel spiking mechanisms in SpikeLLM. (Left) Per-channel spiking self-attention. Salient channels in the KV caches are quantized by multi-step spikes, while the value and attention map copy part channels accordingly. (Right) Linears with per-channel spiking activations or weights. We detect salient channels by gradient or hessian metric for activations or weights respectively.
Refer to caption: extracted/5710913/fig.png

### 4.2 Optimal Brain Spiking

Traditional quantization leverages per-channel smooth techniques for activation outliers and mix-precision quantization for salient weights respectively. Thanks to the auto-regressive encoding in spiking neurons, we address accurate quantization for both salient activations and weights in a unified algorithm.
As shown in Fig. 1, we apply GIF neurons as quantizers for self-attentions (including the query and key-value caches), activations, and weights.

Based on the fact that not all channels are equally important in LLMs, a per-channel spiking mechanism is proposed to address outlier and salient value quantization issues in Section 3.2. This is addressed by expanding salient channels in activations or weights with more spiking steps compared with unimportant channels. Given the GIF neuron to quantize activations, we first detect salient channels $C$ and the other channels $C^{\prime}$, and then allocate $T$-step spikes to simulate channels in $C$ and fewer $T^{\prime}$-step (usually $T^{\prime}=1$) for others in $C^{\prime}$, and the spiking linear layer can be represented as:

$$ $\displaystyle{\mathbf{x}}^{(\ell)}$ $\displaystyle=\frac{V_{th}}{T}\sum_{t=1}^{T}\mathbf{W}\mathbf{s}^{(\ell)}(t)+{ \mathbf{W}\min(\mathbf{x}^{(\ell)})}+\mathbf{b}$ (9) $\displaystyle\simeq\frac{V_{th}}{T}\sum_{t=1}^{T}\mathbf{W}\mathbf{s}^{(\ell)} (t)|_{\mathbf{s}\in C}+\frac{V^{\prime}_{th}}{T^{\prime}}\sum_{t=1}^{T^{\prime }}\mathbf{W}\mathbf{s}^{(\ell)}(t)V_{th}^{\prime}|_{\mathbf{s}\in C^{\prime}}+ {\mathbf{W}\min(\mathbf{x}^{(\ell)})}+\mathbf{b},$ $$

where $V_{th}=\frac{\max{(\mathbf{x}_{C})}}{T}$ and $V_{th}^{\prime}=\frac{\max{(\mathbf{x}_{C^{\prime}})}}{T^{\prime}}$ are per-channel spiking thresholds in $C$ and $C^{\prime}$, which confirms not clipping max values (shown in Eq.6). As shown in Fig. 1, we apply this per-channel spiking mechanism in KV-caches and activations, or weights. If the salient channels in KV-caches and activations spiking T steps, the other side of the matrix multiplication keeps one-step quantization, and copies part of channels for the same T steps accordingly.
In turn, if the salient channels in weights spiking, the corresponding channels in activations are copied. Thus, it is essential to detect the salient channels in both weights and activations, and an Optimal Brain Spiking framework is proposed.

Our Optimal Brain Spiking is a weight-activation generalization of the classic Optimal Brain Surgeon (OBS) framework []. Different from OBS which focuses on weight pruning with only the second-order differentiation, we focus on detecting salient channels in both activations and weights via both first and second-order differentiation. Given a post-training model well-optimized under a loss function $\mathcal{L}$, any weights or activations $\mathbf{x}$ in the model can be expressed by a second-order Taylor expansion around its optimal value $\mathbf{x}^{*}$:

$$ $\displaystyle\mathcal{L}(\mathbf{x})\simeq\mathcal{L}(\mathbf{x}^{*})+(\mathbf {x}-\mathbf{x}^{*})^{\top}\nabla\mathcal{L}(\mathbf{x}^{*})+\frac{1}{2}( \mathbf{x}-\mathbf{x}^{*})^{\top}\mathbf{H_{\mathcal{L}}}(\mathbf{x}^{*})( \mathbf{x}-\mathbf{x}^{*}),$ (10) $$

where $\nabla\mathcal{L}(\mathbf{x}^{*})$ and $\mathbf{H_{\mathcal{L}}}(\mathbf{x}^{*})$ is the first-order differentiation and the second-order Hessian matrixes under the final loss $\mathcal{L}$ and we define $\delta\mathcal{L}(\delta\mathbf{x})=\mathcal{L}(\mathbf{x})-\mathcal{L}(
\mathbf{x}^{*})$. Specifically, for a linear layer with weights $\mathbf{W}$ and activations $\mathbf{X}$, we donate the quantization function as $\mathcal{Q}(.)$ and we have:

Theorem 1. Optimal Brain Spiking. Given the layerwise objective to minimize the squared error, $\text{argmin}||\mathbf{W}\mathbf{X}-\mathcal{Q}(\mathbf{W})\mathcal{Q}(\mathbf
{X})||_{2}^{2}$, the activation saliency is $\mathbf{X}\circ\mathbf{W}^{\top}\mathbf{WX}$, and the weight saliency is $\frac{\mathbf{W}_{ij}^{2}}{[\mathbf{H}_{ii}^{-1}]^{2}}$, where the $\circ$ is Hadamard product.

###### Proof.

For activations, the gradient is not zero in a well-optimized model in Eq. 10, and we use the first-order Taylor expansion to approximate the effect of activation perturbations $\delta\mathbf{x}$: $\delta\mathcal{L}(\delta\mathbf{x})\simeq\delta\mathbf{x}^{\top}\nabla\mathcal
{L}(\mathbf{x}^{*})$. Thus, the activation salient matrix is directly calculated according to $\delta\mathcal{L}(\delta\mathbf{x})$:

$$ $\displaystyle\quad\text{Saliency}(\mathbf{X})=\mathbf{X}\circ\mathbf{W}^{\top} \frac{\partial\mathcal{L}}{\partial\mathbf{WX}}=\mathbf{X}\circ\mathbf{W}^{ \top}\mathbf{WX}$ (11) $$

For weights, the gradient is zero because the optimizer directly optimizes weights to the local minimum after pretraining. Thus, the first-order term is zero in Eq.10 and $\delta\mathcal{L}(\delta\mathbf{w})$ has to approximate via the second-order term in Taylor expansion: $\delta\mathcal{L}(\delta\mathbf{w})\simeq\frac{1}{2}\delta\mathbf{w}^{\top}
\mathbf{H_{\mathcal{L}}}(\mathbf{w}^{*})\delta\mathbf{w}$, which is proved in OBS []. And we apply the same weight saliency metric as OBS-based methods [, , ]:

$$ $\text{Saliency}(\mathbf{W}_{ij})=\frac{\mathbf{W}_{ij}^{2}}{[\mathbf{H}_{ii}^{ -1}]^{2}},\quad\mathbf{H}=\mathbf{X}\mathbf{X}^{\top}.$ (12) $$

∎

Remark 2. Per-Channel Spiking Mechanism. Given saliency matrixes $\text{Saliency}(\mathbf{X})$ and $\text{Saliency}(\mathbf{W})$ from Optimal Brain Spiking, per-channel means of first- (or second-) order differentation-based saliency are significant enough to divide salient channels in Eq.9.

The saliency matrix $\text{Saliency}(\mathbf{X})$ and $\text{Saliency}(\mathbf{W})$ have the same shape with activations $\mathbf{X}$ and weights $\mathbf{W}$, which are inefficient to store.
As shown in Fig.2, we calculate per-channel or per-token means of the saliency matrix and find both the per-channel saliency in activations and weights is robust enough to detect salient channels while per-token is insignificant.
Based on Eq.11, 12 with a few calibration data, we first compute per-channel saliency and generate masks to select the most salient channels in Eq.9, and then store these lightweight masks for inference.

Figure: Figure 2: Comparisons of first- (or second-) order saliency metrics in the first linear layer. (Left) For activations, the first-order gradient saliency is insignificant in the token dimension. (Mid) For activations, the first-order gradient saliency is significant in the channel dimension. (Right) For weights, the second-order Hessian saliency is significant in the channel dimension.
Refer to caption: extracted/5710913/fig2.png

###### Proof.

## 5 Experiments

We evaluate both the effectiveness and efficiency of the brain-scale SpikeLLM with 7~70 B parameters from two aspects: (i) general weight-activation quantization in very low-bits is our main focus; (ii) towards additive large language models, binary or ternary quantization is also performed.

Meraging Steps and Spiking Steps. To simulate M-bit quantization, we set $L=2^{M}$ in the GIF neuron 7 to make it have the same quantization levels. For salient channels, GIF neurons auto-regressively encode for T steps, and each step has L levels.

Training Details. We follow main streams of post-training quantization (PTQ) [] and calibration [] pipelines. (i) For low-bit quantization setting, our primary baseline in OmniQuant [] and we keep the same training settings. In details, we randomly select 128 calibration data from WikiText2, which are 2048-token chunks. We select LLAMA-1 (7B) [] and LLAMA-2 (7B, 13B, 70B) [] as full-precision base models. The layerwise calibration is used and 4W4A (4-bit activation, 4-bit weight), 2W8A, or 2W16A quantizations are trained. We report full details pipelines in Appendix A. (ii) For the addition SpikeLLM setting, we follow the GPTQ [] framework using the same 128 WikeText2 calibration data. With the same setting as PB-LLM, we use LLAMA-2-7B and set a similar cost.

Evaluation Tasks. Following our primary baselines, OmniQuant [] and PB-LLM [], we evaluate preplexity (PPL) of language generation in WikiText2 [] and C4 [] benchmarks. As in previous works, we report zero-shot accuracy in PIQA [], ARC-easy [], ARC-challenge [], BoolQ [], HellaSwag [], and Winogrande []. The same evaluation methods with OmniQuant are applied.

**Table 1: Weight-activation quantization results of LLaMA Models. Saliency and ACEs are the rates of salient channels and the ACE metric for operations. OmniQuant${\dagger}$ indicates we retrain the OmniQuant of the LLAMA-1 model with the unified scheme (see Appendix A). $\text{SpikeLLM}_{\texttt{T=2}}$ and $\text{SpikeLLM}_{\texttt{T=4}}$ indicate the spiking time steps are 2 and 4 for salient channels respectively. We apply activation spiking for the 4W4A bit-width and weight spiking for the 2W8A or 2W16A bit-width.**
| Method | Saliency | #Bits | ACEs | PIQA | ARC-e | Arc-c | BoolQ | HellaSwag | Winogrande | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLAMA-1-7B | – | FP16 | 1$\times$ | 77.47 | 52.48 | 41.46 | 73.08 | 73.00 | 67.07 | 64.09 |
| SmoothQuant | – | W4A4 | 0.0625$\times$ | 49.80 | 30.40 | 25.80 | 49.10 | 27.40 | 48.00 | 38.41 |
| LLM-QAT | – | W4A4 | 0.0625$\times$ | 51.50 | 27.90 | 23.90 | 61.30 | 31.10 | 51.90 | 41.27 |
| LLM-QAT+SQ | – | W4A4 | 0.0625$\times$ | 55.90 | 35.50 | 26.40 | 62.40 | 47.80 | 50.60 | 46.43 |
| OS+ | – | W4A4 | 0.0625$\times$ | 62.73 | 39.98 | 30.29 | 60.21 | 44.39 | 52.96 | 48.43 |
| OmniQuant${\dagger}$ | – | W4A4 | 0.0625$\times$ | 63.28 | 46.38 | 27.56 | 62.23 | 40.24 | 52.49 | 48.70 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W4A4 | 0.0687$\times$ | 65.83 | 47.98 | 27.82 | 64.22 | 43.49 | 53.28 | 50.44 |
| LLAMA-2-7B | – | FP16 | 1$\times$ | 78.45 | 69.32 | 40.02 | 71.07 | 56.69 | 67.25 | 63.80 |
| OmniQuant | – | W4A4 | 0.0625$\times$ | 62.19 | 45.62 | 25.43 | 60.89 | 39.15 | 52.17 | 47.58 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W4A4 | 0.0687$\times$ | 64.47 | 48.74 | 27.30 | 63.27 | 43.29 | 56.83 | 50.65 |
| OmniQuant | – | W2A8 | 0.0625$\times$ | 51.90 | 27.82 | 19.97 | 38.26 | 25.85 | 50.36 | 35.69 |
| $\text{SpikeLLM}_{\texttt{T=4}}$ | 0.05 | W2A8 | 0.075$\times$ | 54.62 | 30.89 | 20.90 | 53.91 | 30.75 | 53.12 | 40.70 |
| OmniQuant | – | W2A16 | 0.125$\times$ | 57.13 | 35.02 | 21.16 | 53.46 | 29.32 | 50.36 | 41.08 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W2A16 | 0.138$\times$ | 65.61 | 48.15 | 27.39 | 60.46 | 39.01 | 52.80 | 48.90 |
| LLAMA-2-13B | – | FP16 | 1$\times$ | 78.78 | 73.36 | 45.56 | 68.99 | 59.73 | 69.61 | 66.01 |
| OmniQuant | – | W4A4 | 0.0625$\times$ | 67.03 | 53.96 | 30.55 | 62.91 | 44.83 | 53.91 | 52.20 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W4A4 | 0.0687$\times$ | 66.49 | 55.30 | 30.12 | 64.16 | 47.43 | 51.54 | 52.49 |
| OmniQuant | – | W2A8 | 0.0625$\times$ | 57.51 | 35.27 | 19.11 | 61.31 | 29.95 | 49.41 | 42.09 |
| $\text{SpikeLLM}_{\texttt{T=4}}$ | 0.05 | W2A8 | 0.075$\times$ | 64.09 | 48.74 | 24.23 | 62.29 | 40.38 | 52.49 | 48.70 |
| OmniQuant | – | W2A16 | 0.125$\times$ | 63.55 | 45.16 | 23.55 | 62.45 | 39.84 | 53.12 | 47.95 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W2A16 | 0.138$\times$ | 67.63 | 53.70 | 28.33 | 63.64 | 45.10 | 57.22 | 52.60 |
| LLAMA-2-70B | – | FP16 | 1$\times$ | 81.07 | 77.74 | 51.11 | 76.70 | 63.99 | 77.03 | 71.27 |
| OmniQuant | – | W2A16 | 0.125$\times$ | 62.57 | 43.86 | 22.78 | 56.42 | 39.60 | 52.49 | 46.29 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W2A16 | 0.138$\times$ | 76.44 | 66.92 | 38.31 | 66.88 | 51.86 | 59.19 | 59.93 |

**Table 2: Comparisons between SpikeLLM and OmniQuant in the same pipeline with the Wikitext2 and C4 PPL metrics. We do not evaluate the W4A4 and W2A8 settings for LLAMA2-70B because the grouped-query attention (GQA) makes training unstable in the OmniQuant pipeline.**
|  |  |  | LLAMA-2-7B | LLAMA-2-13B | LLAMA-2-70B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Saliency | #Bits | Wikitext2 | C4 | Wikitext2 | C4 | Wikitext2 | C4 |
| OmniQuant | – | W4A4 | 15.25 | 19.35 | 12.40 | 15.87 | – | – |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W4A4 | 11.36 | 15.87 | 9.71 | 12.10 | – | – |
| $\text{SpikeLLM}_{\texttt{T=4}}$ | 0.05 | W4A4 | 11.41 | 14.34 | 9.75 | 12.17 | – | – |
| OmniQuant | – | W2A8 | 287.64 | 445.21 | 53.87 | 72.33 | – | – |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W2A8 | 22.13 | 30.45 | 13.56 | 18.73 | – | – |
| $\text{SpikeLLM}_{\texttt{T=4}}$ | 0.05 | W2A8 | 28.78 | 44.80 | 12.80 | 17.05 | – | – |
| OmniQuant | – | W2A16 | 38.05 | 98.74 | 17.14 | 27.12 | 10.04 | 19.31 |
| $\text{SpikeLLM}_{\texttt{T=2}}$ | 0.10 | W2A16 | 14.16 | 19.73 | 9.45 | 13.86 | 6.35 | 9.62 |
| $\text{SpikeLLM}_{\texttt{T=4}}$ | 0.05 | W2A16 | 14.50 | 19.82 | 9.17 | 12.37 | – | – |

### 5.1 Low-Bit Quantization Results

As shown in Table 1, we compare SpikeLLM with state-of-the-art weight-activation quantization methods including SmoothQuant, LLM-QAT, and OmniQuant, which shows SpikeLLM improves significantly based on OmniQuant with a few additional spikes to enhance salient channels. For LLAMA-2-7/13B in Table 1, this performance enhancement is dramatic: for example, for the 2W8A and 2W16A quantization of 7B, improvements are 5.01% and 7.82% respectively with 20% and 10% additional spikes. In Table 2, their WikiText2 PPL of $\text{SpikeLLM}_{\texttt{T=2}}$ also significantly decrease 92.31% and 62.79% compared with baselines.

Next, we confirm the efficiency of SpikeLLM with almost the same operations compared with quantization. As shown in Fig.3 (Left), We increase average spiking steps in activations and decrease in weights to keep almost the same operations, which indicates SpikeLLM exceeds all of its equal-operation settings. To evaluate the efficiency of the Optimal Brain Spiking framework, as shown in Fig.3 (Mid, Right), we compare SpikeLLM with equal-operation baselines with randomly selected additional spiking channels in weights and activations respectively, which show stable better performance for SpikeLLM. The first- or second-order differentiation-based saliency methods are crucial for activation or weight quantization.
Based on these observations, we report our proposed Optimal Brain Spiking framework (as well as GIF neurons) is accurate and robust enough to enhance or even take place traditional quantization methods for both weight and activation quantization.

Figure: Figure 3: Effiency of Generalized Integrate-and-Fire neurons and Optimal Brain Spiking in LLAMA-2-7B. (Left) We keep the same computational cost in different GIF settings and compare it with the OmniQuant-4A4W baseline (in red). (Mid) We compare Optimal Brain Spiking with randomly increased spiking steps in activations with the same cost. SpikeLLM or SpikeLLM-KV indicates two settings: spiking activations or spiking both activations and KV-caches. We do not randomly increase spiking steps in KV-caches for the Random-Spike-4A4W baseline, because it is unstable and occurs N/A gradient. (Right) We compare Optimal Brain Spiking with randomly increased spiking steps in weights. SpikeLLM indicates increasing spiking steps as 2 in parts of weight channels.
Refer to caption: extracted/5710913/fig3.png

### 5.2 Additive Spiking LLMs

To confirm the additive computation and event-driven sparsity, as in previous binary spiking neural networks, we additionally build $\text{SpikeLLM}_{\texttt{Ter}}$ based on the ternary GIF neuron in Eq.8 as the spiking weight quantizer. We select PB-LLM for comparison since binary weight neural networks (BNNs) can be implemented by accumulating operations.
One of the biggest differences between SpikeLLM and PB-LLM is that the spiking neuronal dynamics auto-regressively spike more steps for salient values, instead of the deployment-unfriendly mixed-precision quantization in PB-LLM. In this section, we change the per-channel spiking to unstructured elementwise spiking steps, which is consistent with PB-LLM to improve performance. We further discuss the structured and unstructured spiking issues in Appendix B.
In Table 3, we set 3 different spiking steps, where the additional step mask is very small compared to PB-LLM because there are significantly fewer salient weights.

As shown in Table 3, thanks to the spiking dynamics, SpikeLLM gets rid of 8-bit salient outliers in PB-LLM and achieves higher performance. Thus, the fully additive linear layers are achieved. We use the equivalent spiking steps (Table 3) for the simplified operation evaluation and ignore the sparsity, in which the ternary spikes can be more sparse than binary ones via frequency encoding [].
In Fig.4, SpikeLLM exceeds PB-LLM (BNN) in both effectiveness and efficiency.
As the first trial, a large spiking neural network with fully additive linear layers is achieved.

**Table 3: Comparisons between SpikeLLM and PB-LLM in LLAMA-2-7B towards additive linear layers. $\text{SpikeLLM}_{\texttt{Ter,x:y:z}}$ indicates using the ternary Integrate-and-Fire neurons in Eq.8 as weight quantizers, where x:y:z is percentages of 1, 2, 4 spiking steps in weights. ACs indicate percentages of computation in linears that implement Multipl-ACcumulates (MACs) as ACcumulates (ACs). We use the Equal Steps as the uniform operation metric, where we view per bit in PB-LLM as an equal step. This is according to both binary- and ternary-weight multiplications are FP16 additions in implementation.**
| Method | ACs | Equal Steps | PIQA | ARC-e | Arc-c | BoolQ | HellaSwag | Winogrande | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\text{PB-LLM}_{\texttt{80\%}}$ | 80% | 2.4 | 60.77 | 43.9 | 22.18 | 64.16 | 33.75 | 56.83 | 46.93 |
| $\text{PB-LLM}_{\texttt{90\%}}$ | 90% | 1.7 | 54.03 | 27.9 | 19.37 | 57.09 | 27.12 | 48.38 | 38.98 |
| $\text{PB-LLM}_{\texttt{95\%}}$ | 95% | 1.35 | 53.43 | 26.6 | 19.28 | 51.87 | 26.51 | 49.01 | 37.78 |
| $\text{SpikeLLM}_{\texttt{Ter,70:25:5}}$ | 100% | 1.4 | 65.83 | 51.89 | 25.17 | 68.47 | 40.48 | 60.77 | 52.10 |
| $\text{SpikeLLM}_{\texttt{Ter,80:15:5}}$ | 100% | 1.3 | 60.88 | 42.26 | 24.23 | 68.65 | 34.02 | 54.38 | 47.40 |
| $\text{SpikeLLM}_{\texttt{Ter,85:10:5}}$ | 100% | 1.25 | 55.44 | 31.99 | 20.99 | 61.83 | 30.02 | 52.01 | 42.05 |
| $\text{SpikeLLM}_{\texttt{Ter,90:5:5}}$ | 100% | 1.2 | 53.75 | 28.83 | 19.2 | 37.92 | 28.46 | 48.38 | 36.09 |

Figure: Figure 4: Effiency comparisons of $\text{SpikeLLM}_{\texttt{Ter}}$ and PB-LLM in Wikitext-2, C4 and 6 zero-shot benchmarks. We use the average of equal steps as the operation metric of SNNs and BNNs.
Refer to caption: extracted/5710913/fig4.png

## Appendix A Training Details of the OmniQuant Pipeline.

We use a unified training config as shown in all of our experiments. We train LLAMA-1 (7B), LLAMA-2 (7B, 13B, 70B) for both OmniQuant baselines and SpikeLLMs according to this training scheme. Compared with the original OmniQuant, we do not apply loss augmentation methods except for 70B models for training stablity and we set the quantization group number as 1 in all of the experiments. Therefore, this scheme can be viewed as the simplified version without bells and whistles to focus on the influence of the quantization method itself.

In SpikeLLM, we computute the saliency metric for activations layer by layer during the OmniQuant pipeline. Different from activations, we computute the saliency metric for weights directly using the features from the first embedding layer. Because we find this is able to make the computation of inverse Hessian matrix more stable compared with computing layer by layer.

**Table 4: Training settings on the OmniQuant scheme. LET and LWC indicate learnable equivalent transformation and learnable weight clipping.**
| config | 4W4A | 2W8A | 2W16A |
| --- | --- | --- | --- |
| LET | True | True | False |
| LWC | True | True | True |
| learning rate of LET | 0.001 | 0.001 | N/A |
| learning rate of LWC | 0.01 | 0.01 | 0.01 |
| activation smooth | 0.75 | N/A | N/A |
| batch size | 1 | 1 | 1 |
| loss augmentation | False | False | True |
| epochs | 20 | 20 | 40 |
| group | 1 | 1 | 1 |

We further investigate the influence of different training samples to confirm the robustness of the proposed methods. In the OmniQuant pipeline, training samples are randomly cropped from the Wikitext2 dataset. To compare the performance of SpikeLLM and the OmniQuant baseline, we use a set of random seeds to sample different training data each time. In Fig. 5, we sample data and train for 40 times, using the same seed for both OmniQuant and SpikeLLM each time. SpikeLLM achieves higher average accuracy on 6 zero-shot datasets and lower Wikitext2 or C4 perplexity at the same time, showing consistently better performance. For the same training data, SpikeLLM always performs better. In other experiments in this paper, we keep the random seed as 2.

Figure: Figure 5: Comaprison of different training data for LLAMA-1-7b 4W4A models. In SpikeLLMs, 10% activation channels are set 2 spiking steps.
Refer to caption: extracted/5710913/fig5.png

## Appendix B Structured vs. Unstructured Spiking in Weights.

For weight quantization, as shown in Table 5, unstructured spiking steps usually achieve higher performance compared with structured ones. In our per-channel spiking scheme, it can achieve higher performance by setting per-channel spiking steps as elementwise spiking steps. However, unstructured conditions need additional masks and are less friendly to deployment. Therefore, we keep structured settings in low-bit quantization. But for additive LLMs, the performance is more important in the extreme case, and we choose the unstructured settings. Moreover, the PB-LLM baselines are also unstructured, so that, it can also confirm the fair comparison.

**Table 5: Comaprison between structured and unstructured weight quantization in the OmniQuant pipeline.**
| Method | Saliency | #Bits | ACEs | PIQA | ARC-e | Arc-c | BoolQ | HellaSwag | Winogrande | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLAMA-2-7B | – | FP16 | 1$\times$ | 78.45 | 69.32 | 40.02 | 71.07 | 56.69 | 67.25 | 63.80 |
| OmniQuant | – | W2A16 | 0.125$\times$ | 57.13 | 35.02 | 21.16 | 53.46 | 29.32 | 50.36 | 41.08 |
| $\text{SpikeLLM}_{\texttt{T=2}}$-Structured | 0.10 | W2A16 | 0.138$\times$ | 65.61 | 48.15 | 27.39 | 60.46 | 39.01 | 52.80 | 48.90 |
| $\text{SpikeLLM}_{\texttt{T=2}}$-Unstructured | 0.10 | W2A16 | 0.138$\times$ | 72.63 | 60.06 | 30.89 | 65.05 | 48.52 | 59.51 | 56.11 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1) Since this approach assigns different values of T corresponding to saliency of particular channels, can you please explain the temporal dynamics of the model (preferably during the on chip 1-bit inference phase) with a schematic diagram?

2) In Table 6, SpikeLLM uses  ternary GIF neurons as weight quantizers. What is the quantization level of the activations?

3) The results shown in Table 3 for SpikeLLM variants, all have activation quantization levels >= 4. Can we really call the models spiking?

4) Can the authors please compare their results to current sota quantized LLMs.

Please see weaknesses as well.
soundness: 2
strengths: The paper is well motivated and addresses a genuine problem of the increasing energy cost of LLMs. Experimental results show improved performance compared to the used baselines.
summary: The authors propose SpikeLLM, an energy-efficient large language model designed to tackle the increasing energy demands of current LLMs. They highlight two approaches namely, Generalized Integrate-and-Fire (GIF) neurons  and Optimal Brain Spiking framework to compresses spike length. The experimental results show improvements compared to the baselines used in the paper.
weaknesses: 1) The temporal dynamics of the model is not explained in details. There is very limited results on the activity sparsity of the model, energy-efficiency comparison with other efficient LLMs (Table 5 only compares with LLAMA-1-7B)
2) The authors can use better baselines for comparison such as 1.58 bit LLM [1], IR-QLoRA[2]. 
3) There is no experimental results showing 1-bit inference on neuromorphic chips. Even simulation showing spiking activity and convergence to similar result (as the merged spike) will demonstrate that we can use the model for efficient on-chip inference. No code has been added as part of the experiment. It is hard to understand how one would run the network in a 1-bit setting. If 1-bit (activity) inference cannot be shown experimentally then we cannot actually call this method spiking.

References
[1] Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J. and Wei, F., 2024. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764.

[2] Qin, Haotong, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno. "Accurate lora-finetuning quantization of llms via information retention." arXiv preprint arXiv:2402.05445 (2024).

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Lack of Logical Structure in Related Work: The related work section primarily lists existing studies without a systematic analysis of the development and challenges of SNNs and quantization methods.
2. Unclear Formula Description: The explanation of Equation 5 (lines 239-244) is not intuitive, making it difficult for readers to understand the specific meaning of the GLIF model.
3. Insufficient Visualization: There is a lack of intuitive graphical illustrations to demonstrate how the GLIF model divides long binary sequences into smaller segments over multiple time steps.
soundness: 3
strengths: 1. This study is the first to apply SNNs to large-scale language models with up to 70 billion parameters, filling a gap in the field.
2.: The authors provide a good analysis of issues in quantized LLMs and propose a Spike-Driven Quantization method that effectively addresses these problems.
3. The proposed GIF neurons leverage the time steps of SNNs to mitigate outliers that may result from direct quantization.
4. The method is thoroughly trained and tested on LLAMA-2-7B and LLAMA-2-70B models, validating its effectiveness. To my knowledge, this is the first time that SNNs have been tested on a model as large as LLAMA-2-70B.
summary: This paper is the first to introduce SNNs into LLMs, an innovation that deserves high commendation and demonstrates the broad applicability of SNNs. Additionally, the authors combine quantization techniques with SNNs to propose GIF neurons and an OBS framework. Overall, I consider this paper to make a significant contribution.
weaknesses: 1\I strongly suggest that the author takes the time to offer a comprehensive and meticulous description of the precise experimental results. It is essential to highlight the pivotal discoveries and thoroughly explore their potential ramifications within the broader context of the study.
2\It would greatly enhance the transparency and accessibility of the research if the author decides to open source the code. This action would not only foster a collaborative environment but also enable other researchers to replicate and extend the findings, ultimately advancing the field.
3\To fully ascertain the efficacy and superiority of the proposed method, it is imperative that the author conducts a more extensive series of experiments. These experiments should rigorously compare the performance of the current approach with various artificial neural network (ANN) alternatives, providing a comprehensive analysis of their respective strengths and weaknesses.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Is it feasible to compute the Hessian for such large models? Are any approximations used to get the saliency from the Hessian?
- A related work paragraph on ANN-SNN conversion in the context of transformer based models would add value to the paper
- The reviewer suggests to reconsider the use of the term "auto-regressive" as a synonym for recurrent, recursive or sequential. Auto-regression is a particular processing paradigm for sequences that aim to regress themselves, and is usually not interchangeably used with terms like recurrent or recursive. 
- The reviewer suggests to reconsider the claims on "necessity for introducing SpikeLLM" e.g. in lines 25, 110, 120, 370 in face of works like QuaRot that show that one get's away with simply rotating the weights.
- It seems like the character "\~" is out of place between 7B~70B in multiple places. Perhaps use `\sim` instead
- Please incorporate QuaRot and SliM-LLM in both the related work and the comparision
soundness: 3
strengths: The paper is technically sound, well written and clear in its presentation. Tables 3 and 4 meet the standards of evaluation of the LLM community. Figures 1 and 2 assist the reader in following the presented method. The extend of the material is on par with recent quantization works that have been published recently in top ML conferences (e.g. [1], [2]). The authors compare their method on the standard evaluation benchmarks for LLMs reporting both downstream task performance and language modeling performance of large scale models from LLAMA 7B to 70B.

ANN-SNN conversion as a research field is of relevance to the ICLR community. Prior methods were not evaluated at this large scale. It is expected that ANN-SNN conversion methods would face similar issues regarding outliers as classical LLMs do. The authors use the core component of SNNs, i.e. temporal processing, to counter outliers and to compute channels with potential outliers in higher precision through longer simulation time. It is worth pointing out that using higher precision for such channels has been prominently studied in the literature (e.g. [3]), and saliency-driven calibration has been studied for quantized LLM (e.g. in [4]). Also ANN-SNN conversion based on integrate-and-fire neurons that model increased precision by conducting multiple steps was presented in prior works (e.g. [5], [6]). The core contribution of this paper is therefore the combination of these concepts to demonstrate > 7B parameter LLM quantization by handling outliers with increased simulation time budgets for certain channels. The paper therefore allows the comparison of rate-coded spiking neural networks against classical quantization methods in the large scale domain of LLMs.

1. [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2211.10438)
2. [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://arxiv.org/abs/2308.13137)
3. [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)
4. [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2405.14917)
5. [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://arxiv.org/abs/2406.03287)
6. [SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://arxiv.org/abs/2406.03470)
summary: The paper introduces a post-training quantization scheme for large language models (LLMs). The quantization function is computed by a spiking neural network. The contributions over prior works are a reduction of the requires number SNN simulation steps by grouping time-steps into low-bit integer operations, handling outliers by allowing certain channels to conduct multiple simulation steps, and detecting outliers in weights and activations offline. Results are compared to the OmniQuant baseline and show improved downstream task performance on a standard evaluation protocol.
weaknesses: The fields of ANN-SNN conversion and LLM quantization are quickly moving. It is hence important to keep track of the recent developments in these fields. A weakness of this paper is that it is missing comparisons against some closely related recent papers.

[1. SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://arxiv.org/abs/2406.03470)
Related work that uses integrate-and-fire neurons to implement low-bit quantization in transformers. Arguably this work does not work with as large models as the present paper.

[2. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2405.14917)
Improves over OmniQuant with saliency based metrics, similar to the ones proposed here. The reviewer is missing a comparison in tables 3 and 4.

[3. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://arxiv.org/abs/2404.00456)
Shows that outliers can mostly be eliminated by mathematically equivalent parameterizations of LLMs (through appropriate weight matrix rotations). This work challenges some key claims made by the presented paper. E.g. line 370 claims "necessity to introduce SpikeLLM due to the inefficiency and incompatibility [...] of existing ANNs based quantisation strategies to spike neural networks". If outliers can be eliminated in LLMs, it seems likely that basic ANN-SNN conversion methods such as SpikeZIP would work in low precision even for larger LLMs.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness. Listed below:
1.Please explain/correct Eq3, it seems confusing to me.
2.In eq5, please fix the summation over t
3.Fig4(a) missing baseline label
4.Please provide a more detailed demonstration of the advantages of the proposed method.
5.For SASS and OBP,  Does the saliency calculation need to be performed both during training and inference? etc
6.how scalable is the approach proposed by the authors? etc
7.What are the limits of both ann quantization and spiking llm boost? etc
soundness: 3
strengths: 1. This paper attempts to solve the energy efficiency problem of LLM through SNN. This is a widely concerned issue. The author proposed several SNN based methods to solve the problems faced in traditional model quantization. The research problem is important and attractive, and the methods and experiments are of practical significance.

2.The author proposed three core methods: GLIF, SALIENCY-AWARE SPIKING STEPS (SASS), and OPTIMAL BRAIN SPIKING (OBP). The logic is clear and easy to follow:
-GLIF tries to encode more information in a single spike time step by merging L spiking steps
-SASS distinguishes the importance of different channels and assigns different quantization levels to different channels to balance the computational load and performance
-OBS determines the importance of any weights and activations through gradient methods.

3.According to the author's experiments, the proposed pipeline works well and outperforms the quantized ANN of the same level in terms of energy efficiency and performance. The authors conducted extensive testing on both the diversity of experimental datasets and model sizes.
summary: This paper proposes the first large-scale spiking large language model that can scale the spiking neuron dynamics to handle models with 7 billion to 70 billion parameters. It has the potential to address the energy and computational resource limitations typically associated with running large language models. The main contributions proposed by the authors include:

1. Generalized Integral-Fire (GIF) neurons: This model compresses the spiking length by encoding the pulse train more efficiently. These neurons reduce the spiking length from T bits to T/L*log_2L bits. This novel approach improves coding efficiency while retaining the necessary computational power.

3. Optimal Brain Spiking (OBS) framework:a saliency-based spiking mechanism that dynamically allocates spiking steps based on the importance of channels within the network. Channels that are considered more important receive more spike steps to capture information, while less important channels receive fewer steps. This saliency-based approach enables more efficient quantization and information encoding, further improving energy efficiency.

4. Compatibility with modern hardware: SpikeLLM's GIF neurons and OBS framework can be deployed using both general matrix multiplication (GEMM) of traditional GPUs and event-driven neuromorphic chip architectures. This increases the feasibility of this method for deployment on neuromorphic hardware.

5. Experimental results: Through various benchmarks, SpikeLLM has significantly improved both performance and energy efficiency over traditional quantization methods used in LLM, such as OmniQuant and GPTQ pipelines. For example, it reduces the perplexity on WikiText2 by 24.85% and improves the common sense reasoning accuracy by 2.01% compared to the LLAMA2-7B model with standard quantization.
weaknesses: 1.Please explain/correct Eq3, it seems confusing to me.
2.In eq5, please fix the summation over t
3.Fig4(a) missing baseline label

My main concerns are about the computational complexity of the proposed method. The authors did not elaborate in detail in the article on the computational cost of the proposed GLIF, SASS, and OPTIMAL BRAIN SPIKING. Specifically:

4. For GLIF, In eq5, merging IF neurons of multiple steps into one step does reduce the requirement for SNN running time, but increases the computational cost of a single neuron; on the other hand, the implementation of neuromorphic hardware with multiple quantization levels is also unclear. Therefore, it is not sufficient to measure only auto-regressive steps here. Please provide a more detailed demonstration of the advantages of the proposed method.

5. For SASS and OBP,  Does the saliency calculation need to be performed both during training and inference? What is the additional computational cost? How can KV-caches be combined with the proposed method to further reduce the amount of computation? According to my understanding, eq10 only needs to be calculated once, but eq7 and 9 must be performed during inference. (point out if I am wrong) Please  elaborate on the additional computational cost.

Experiment result:

6. In table4, Most of the experimental results for 70b models are missing. The authors suggest that GQA makes training unstable. It would be good to have a more detailed analysis showing the key reasons and potential research directions here. In other words: how scalable is the approach proposed by the authors? What specific issues need to be addressed for stable scaling to 70b or larger models?

7. In table3, Although the proposed method is better than the traditional quantization method, the performance gap with the original model is still more than 10%, which limits the practicality of the method. The author proposed in Limitations: Continuous pretraining would boost the performance. What are the limits of both ann quantization and spiking llm boost? What is the performance difference with sufficient training time?

Meta-review: The work claims to propose the first spiking LLM, and considers models with up to 70B parameters. Improvements in perplexity, accuracy in common-sense reasoning when compared to quantized LLMs are claimed. Moreover, up to 10-fold reductions in computation are achieved. 

The paper addresses an important problem and the experimental evaluation follows the standards of the LLM community, and the claims are validated with comparisons to recently published baselines. 

The paper makes an important contribution in new methods to improve energy efficiency of LLMs, and will be of interest to the ICLR community. I therefore recommend acceptance.

Decision: Accept (Poster)