Distribution-Aware Multi-Granularity Phase Coding: Towards Lower Conversion Error for Spike-Driven Large Language Models
Authors: Hanyuan Zheng
Haozhen Zhang
Tianshuo Chen
Zhaogeng Liu
Yi Chang
Bin Gu
Venue: ICLR 2026 Poster (2026)
Keywords: Spiking Neural Network
Optimization
Primary area: foundation or frontier models, including LLMs

Abstract: Spiking large language models (LLMs) offer significant advantages on neuromorphic hardware, yet training them from scratch remains prohibitively expensive. A promising alternative is ANN-to-SNN conversion, which reuses pretrained ANN weights while minimizing conversion error. 
However, existing conversion frameworks neglect activation distributions, as reflected in SNN neurons with rate or temporal coding to map uniformly distributed rather than distribution-aligned discrete values, thus causing latent conversion error arising from distribution misalignment. 
To tackle this problem, we propose a distribution-aware multi-granularity phase coding approach, which achieves reasonable discrete value allocation by minimizing conversion error relative to activation distributions. 
Specifically, multi-granularity phase coding extends conventional phase coding with multiple learnable bases, incorporating representational capacity across different granularities. 
Building on this coding scheme, we further propose a novel ANN-to-SNN conversion paradigm designed towards lower conversion error. 
In particular, our paradigm utilizes the activation distributions of hidden layers to sample data for cost-efficient neuron training, without requiring fine-tuning of model weights. 
Theoretically, we provide a convergence guarantee for the neuron training algorithm. 
Extensive experiments on the LLaMA model confirm the effectiveness of both our coding scheme and conversion paradigm.
Concretely, our spiking LLM attains the lowest perplexity with ANN-level accuracy, accompanied by a 42\% reduction in energy consumption of MAC and AC operations. Our code is available at https://github.com/JLU-Solar/PhaseSNN.

Published as a conference paper at ICLR 2026
DISTRIBUTION-AWAL MULTI-GRANULARI PHASE CODING: TOWARDS LOWER CONVERSION ERROR FOR LARGE LANGUAGE MODELS
Hanyuan Zheng',, Haozhen Zhang",, Tianshuo Chen**, Zhaogeng Liu'r, Yi Chang1.2.3, Bin Gu School of Artificial Intelligence, Jilin University", International Center of Future Science, Jilin University? Engineering Research Center of Knowledge-Driver Human- Machine Intelligence, MOE3 (hyzheng25, hzzhang23, tschen25, 2g (allll22@@@mmm @mamlmmalllll.lll iss jlu. edu. en, yichange jlu. wuu.edu..cmm cn, jsgubinggmain 1. com
ABSTRACT
Spiking large language models (LLMs) offer significant advantages neuro- morphic hardware, yet training them from scratch remains prohibitively expen- sive. A promising alternative is ANN-1o-SNN conversion, which reuses pre- trained ANN weights while minimizing conversion error. However, existing con- version frameworks neglect activation distributions, as reflected in SNN neurons with rate or temporal coding to map uniformly distributed rather than distribution- aligned discrete values, thus causing latent conversion error arising from distribu- tion misalignment. To tackle this problem, we propose distribution-awar multi- granularity phase coding approach, which achieves reasonable discrete value allo- cation by minimizing conversion error relative to activation distributions. Specifi- cally, multi- granulitaanu phase coding extends conventional phase coding with mul- tiple learnable bases, incorporating representational capacity across different gran- ularities. Building on this coding scheme, we further propose a novel ANN-to- SNN conversion paradigm designed towards lower conversion error. In particular, our paradigm utilizes the activation distributions of hidden layers to sample data for cost-efficient neuron training, without requiring fine-tuning of model weights. Theoretically. we provide a convergence guarantee for the neuron training algo- rithm. Extensive experiments on the LLaMA model confirm the effectiveness of both our coding scheme and conversion paradigm. Concretely, our spiking LLM attains the lowest perplexity with ANN-level accuracy, accompanied by a 42% re- duction in energy consumption of MAC and AC operations. Our code is available at http://hhhhbbbbcoommmmm/eeeeeegp
INTRODUCTION
Large language models (LLMs), exemplified by GPT-4 (Achiam et al. 2023). Qwen3 (Yang al., 2025). and LLaMA3 (Dubey et al., 2024). achieve remarkable performance across a wide range of natural language processing tasks through training on massive text corpora. However, the trans- former architecture in L.MMs relies on dense matrix multiplicatinns, where intensive Floating-Point Multiplication and Addition (MAC) operations result in prohibitive energy consumption (Vaswani et al. 2017 De Vries 2023). This substantial challenge during training and inference necessitates the pursuit of energy-efficient paradigms for LLMs. In contrast to conventional neural networks, spiking neural networks (SNNs) have received increasing attention due to their energy efficiency in mimicking biological neurons, thereby offering a promising solution. Building on this foundation, spiking L.MSs have recently been developed, showing promise for efficient execution neuromor-
phic hardware (Xing et al. 2025 Chen et al. 2025a Zhengzheng & /Zhu 2025).
Research on SNNs has primarily focused two approaches: direct training and ANN-to-SNN conversion. Direct training methods (Neftci et al. 2019 Zenke & Vogels 2021 Lee et al. 2016
"These authors contributed equally as co-first authors and are listed in random order.
1 Co-coorrepooning authors.

---

Published as a conference paper at ICLR 2026
typically adopt surrogate gradients during backpropagation to address the non-differentiabiiiy of spiking neurons. Nevertheless, training SNNs from scratch is prohibitively costly in both time and resources, particularly at the scale of LLM parameters. In contrast, ANN-to- SNN conversion (Tong et et 2022 Hao et al. 2023a Yang et al. 2022 Chen et al. (0255) offers a more efficient paradigm, typically reusing pretrained ANN weights in the spiking model while minimizing conversion error to achieve effective conversion. Since minimizing conversion error is often less costly than direct training, ANN-1o-SNN conversion demonstrates greater generality in resource-constrained environ-
ments DDigg et al. (021).
Unfortunately, there exists the conversion error arising from distribution misalignment, which is a long-standing inherent problem in such conversions (Datta & Beerell (0022) However, current ANN-to-SNN conversion frameworks for I.LM tend to overlook non-uniform activation distribu- tions, leading to latent errors owing to distributional misalignment (Chen et al. 2025b). As shown in Figure [] activations within single layer are generally non-uniformly distributed, and activation distributions differ across layers. Unfortunately, in the coding schemes of existing spiking LLMs, rate (Wu et al. 2019 Sengupta et al. 2019) or temporal coding methods (Mostata 2017] Zhao et al.) 2025) typically map discrete values by discretizing activation values into uniformly partitioned in- tervals, rather than aligning with the large-scale non-uniform activation distributions observed in practice. Furthermore, distinct activation distributions across different components of large models pose an additional challenge, highlighting the need for a learnable and adaptive framework capable of handling heterogeneous distributions
(Zhang al. 2018).
To address the challenge mentioned above, we introduce an alternative coding scheme referred to as phase coding (Kim et al., 2018 Zhang et al., 2020) and sigmificantly en- hance by proposing distribution- aware multi- multi-granularity phase cod- ing. Conventional phase coding can realize non-uniform allocation of mapped discrete values by adjusting the base. Building this observa- tion, the proposed distribution-awant multi-granularity phase coding inte- grates representational capacities at different granularities through multi- ple learnable bases, thereby offering enhanced flexibility in discrete value allocation. The final outcome is that it can achieve a more reasonable dis- crete value allocation by minimizing distribution-related conversion errors, which is essential for ensuring the performance of spiking LLMs after conversion.
2003
SIL sohhooe down prg soo
1000
1000
Number asstraaionn
500
Met Mel whwhhbs
0.04
5990
0.08
2003
o.
0.00
20
SAU
us soy
Grouge
80
dlven peej
Softstar
EA2
0.03
0.04
0.00
100
Activatinn Ranage
(b)
Figure 1: Distributions of the absolute activation values across layers in large language model. (a) The activa- tion values of each layer are truncated between and 0.05, and the histogram reveals the uneven distribution of activa- tion values. (b) The activation values are divided into 100 groups, with each group representing the average, showing the uneven distribution trend of activation values.
Furthermore, we develop novel ANN-to-SNN conversion paradigm built upon the aforementioned coding scheme. The central component of the paradigm is a cost-efficient alternating optimization neuron training algorithm, designed to minimize conversion error relative to activation distributions. Specifically, we tune only the neuron parameters using data pre-sampled from the corresponding hidden-layy layer activation distributions, which eliminates the forward and backward propagation of net- work layers and renders our paradigm highly cost-efficient in conversion. In summary, our paradigm yields spiking LLMs with both low conversion error and highly cost-efficient conversion under convergence guarantee.
Our contributions are summarized follows:
uultiiggranularity Phase Coding. We propose a distribution-awarre multi- muati-granularityy phase coding scheme with multiple learnable bases, which enables flexible and adaptive allocation of discrete value mappings.
Distribution-Aware Conversion Paradigm. We establish a distribution paradigm that breaks the uniform discretization of rate and temporal coding, facilitating faithful ANN-to-SNN conversion.
2

---

Published as a conference paper at ICLR 2026
Theoretical Convergence Guarantee. We analyze the convergence for the proposed al- ternating optimization neuron training algorithm, based on the gap between the objective function before and after smoothing.
A remarkable Spike-Driven LLM. Our Spiking LLM achieves the lowest perplexity while preserving ANN-level accuracy, setting new state-of-the- results, meanwhile re- ducing energy consumption of MAC and AC operations by 42.0% compared to its ANN counterpart.
RELATED WORKS
2.1 ANN-TO-SNN CONVERSION
Existing ANN-to-SNN conversion methods are primarily divided into one-stage and two-stage ap- proaches (Chen et al. 2025a). The former involves not performing any further optimization on the converted SNN and directly converting the ANN to an SNN model. This approach is commonly used when the target ANN is built upon ReLU functions, as the output of ReLU can be effectively approximated by the firing rate of spiking neurons (Cao et al. 2015). Building on the insight dis- cussed above, both Diehl et al. 2015) and SSngupta et al. (2019) employ normalization techniques to further improve conversion performance. Additionally. ly. Bu et al. (2033) propose, from a theoreti- cal perspective, the use of QCFS functions replace ReLU functions in order effectively reduce conversion error. The two-stage approach, on the other hand, focuses on optimizing the converted SNN to ensure its performance. HHoo al. (2023a) classify the unevenness error into four cases and propose an optimization strategy based on residual membranc potential to reduce error. Hao et al. (2023b) focus on addressing the conversion error caused by one additional (or one less) spike by shifting the initial membrane potential. Chen et al.) (2025a) adopt coarse-to-fine fine calibration opti- mization strategy to optimize the converted SNN. However, these approaches either struggle to scale to transformer-basee LLMs or still incur high optimization costs for the converted spiking LLMs.
2.2 SPIKING LLM
Spiking LLMs, noted for their low energy consumption, are gradually emerging as promising direction in the field of large-scale models. Despite this promise, research on spiking LLMs re- mains limited. Early efforts include SpikingBERT (Bal Sengupta, 2024). which leverages the average spiking rate of neurons at equilibrium and incorporates know ledge distillation to enhance both training efficiency and model performance. SpikeGPT (Zhu et al., 2023) adapts RWKV by combining spiking activations with sequential attention, demonstrating that autoregressive language generation is feasible within the spiking paradigm. More recently, SpikeLLM (Xing et al. 2025) has introduced a hybrid co-architecture that integrates SNNs with quantized ANÑs, scaling spiking models to the billion- billion-paramenn parameter regime (7-70B) and achieving improved energy efficiency. FAS (Chnn al. 2025al enables the conversion of pretrained ANN-based LLMs into spiking counter- parts through a two-stage calibration, resulting in lower energy consumption and latency. However, existing Spiking L.M frameworks typically rely uniform rate coding, which overlooks the non- uniform distribution of activations and consequently introduces latent conversion error.
PRELIMINARY
Spiking coding is a scheme that determines how continuous values are encoded into sequence of spikes, with rate coding, temporal coding, and phase coding being the most widely used schemes. In particular, phase coding combines the characteristics of temporal coding and rate coding, achieving higher representational density than other coding schemes under the same total timestep T.
Specifically, similar to temporal coding, where spikes produced at different timesteps f represent different values, phase coding assigns distinct weights to each i within the total timestep T'. Unlike the uniform phase values induced by the typically employed linear proportion T-t in temporal cod- ing (Rueckauer& & Liu 2018 Han & Roy 2020), phase coding assigns each timestep f a phase value B where B is the base of the phase. At the same time, preserves the multi-spike representation inherent in rate coding, thereby enhancing the representational capacity within finite total T. By
T"
3

---

Published as a conference paper at ICLR 2026
combining these advantages, phase coding achieves expansion of the number of encoded discrete values to 2", which reduces the total number timesteps in an exponential manner.
The biological manifestation of generic phase coding has been demonstrated by (Montemurro al, 2008 and is further advanced in ANN-to-SNX conversion (Hwang & Kung 2024; Hwang et al, 2024 Its corresponding neuron dynamic procedure is characterized by the threshold (tt,, reset strength h(t), and output weight d(t), as detailed the following equation:
T
u(1) = Σ
I(trr), ult + 1) = v(t) h(t)s(t), O(t) = d(t)s(t),
(1)
upe=1
stt) = ())))))(( o (((t) - -
where (--) denotes the Heaviside step function and the initial membrane potential u(1) receives the output ((prr) from the pre-layer. For timestep (1. of the current neuron, v(t) denotes the membrane potential, s(1) E {0, denotes the binary spike, O(t) denotes the output signal. Typically, for the conventional phase coding, hu(), d(l), and 0(l) specified as B- as follows:
cft 1) vtt) - B-)s O(t) s(), s(t) (vt())
(3)
Notably, the activation value can be approximated within limited timestep T.
4 METHODOLOGY
In this section, we first explain our motivation from the perspective of information theory. Next, we introduce our multi-granularity phase coding. Finally, we elaborate on the proposed distribution- aware ANN-to-SNN conversion paradigm, shown in Figure 2
Pretrained ANN Large
Embedding
Spiking Self-Attention
Language Model
x
Spiking
Linear
Linear
Linear
Uniform Initialization
RMSNorm
ROPE
ROPE
Qg
Kg
V
uniform interval
Spiking Self-Attention
+N.
TI X d
Alternatiig Optimization Neuron Training
Transpose
D
Spiking RMSNorm
d
Scale
Spiking Soltmax
Linear
distribution-wwarr non-uniform interval
Spiking MLP
Lineer
Multi Saati-Granularity Phase Coding
Spiking
MLP
Lineer
Spiking
SiLU
Our Spiking Large
Language Model
Head
M FS Neuron
*
Spike Dot Product
Hardmard Product
Figure 2: ANN-to-SNN conversion paradigm based multi-granularity phase coding. Left: con- version pipeline with alternating neuron optimization. Right: spiking LLM built on SNN neurons.
4.1 MOTIVATION FROM INFORMATION THEORY
Considering the distribution of activation values, the ANN-10-SNN conversion error can be charac- terized by the mean squared error as follows:
E = ((=)((-x)) dx,
(4)
4

---

Published as a conference paper at ICLR 2026
where 2 denotes the SNN neuron's approximation of the ANN activation value The conversion error can be regarded as equivalent to the quantization distortion information theory. From the information-theores
perspective, SNN coding is analogous to quantization in its allocation of dis- crete values. When an SNN employs M allocation intervals, X(x) represents the relative density of these intervals. Consequently, the conversion error equivalent to the quantization distortion, as formalized in Theorem
Theorem 1 (cf., (Gray & Neuhoff 2002)). For an arbitrary quantizer 9. the asymptotic average distortion with M quantization intervals can be expressed by rewriting Bennett's 's integral in terms of the point density function:
D(g) -(1)= p( /
/
p(z) {²(x))
dx,
(5)
12 M
where p(z) denotes the probability density function (PDF) of the input signal, and X(x) denotes the point density function of the quantizer.
To minimize the ANN-to- ANN-to-SNN conversion enror, Corollary I specifies the optimal allocation princi- ple, namely assigning larger (zx) to regions with higher probability density.
Corollary 1. Let (()) be the asymptotic distortion in Theorem 7 The point density function M(x)
that minimizes D(1), subject to the normalization constraint f Mr)da 1, is given by:
[p(=)) flp(u)))/3dd
X xz) (x)
= (x)) [p(x))1//3
(6)
In LLMs, the activation distribution p(z) is inherently non-uniform. Consequently, the optimal allocation function A (z) also exhibits non-uniformity. This implies the necessity of a distribution- aware coding strategy in SNNs, whereby regions of higher activation density are allocated more quantization intervals, while regions of lower density receive fewer intervals.
4.2
PHASE CODING
Motivated by this, introduce multi-granulan phase coding, which adaptively allocates bases {Bi, B2,.... B,} of different granularities to non-uniform activations within small timestep T.. In particular, the alteration of phase values introduced by multi gultiigranularity phase coding, relative to conventional phase coding, formally defined as follows:
(B-')= + BB² "....,
(++1) ( (""
This design offers a more flexible discrete value allocation, effectively minimizing the expected conversion E. A more intuitive illustration is provided in Figure The non-uniform discrete value allocation introduced by multi-granularity phase coding allows us to align the mapped discrete values with the activation distribution by tuning the bases (Figure 3 left). Consequently, intervals with denser activations are allocated more discrete values rather than being uniformly distributed. The distinction conversion eтrоr relative uniform discrete value allocation illustrated in Figure
3 (right), where the reduction in conversion error can be easily observed from the shaded area.
Distribution
Identity Mapping
Conversion Error Ours Sub-Intervals
Conversion Error in Rate or Temporal Encoding Sub-Intervals
Output Activation Value
Ours
Rate Temporal
Eg
Eg'
Input Activation Value
Eg p(x)(e- - dx
Eg
JR p(x)( xx
Figure 3: Comparison of discrete value allocation and conversion error between multi-gramularity phase coding and rate temporal coding. The shaded area ER and Eg represent conversion emrors under different codings.
By incorporating the dynamics from Section 3 we can obtain the neuron dynamic procedure with multi-granularity phase coding, which is obtained by extending the conventional formulation in
5

---

Published as a conference paper at ICLR 2026
Equation 3 and is formulated as follows:
o(t + -
= v(t) - h(1))(t), 0(1) - d(t)s(()) s(1) - (((()) 0()),
(8)
(d(t))?= (BT Bi ...., B,', B2 (1+1) B,
For clarity, denote (h(t)] {0(t)) {d(t)) BB h. 0, d, and B. In contrast to Equation (3). we remove the constraint 8(t) = h(t) = d(t) and instead treat (h. 0) as learnable parameters that are decoupled from d. Equation (8) introduces
multi-granularity design, in which the neuron parameters are no longer constrained to single base, but instead are constructed from multiple bases. This generalizes the conventional single-base scheme in Equation (3)
4.3
CINTBBTTION-AARR CONVERSION PARADIGM
The proposed spiking LLM architecture is illustrated in Figure 2 By introducing SNN neurons with multi-granularity phase coding prior to linear layers and matrix operations, activation values converted into spike signals, thereby avoiding floating-point matrix multiplications. For the SNN neuron with multi-gramule gramularttll phase coding, our objective is align its discrete value allocation with the activation distribution while minimizing conversion error, as introduced in Section 4.1 Toward this end, we consider the following expected conversion error:
/ [(N) (h, 0). B) x)) d))dr, (h.0).
min
where p(x) is the PDF of activation x, and SN(.) = - O(l) is the discrete value obtained by mapping the activation value through the SNN neuron. In practice, for each neuron, the correspond- ing activation distribution can be estimated from a batch of input text and then downsampled to construct a training dataset X composed of activation samples. Ultimately, we formulate the target problem as an empirical conversion error minimization problem, as follows:
min [ISN(X: (h, 0), B) x112.
(10)
(h.0).B
To effectively solve the optimization problem formulated in Equation (10). we propose an alternat- ing optimization neuron training algorithm, as presented in Algorithm I Specifically, we alternate between optimizing (h, } and B. Due to the non-differentiabiiity of the Heaviside step function, updates (h. 0} are carried out using a sigmoid- case of fixed number of granularities, we address timestep allocation through an adaptive granular- ity allocation method, with full details provided in Appendix C With Algorithm [ model weights do not require fine-tuning, and the neuron training dataset is obtained through pre-sampling. Com- bined, these climinate the need for forward and backward propagation through network layers and restrict propagation to neurons alone, rendering our training algorithm highly cost-efficient. Further- more, the handling of other nonlinear operations the model (e.g., RMSNorm, ietrraiionnaciivatinni multiplication, Softmax, and SiLU activation function) is provided in Appendix B
surrogate gradient (Wu et al.) 2018). In the
5
ANALYSIS
In this section, we first introduce several necessary assumptions and then present the convergence analysis results of the neuron training algorithm based on activation distributions.
5.1 ASSUMPTIONS
We denote the original objective function ISN(X: smoothed objective function by g(,, 0, B). In updating and 0, surrogate gradients are employed to carry out gradient descent. This process can be regarded as gradient descent utilizing the true gradient of the smoothed objective function. Based on this perspective, we analyze the convergence behavior of the alternating optimization scheme. Since the parameters h and 0 are optimized simul- taneously, we introduce the variable o to represent them collectively. more detailed explanation of the symbols employed in the analysis is provided in Appendix A
X by f(h, 0 B), and the

---

Published as a conference paper at ICLR 2026
Algorithm Alternating Optimization Neuron Training Algorithm
Input: Training dataset X, optimization steps N, N1 and Ng. learning rate 77 and 72,, neuron
parameters {l(1), (())]=) and (Bi _T,,.... B2 B2 -t++1)
B=(++)
B ).
2: Initialize (ho (0) 00 BB-
3: for i 0, 1... ,N 1--10 do
4: for 0, ,N ,1
5: Compute LMSE (3) 005) X, (.B
6: Update (h.
(nt+t) 0(1+1)) (h (f) 0 WnoLMsE(h,) ,001: x, (())).
7: end for
8:
Update {h i 1+1 {
(0) (0)
(N))
0 (.)
9:
for j - 0, ... 2 do
10:
Compute LMSE B ())
X h
1 0 (+1)
11: Update B B =( 1/2 V WBBLME(BBB 0 x, h 0
end for
13: Update B
(0) :+1 B
(Na)
14:
end for
15:
Output: Neuron parameters (h" 0 B".
Assumption (Lipschitz Gradient). There exist constants L1, L2 > 0, for Vol, and VB,, Ba.
such that:
Vog(o,, B) - V vos(o2, B)ll S Li 01 -211:
(11)
(o, Bi) (i, B2)ll S L2 B| B2ll- Assumption 2 (Polyak- (PL.) condition). There exist constants 11, U1,02,13
(12)
such
that:
IVg(o, B) 2 2n1 (g B) -3),
(13)
V Voo(o, B) 2 (p2(1(o, B) -1) (o. B)),
(14)
Bf(o, B) 2 (f B) prlo) (o, ())
(15)
where 9" denotes the global minimum value of g(o, B), (B) B) is the minimum of g B) with B fixed, and (o B) is the minimum of (o, B) with o fixed.
Assumption 3. For arbitrary o and B, we have (o B) g(o, B).
Remark 1. Assumptions and 2 are frequently invoked the theoretical analysis of gradient- based optimization methods (Malinovsky et al. 2024 Zhou 2018 Khaled & Richtárik (020)), while Assumption 3 ensures that the smoothed loss function does not deviate excessively from the original function. This requirement is reasonable,
studies on the approximation of step functions have already demonstrated that the error between a smoothed function and a step function can be made small Kyurkchiev & Markov 2015 Thevv al. 2015
5.2 CONVERGENCE RESULTS
Based on the above assumptions, we conduct a convergence analysis of Algorithm and examine whether the stability of the algorithm is significantly affected by the surrogate gradient.
Theorem 2. Suppose Assumptions
hold, and let 0 < Yh I min( ł, L 0
min( L. Tnpa). of Ok-1 0x
(N:)
Ok ( BB-1, and B ((Na) BB, then for Algorithm l
the following inequality holds:
((04+1,B41)) BB+1) 5
(1 (1-1mma)111111 [f Bx) -
(16)
where
C C =0oo -11mm)(1 - (3372) + 2x +
(1 (3722 (g(o, B²)
2
(17)
+ 1372
| V (0) (ox41,)]|1 B)Il2
2/1

---

Published as a conference paper at ICLR 2026
Remark 2. Theorem comprises 2 a linear convergence term (1- (1_-p)))(11ppppp and an error term C, resembling the results obtained in analyses such as (Nguven et al. 2017 Yuan etal. 2024). We note that the error term is largely determined by the smoothed objective function. A suitable smoothing technique can ensure that the error constant C remains sufficiently small, thereby enabling the iterative process of the algorithm approach the global optimum. The detailed proof of Theorem can be found in Appendix
ff B&)
Table I: Results LLaMA-2-7B. "Time Cost" denotes the training time required for the method to obtain the SNN. "Grain" denotes the number of granularities.
PPL Perf. +
T
Time Cost
Wikitext2
C4
Redpajama
Pile
Avg. PPL
LLaMA-2-7B
N/A
N/A
5.47
6.97
5.61
4.63
5.67
Lp
8
5h 54m
5.86
7.51
6.08
4.97
6.10
TTFSFormer
128
N/A
11.88
16.47
13.18
9.32
12.71
LAS
N/A
34.26
40.39
32.18
20.10
31.73
SpikedAttention
2m 02s
19.02
25.05
20.77
14.80
19.91
8
Ours (Grain=2)
2m Ols
6.71
8.96
7.23
5.74
7.16
Ours (Grain=3)
2m 04s
7.10
9.71
7.80
6.09
7.68
LAS
N/A
6.05
7.88
6.37
5.13
6.36
SpikedAttention
2m 28s
11.64
15.47
12.84
9.47
12.36
10
Ours (Grain=2)
2m 25s
5.50
7.05
5.68
4.67
5.73
Ours (Grain=3)
2m 27s
5.53
7.06
5.69
4.68
5.74
ACC Perf. f
T
Time Cost
WinoGrande
AreC
ArcE
PiQA
Avg. ACC
LLaMA-2-7B
N/A
N/A
69.06
46.33
74.54
79.05
67.25
SpikeLLM
8
5h 54m
67.40
42.58
71.46
77.75
64.80
TTFSFormer
128
N/A
70.56
44.88
73.11
78.94
66.87
LAS
N/A
69.46
45.56
73.65
77.97
66.66
SpinedAttention
2m 02s
68.19
41.38
68.77
77.20
63.89
Ours (Grain=2)
2m 0ls
70.56
46.16
73.99
77.97
67.17
Ours (Grain=3)
2m 044s
70.96
46.08
74.33
77.86
67.31
LAS
N/A
70.64
45.90
73.95
78.24
67.18
SpikedAAtention
2m 28s
67.32
40.10
68.73
76.44
63.15
10
Ours (Grain=2)
2m 25s
70.48
46.50
73.91
78.29
67.30
Ours (Grain=3)
2m 27s
70.88
46.16
73.78
78.13
67.24
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Baseline. We consider the following three baselines: Full-precision ANN, evaluated under the zero-shot setting, provides standard performance reference for spiking LLMs. SpikeLLM (Xing et al. 2025)) integrates SNNs with quantized ANNs to build a spike-driven large language model. TTFS (Zhao al. 2025) applies lime-t--fist-sppkke coding transformer architecture and achieves the spiking transformer based on temporal coding. LAS (Chen et al., 2025b) employs SNN neurons with (tt) htt) d(t)
2 for ANN-to-SNN conversion, which can be viewed as special instance of our method without optimization. Spiked Atention (Hwang et al.) 2024) leverages single-spike phase coding construct spiking transformer, and we extend this design to the LLaMA model in this work. The implementation details of all methods are provided in Appendix D.I
Datasets and Metries. To effectively evaluate different methods, we adopt perplexity and accuracy as evaluation metrics. For perplexity, we conduct evaluations on Wikitext2 MMertyy et al.) 2016), C4 (Raffel et al.) 2020). RedPajama (Weber et al. 2024),, and Pile (Gao et al.) 2020). For accuracy, we evaluate zero-shot reasoning performance on WinoGrande (Sakaguchi et al. (02T)). ArcC, ArcE
(Clark et al. 2018), and PiQA (Bisk et al. 2020 We report the accuracy (acc) for WinoGrande
8

---

Published as conference paper at ICLR 2026
and the accuracy norm for AreC, ArcE, and PiQA. All accuracies measured using lm_eval v0.4.2 Sutawika et al. 2024).
6.2 MAIN RESULTS
We report the results of our method under different granularities, along with comparisons to other baselines. As shown in Table and 2 our method delivers optimal overall performance by main- laining accuracy close to that of ANNS alongside low perplexity, highlighting the superiority of our paradigm over existing baselines. Notably, compared to LAS, which represents a special case of our method without optimization, effectiveness of the proposed neuron training algorithm. With respect to SpikedAttentions even with relaxation of its single-spike phase coding (as described Appendix [DII), it fails sustain satís- factory performance in our limited total timestep experimental setting (7 E {6, 8, 10)). Relative to SpikeLLM, our method demonstrates a substantial advantage in time cost, as its training process en- tails forward and backward propagation through the decoder layers, whereas our paradigm entirely eliminates this overhead. For TTFSFormer, our method can achieve comparable performance with fewer timesteps due to its time-to- time-to-first-ppik coding. Furthermore, due to the enhanced represen- tational capacity of multi-granularity phase coding, our method achieves ANN-to-SNI conversion for LLMs with limited total timestep. The ablation study and the results larger-scale LLM are provided in the Appendix D.2 and D.3 respectively.
method yields significant perplexity reduction, validating the
first-spike
Table 2: Results on LLaMA-3-8B. "Time Cost" denotes the training time required for the method to obtain the SNN. "Grain" denotes the number of granularities.
PPL Perf. 4
T
Time Cost
Wikitext2
C4
Redpajama
Pile
Avg. PPL
LLaMA-3-88
N/A
N/A
6.14
8.88
7.44
5.52
7.00
LL
8
6h 13m
> >100
>10
>100
>100
>100
TTFSFormer
128
N/A
6.72
9.82
8.12
6.03
7.67
LAS
N/A
93.13
>10
> >100
>100
100
Spiked/Attention
Im 38s
>10
>00
>100
83.95
>10
Ours (Grain=2)
Im 36s
8.04
12.25
10.14
7.23
9.42
Ours (Grain=3)
Im 35s
8.53
13.18
10.85
7.73
10.07
LAS
N/A
7.06
10.49
8.57
6.51
8.16
SpikedAttention
2m 02s
9.46
14.02
12.08
8.17
10.93
8
Ours (Grain=2)
2m 01s
6.32
9.14
7.69
5.74
7.22
Ours (Grain=3)
Im 58s
6.37
9.22
7.73
5.79
7.28
ACC Perf. 1
T
Time Cost
WinoGrande
AreC
ArcE
PiQA
Avg. ACC
LLaMA-3-8B
N/A
N/A
72.85
53.33
77.74
80.85
71.19
SpikeLLM
8
6h 13m
69.38
49.23
73.11
78.67
67.60
TTFSFormer
128
N/A
72.69
52.90
77.65
79.33
70.64
LAS
N/A
71.19
51.88
73.48
79.33
68.97
Spiked/Atenttio
Im 38s
63.30
32.25
54.00
66.21
53.94
Ours (Grain=2)
Im 36s
73.16
47.87
73.74
77.64
68.10
Ours (Grain=3)
Im 35s
73.24
49.23
73.82
76.82
68.28
LAS
N/A
74.82
54.52
77.48
80.74
71.89
SpikedAttention
2m 02s
69.53
49.91
74.33
77.20
67.74
8
Ours (Grain=2)
2m Ols
72.69
54.35
78.11
80.14
71.32
Ours (Grain=3)
Im 58s
73.09
54.10
78.41
80.30
71.48
6.3 ENERGY ANALYSIS
To estimate energy consumption, we adopt the following theoretical energy estimation approach
according to Rathi & Roy 20200; Li et 2021) Zhou et al. (202 Deng et al. 2024)) Zhao al.
2025); Wang et al. (2022)) Chen et al. 2005a
Etotal
EMAC CountMAC EAC CoUntAc.
(18)

---

Published as a conference paper at ICLR 2026
where EMAC and EAC denote the energy consumption of single MAC and AC operation, spectively, while CountMAC and Countac denote the counted numbers of MAC and AC opera- tions during model inference. We measure MAC and AC operations using AMAC R 4.6p./ and EAC Ri 0.9pJ, as reported in 45 nm CMOS technology (Horowitz) 2014). The results indicate that, for single sample, the ANN-based LLM consumes 18.00 J, whereas distribution-aware spik- ing LLM consumes only 10.44~10.46 J as summarized in Table 3 ji..., our spiking LLM achieves 42.0% reduction in energy consumption of MAC and AC operations compared to its ANN coun- terpart. Furthermore, we anticipate that continued progress in neuromorphic hardware will further enhance the efficiency of our spiking LLM, potentially leading to even larger reductions in both operations and energy consumption.
Table The calculation count and the energy cost of ANN and our spiking LLMs with = 6.
Model
Method
Avg. ACC
Avg. PPL
Calculation Count
Energy Cost (J)
ANN
71.19
7.00
3912.08G MACs 0.17G ACs
18.00
LLaMA-3-8B
(Grain=2)
68.10
15,87G MACs + 11521.88G
10.44
Ours (Grain=3)
68.28
10.07
15.87G MACs + 11539.14G
ACs
10.46
CONCLUSION
To overcome latent conversion error arising from distribution misalignment, we propose multi- granularity phase coding, enabling SNN neurons to allocate mapped discrete values adaptively with respect to the activation distribution. Building on this coding scheme, we introduce a novel ANN- to-SNN conversion paradigm that leverages a cost-efficient alternating optimization neuron training algorithm to minimize conversion errors with respect to activation distributions. In future research, we intend to further advance our ANN-to-SNN conversion paradigm based on multi-granulann phase coding, targeting smaller total timestep and improved energy efficiency.
ACKNOWLEDGMENTS
Dr. Zhaogeng Liu was supported by the Young Scientists Fund (C Class) of the National Natural Science Foundation of China under Grant No. 62506142.
ETHICS STATEMENT
All participants in this work, as well as the paper submission, adhere to the ICLR Code of Ethics ( https//iccccccpublicCCooooEeeett
REPRODUCIBILITY STATEMENT
We affirm that the results of this work are fully reproducible. Appendix A provides the theoretical proofs. Appendix deI the experimental implementations, and the source code is available at https: /Etps: com/n http://jjjnngggySSS/pee StipghyyySpiiiiLL...
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint ar 2rXi::2303.08874, 2023.
Malyaban Bal and Abhronil Sengupta. SpikingBERT: Distilling bert train spiking language mod- els using implicit differentiation. In Proceedings the volume 38, pp. 10998-1 1006, 2024.
the AAAI conference on artificial intelligence,
10

---

Published as a conference paper at ICLR 2026
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelli- gence, volume 34, pp. 7432-7439, 2020.
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-aceuracy and ultra-low-latency spiking neural networks. arXiv preprint arXiv:2303.04347, 2023.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1)544-66, 2015.
Long Chen, Xiaotian Song, Andy Song, BaDong Chen, Jiancheng Lx, and Yanan Sun. FAS: Fast ann-snn conversion for spiking large language models. arXiv preprint arXiv:2502004405. 2025a.
Long Chen, Xiaotian Song, and Yanan Sun. Las: Loss-less ann-snn conversion for fully spike-driven large language models. arXiv preprint arXiv:2505. 09659, 2025b.
Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers in Ilms, arXiv preprint arXlv:2410.05265, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv:1803.055. 05457, 2018.
Manon Dampfhoffer, Thomas Mesquida, Alexandre Valentian, and Lorena Anghel. Are snns re- ally more energy-efficient than anns? an in-depth hardware-aware study. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(3):731-741, 2022.
Gourav Datta and Peter A Beerel. Can deep neural networks be converted to ultra low-latency spiking neural networks? In 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 718-723. IEEE, 2022.
Alex De Vries. The growing energy footprint of artificial intelligence. Joule, 7(10):111-2119, 2023.
Shikuang Deng, Yuhang Wu, Kangrui Du, and Shi Gu. Spiking token mixer: An event-driven friendly former structure for spiking neural networks. Advances in Neural Information Processing Systems, 37:128825-12288 1288466 2024.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference neural networks (IJCNN, pp. 1-8. icce, 2015.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference deep spiking neural networks. arXiv preprint arXiv:2105. 1164, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. -2077 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101. 00007,000 2020.
Robert M. Gray and David L. Neuhoff. Quantization. IEEE transactions on information theory, 44 (6):2325-2383, 2002.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European conference computer vision, pp. 388 404. Springer, 2020.
Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion error through residual membrane potential. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11-21, 2023a.
11

---

Published as a conference paper at ICLR 2026
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between anns and snns by calibrating offset spikes. arXiv preprint WrXi::2302.10683. 2023b.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers' (ISSCC), pp. 10-14. IEEE, 2014.
Sangwoo Hwang and Jaeha Kung. One-Spike SNN: Single-spike phase coding with base manip- ulation for ann-to-snn conversion loss minimization. IEEE Transactions on Emerging Topies in Computing, 13(1):162- 172-11222-722222 2024.
Sangwoo Hwang. Scunghyun Lee, Dahoon Park, Donghun Lee, and Jacha Kung. SpinedAttention: Training-fre free and fully spike-driven transformer-to-smn conversion with winner-oriented spike shift for softmax operation. Advances in Neural Information Processing Systems, 37:67422- 67445, 2024.
Anton Iliev Iliev, Nikolay Kyurkchiev, and Svetoslav Markov. On the approximation of the cut and step functions by logistic and gompertz functions. Biomath, 410111-151011,, 2015.
Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint WwXiv:2002.03322 2020.
Jachyun Kim, Heesu Kim, Subin Huh, Jinho Lee, and Kiyoung Choi. Deep neural networks with weighted spikes. Neurocomputing, 311:373-386, 2018
Nikolay Kyurkchiev and Svetoslav Markov. Sigmoid functions: some approximation and modelling aspects. LAP LAMBERT Academic Publishing, Saarbrucken, 4:34, 2015.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Yuhang Li, Yufci Guo, Shanghang Zhang, Shikuang Deng, Yongging Hai. and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in 2021.
neural information processing systems,
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018.
Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, and Peter Richtárik. Randomized Asymmetric Chain of LoRA: The first meaningful theoretical framework for low-rank adaptation. arXiv preprint arXiv:4410008305, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 17X1v:1609.0770 2016.
Marcelo A Montemurro, Malte J Rasch, Yusuke Murayama, Nikos K Logothetis, and Stefano Panz- eri. Phase-of-firing firing coding of natural visual stimuli primary visual cortex. Current biology, 18 (5):375-3800 2008
Hesham Mostafa. Supervised learning based temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems, 29(7):327733255, 2017.
Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the power of gradient-based optimization spiking neural networks. IEEE Signal Processing Magazine, 36(6):5 2019.
Nam Nguyen, Deanna Needell, and Tina Woolf. Linear convergence of stochastic iterative greedy algorithms with sparse constraints. IFFE Transactions on Information Theory, 63(1 1):6869 6895, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(110):1-66 167,, 2020.
12

---

Published as a conference paper at ICLR 2026
Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimiza- tion in deep spiking neural networks. arXiv preprint 0rXiv:2008.03658, 2020.
Bodo Rueckauer and Uhih-Chii Liu. Conversion of analog to spiking neural networks using sparse temporal coding. In 2018 IEEE international symposium circuits and systems (ISCAS), pp- 1-5. IEEE, 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An ad- versarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Abhronil Sengupta, Yuting Ye, Robert Wang. Chiao Liu, and Kaushik Roy. Going Deeper in Spiking Neural Networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, ben fattori, Charles Lovering, farzanehnakhaee70, Jason Phang, Anish Thite, Fazz, Thomas Wang, Niklas Muennighoff, Aflah, sdtblek, nopperl, gakada, tttyuntian, researcher2, Chris, Julen Etx- aniz, Hanwool Albert Lee, Zdenčk Kasner, Khalid, Jeffrey Hsu, Anjor Kanekar, Pawan Sasanka Ammanamanchi, Vicki Boykis, and AndyZwei. 2024.
v0.4.2, March
Qwen Team et al. Qwen2 technical report. arXiv preprint arXXv:2407.10671, 2(3), 2024.
Bu Tong, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ANN- SNN conversion for high-accuracy and ultra-low-latee latency spiking neural networks. In Interna- tional Conference on Learning Representations, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Yuchen Wang, Malu Zhang, Yi Chen, and Hong Qu. Signed neuron with memory: Towards simple, accurate and high-efficient ann-snn conversion. In I/CAI, pp. 2501-2503 2022.
Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. RedPajama: open dataset for training large language models. Advances neural information processing systems, 37:1 16462- 116492, 2024.
Jibin Wu, Yansong Chua, Malu Zhang, Qu Yang, Guoqi Li, and Haizhou Li. Deep spiking neural network with spike count based learning rule. In 2019 International Joint Conference on Neural Networks (IJCNN, pp. 1-6. IEEE, 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-tempores backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Xingrun Xing, Boyan Gao, Zheng Liu, David A Clifton, Shitao Xiao, Wanpeng Zhang, Li Du, Zheng Zhang, Guoqi Li, and Jiajun Zhang. SpikeLLM: Scaling up spiking neural network to large language models via saliency-based spiking. In ICLR, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Qu Yang, Jibin Wu, Malu Zhang, Yansong Chua, Xinchao Wang, and Haizhou Li. Training spiking neural networks with local tandem learning. Advances in Neural Information Processing Systems, 35:12662-12676, 2022.
Kang You, Zekai Xu, Chen Nie, Zhijie Deng, Qinghai Guo, Xiang Wang, and Zhezhi He. Spikezip- tf: conversion is all you need for transformer-basee snn. In Proceedings of the 41st International Conference Machine Learning, pp. 57367-57383, 2024.
Xinzhe Yuan, William de Vazelhes, Bin Gu, and Huan Xiong. New insight of variance reduce in zero-order hard- wond-hhresholding: Mitigating gradient error and expansivity contradictions. ICLR, 2024.
13

---

Published as a conference paper at ICLR 2026
Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural computation, 33(():8999 999, 2021.
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 365-382, 2018.
Ming Zhang, Zonghua Gu, Nenggan Zheng, De Ma, and Gang Pan. Efficient spiking neural net- works with logarithmic temporal coding. IEEE access, 8:981 198855-98167, 2020.
Lusen Zhao, Zihan Huang, Jianhao Ding, and Zhaofei Yu. TTFSFormer: A ttfs-based lossless con- version of spiking transformer. In Forty-second International Conference on Machine Learning, 2025.
Tang Zhengzheng and Eva Zhu. BrainGPT: A brain-inspired SNN-based large language model, 2025.
Xingyu Zhou. On the fenchel duality between strong convexity and lipschitz continuous gradient. 2018.
arXiv preprint arXiv:
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.1544 2022.
Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason K Eshraghian. SpikeGPT: Generative pre-trained language model with spiking neural networks. arXiv preprint 2rXiv:2302.1399) 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A THEORETICAL ANALYSIS
A.1 DEFINITION OF NOTATION
Table 4: Notation
Notation
Description
[ 0, B)
Objective function.
g(,, 0, B)
Smoothed objective function. The parameter value at the k-th iteration. Output weight parameters of SNN neurons.
Ok or Bx
d
B
The phase variable to be optimized.
h
Reset strength parameters of SNN neurons. Threshold parameters of SNN neurons. Treat h and O single parameter o. The minimum of g(o B) with B fixed. The minimum of fo, B) with fixed. Learning rate for updating and (for updating o). Learning rate for updating B.
gx (o, B)
f-(o) (o, B)
7
722
L1
Lipschitz constant of function g(o, B) w.r.. variable o.
La
Lipschitz constant of function f(o, B) w.r.t variable B. PI. condition constant. Maximum error between f(o, B) and g(o, B) under identical parameters.
A.2 PROOFS
Lemma 1. Assumption 3 holds, then If -g"l S a, where f* denotes the global minimum value of (o, B).
Proof Lemma
If f" = f(; B,) and Assumption 3 holld we have,
g" s B,) I f(o,, B,) + + Similarly, if g(0g. Bj) have,
(19)
f s f(o,, Bj) g(o,, Bj) to 9" +o.
(20)
So we have,
- - So.
(21)
Proof of Theorem
From Assumption we have,
Li g( 1, Bx) S g(0x. B₁) + (Vog(ox, Bx), ot+1 - ok) |
(22)
Due to OK+1
V Vag(o;, BR), we obtain,
Tn
g(o++1, B&) < g( Bb) m vllVVo((,, BB)
Vog(Oz, Bx)l
2
(23)
= BB) + Lin
m V (og B
2
Let m < min( 1 T T we have,
m Vog(or, Bx)
g(ox-1. Bx) S g Bb)
(24)
2
15

---

Published as a conference paper at ICLR 2026
From Assumption we have,
Vg B)I = [o,, + B)ll 24119(o, B) g"1.
(25)
Rearranging the above equation, we have,
[Vog(o, B)ll > 2m (g(o, B) -) VV Bg(o, B)]l
(26)
g(o++1, S g( B₂) uith lo(ox. Bx) - -g1 " Subtracting 9² from both sides, we have,
VBglos. Bx)l
(27)
g(ox+1, B&) - 9" - g(ok, Bu) g" - -1m (g Bx) 9'1
m 2
V Bg Bx)
(28)
(1 - wim) lø(ox. B&) - g"1+ "1 VIV B
Next, we prove the inequality result obtained when updating B. From Assumption we have,
L2
f(ox41, Bx41) - f(o++1, Bx) + (VBf(ou+1. Ba), Bu+1 B&) + 2
B - Bxl3. (29)
Due to BB+1 B₁ V B. Bs), we obtain,
Leni 2
f(0:-1: & f(o++1, Bb) 72 72 VBllVB((o Bx)l
[BB(O Bx)l
(30)
f(ox+1, Bb) L m) Bx)l
Let < 722 S min La Long ) we have,
f(04+1, BR+1) S f(0k+1, B&) 12 VBJ(o B-)
(31)
From Assumption we have,
Vai(ou41 B 2 2p3 f(ox+1, BB) (ox+1,
(32)
Substituting Equation 32 into Equation (31) we obtain,
f(o++1, B I f(ox+1, B₂) p372 ((0tt1, (
(33)
Subtracting f from both sides, we have,
[(0411, Bx+1) - f(o+1 Bb) l 1133/2 [[ Bτ) (ox+101 B)
(34)
- (372)) [f(0 Bx) fr1 u372 polo) (ox+1. B) - -
Next, combine the results obtained above. Applying Assumption 3 and Lemma to Equation 28 we have,
f(o+11 BB) - 22 (1 pmm) [f(or: B&) 2a]
V (o,, Bx)
(35)
f(0K11, B&) 5r pmm) [o B&) f + +20
+
?
V
(o BB) + 20
2
(1 pmm) [f B&) - -f 20 pmm)
7
V Bg Bx)l
(36)
Substituting Equation 36 into Equation 34 we obtain,
f(o+11, Bkt1) f < (1 Ha722) (1001111mmml) [f(ok, [f Bx) - fr1 20 1117))1111111111111 H3Tt2
m1 (1 - -33)))22 V B2) + M3712 (0(0) B) - -5-
(37)
16

---

Published as a conference paper at ICLR 2026
Since (0)0 B) - f* g (0x+1 B) g" + 2a and from Assumption 2 we have,
(ox+1 B) f V Vog (o) (o++1, B)1
+ 2a.
(38)
2p2
Substituting Equation (38) into Equation
36
obtain,
f(ox+1, Bk+1) f - (1 p372) (1mm))11-mm)(foe
of -x)- - - (2-22022-1111111
- 11111 M3722)
(1 - (1-3332))BBB000 Bb]l 2 1372 V Vog (0) (0) (oN+1. B)1
+ 2
22
2/12
(39)
Simplifying yields,
f(o+11, Bk+1) - < (1-132)) (m)) Bx) - C.
(40)
where
C ==20 - (1mm -13372) 2x +
2 - (1332))) B&)l1
(41)
+ |332 | ogg (o)
(ox+11 B)
2u2
The above corresponds to the case where both the inner iteration counts of and B are equal to one. We now consider the case where the inner iteration count of o is N1, and that of B is N2- In fact, the above result still holds when the number of inner iterations is not equal to one. This because, based on Assumptions [ and 2 and by employing an argument similar to that used in deriving Equation (33), we can establish the following inequality:
" (( ((())) 9- (B) B
(a)) s (1 (--m)))
[( (a)) 9¹ (B B B
(42)
(Ni) B (o (Ni) f(ott) k+1
k+1 B) & (1 ma72)
f( k+1 - (o
(Ni) (iz)
(ok11 B)
(Ni)
(43)
Since < 7n S min(
1 Li Limi
and < 1/2 & min 1
1a L2gg) let 0k = -k- Ok
(Ni)
Ok.
B (0) = B and B = B we have,
g
(i) Ba), (1++1) BB) < g(
(44)
f(ox+1, Bl < f(o++1 B(rr))
(45)
By applying the above equation, we obtain,
(N1) g(04+1, B+) -g g(0 B&) g( Bs),
(46)
f(o+11. BB41) = f(o.11,
(2)) < foxt11 k+1
(47)
For the case in which the number of inner iterations differs from one, applying Equations 46 and
47) yields a result analogous to Equations (34) and (36), with the distinction that o and B here represent the values at the end of each inner loop. We have,
((o.1,,B411) - f < f(ox+1. B
(48)
< (1 p [f Bx) -f Hs72 ( (ox+1: B)- -
f(o 8r < f( k+1 B&) f
" [V B2)
< (1 mmm) (O): Bt) - - 20 /1m)
(49)
Based Equations
48 and
49
obtain exactly the same result in Equation 39
17

---

Published as a conference paper at ICLR 2026
B CONVERT NONLINEAR OPERATION IN LLM
Attention Layer. The attention architecture of our method is presented as follows:
T K Σ K.t,
T
T
Q ~ Σ Qc
V =
s,s.
(50)
A=1
where Q, K and V denote the query Q, key and value V, and Qa, K and V., denote the spiking query, key and value at timestep I.
Then, we need enable the which occurs between the query Q and key K as well as attention array A = QK and value V. Fortunately, You et al. (2024) have paved the way for such AA multiplication. Specifically, taking the multiplication between query and key as example, it can be written as:
(AA) multiplication in attention within SNN,
T
A =
Q
K2
Q
K.,
1=1
1=1
(Sq. Katt
Qa,t SK,,
Qs,t K.,e,,
(51)
where Sq. and Sk,; represent the accumulated spike output of query and key from 1 to Therefore, the result AA multiplication at each time / Sq.: K,, Qs,t Sk - Qs,t K,
Spiking Softmax, Spiking RMSNorm and Spiking SiLU Activation. Inspired by the literature You et al.) 2024), we use the following process to enable Softmax, RMSnorm, and SiLU activation in SNN.
T(t) = [(I - 1))
(52)
O(t) ((((t))
(53)
O(t) = (()) - (t
(54)
where I(t) is the accumulated input at timestep; [(t the input i timestep; o(-) is the Softmax, RMSnorm and SiLU activation and O(t) is the output at timestep.
Spiking MIP. Except for the linear layers MLP. the important operation is the Activation- Activation Hadamard product, which exists between the output of linear layer and the output of the spiking SiLU function. It can be written as
T
T
A
Σ
A
B
(55)
A. O O A,
O B
A B
Σ
Liigt
where A denotes the output of the up-proj and B denotes the output of Spiking SiLU. Therefore,
the result of Hadamard product each time is A, B. E Ei=1itt A.OB, +A,OB,
C ADAPTIVE GRANULARITY ALLOCATION
We propose differentiable adaptive search algorithm for granularity allocation in phase encoding. In particular, we select the optimal granularity allocation based on the data distribution. To achieve this, perform 10,000 down-sampling operations the activation values during the training pro- cess of LLaMA-2 and LLaMA-3 the experiment. Furthermore, we leverage the differentiable search framework presented in (Liu et al. 2018) to compute the Softmax of the different granularity allocation weights, as follows:
exp(as)
σ(x)=
oEO Eveo cxp(a,)) (z).
Σ
(56)
18

---

Published as a conference paper at ICLR 2026
Here, O represents the set of granularity candidates, and a represents the granularity allocation coefficients. Based on this, our optimization objective can be formulated as:
min LMSE(a;
(57)
s.t.
W0
= arg min (sss(w;)))
(58)
where w(,, 0: B) represents the parameters of the SNN neurons. We obtain the optimal architecture parameter a as shown in Algorithm 2 and subsequently retrain the SNN neuron with the optimal granularity allocation.
Algorithm 2 Differentiable Architecture Search with Adaptive Granularity Allocation
Input: Training dataset X, neuron patameter u (h, 0; B), learning rate yr optimization steps No.
2: for io, NN do
3: Obtain w" by using Alogrithem Junder N1 = N2 1.
4: Update a by a = a VaLusE w", X))
5:
end for
6: return ax.
D MORE EXPERIMENTAL RESULTS
D.1 IMPLEMENTATION DETAILS
We conduct experiments on a server equipped with multiple 80GB NVIDIA A100 GPUs. For full pull-precision ANNs of the LLaMA family, use open-source models from the HuggingFuce and evaluate their performance under the FP16 setting. For SpikeLLM, we employ the released open- source implementation and assess its performance in the W4A8 configuration (4-bit weight and 8- bit activation). For TTFSFrrmer, since the original implementation is not available on LLaMA, we implemented simplified version by applying time-to-first-ppee coding within our code framework. For LAS, since its available open-source implementation is not adapted to the LLaMA model, we construct simplified implementation by setting the SNN neuron parameters as (l) h(t)
d(t) T 2- and regard it as the special case of our method without optimization. For the same reason, we also implement a simplified version of SpikedAttention in our code framework by setting the SNN neuron parameters as ((() h(t) d(() = 7 2- and applying the single-spike technique worth noting that under limited total timestep experimental setting, the single-spike technique leads to collapse in performance. Therefore, we relax this technique to allow two spikes instead one. To handle outliers in LLMs, we apply the Hadamard rotation and prefixed outlier tokens techniques introduced in the literature (Chen al. 2024).
(Hwang & Kung 2024).
D.2 ABLATION STUDY
Effectiveness of MuanirGranularity. To verify the effectiveness multi-granularity in our pro- posed phase coding, we perform an ablation study varying the number of granularities, and the results are presented in 5 We observe that the best accuracy and perplexity are not achieved with single granularity, which demonstrates the effectiveness of design. is worth noting that increasing the granularity does not necessarily lead to better results. When we increase the number of granularities, the model can non non-unifommly allocate discrete values more flexibly. The solution space with Grain = 2 or 3 strictly contains the Grain = solution space, so, in principle, more gran- ularities can only help. However,
practice, this larger solution space also makes optimization more non-convex and prone to local minima. Our ablation studies also confirm exactly this trade- off. When we push granularity to the extreme (e.g., setting the number of granularitics equal to the timestep T), the performance actually is not the best, indicating that excessive granularity makes the optimization harder and the solution is more likely to be suboptimal.
Interaction between the Timestep T and the Number of Granularities. We observe that the effect of granularity depends on the timestep T, and we should study this more systematically. Specifically, we provide bases B and training loss curves for different combinations of timestep and
19

---

Published as a conference paper at ICLR 2026
2x10
1x30
1.0+10-1
2,000
9x:0
2.000
1.6x10
1,995
Bx10
14410
1,990
Fx10
1,2110
1.903
6x:00
1x20
LOS5
5x10
1.97%
0x30-
1.970
1.992
1.965
5x20
3x10
1.900
4x10-1
100
150
200
230
300
350
400
2.5x10
50
100
150
200
250
300
350
400
Epach
Epoch
T=6, g=1
T=8, 9=1
2x10
1x10
188x0-
2.00
9x10
2,000
16x10
144x00
12x10
6x10
valuea
1x10
1.98
LosS
5x10
1,994
8X10
1.992
6x10
1.900
1.96
5x30
3x10
4x10
200
250
1.95
25x10
150
250
Epoch
Epoch
T=6, 9=2
T=8, g=2
2+10
1x30-
18x10
9x10
16x10>
1995
Bx10-4
2.000
7x30
1.955
Base values
1x10
LooS
5x.0
1. 1.900
4x10
1.955
6x10
5x10
3x10-
4x10
50
100
150
200
250
300
350
400
1.965
2.3x30
50
100
150
200
250
300
350
400
Epoch
Epoch
T=6, g=3
T=8, g=3
h, training stage - B training stage --- Bo
---
B₁
B2
The alternating red and blue curves trace the loss dynamics during staged optimization, with red intervals indicating updates to h and 9, and blue intervals corresponding to B training. Dashed curves denote the evolution of base values across granularities Br, B1, and B2.
Figure 4:
granularity, showing how the optimization converges. These curves in Figure 4 clearly illustrate the evolution of and the training loss during the optimization process under different timesteps and numbers of granularities. Conceptually, in our phase-coding neuron, the number of representable discrete values grows as 2". When is large, the discrete representation is already quite dense, so redistributing these discrete values via multi-granularity provides smaller gains. This explains why, in Tables I 2 and 9 the improvement from increasing Grain a larger T appears modest. In contrast, when T is lower, the total number of discrete values is more limited, so where these values are placed becomes much more critical. In this regime, multi- granularity can reduce conversion error by allocating more resolution high-density regions of the activation distribution.
20

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study the number of granularities with T 8.
Models
Grain
Avg. ACC
Avg. PPL
67.03
7.01
2
67.17
7.16
LLaMA-2-7B
3
67.31
7.68
T
67.14
7.44
I
71.09
7.24
2
71.32
7.22
LLaMA-3-8B
3
71.48
7.28
T
71.40
7.27
Comparison between Joint Optimization and Alternating Optimization. To demonstrate the effectiveness of alternating optimization, we additionally conduct an ablation study using joint op- timization. In this setting, we observed a worse downstream performance this is because, without decoupling the two stages, updates to B and h. 0 interfere with each other. In our method, these parameters play different roles. The bases B determine the distribution of discrete representable values. By using multiple bases, shape how discrete values are distributed to better match the activation distribution. The neuron parameters h, determine how a given con- tinuous input is mapped one of those discrete values, i.e., how the spike dynamics choose which discrete value used. We explicitly separate them so that our method to effectively minimize the conversion error arising from distribution misalignment, Moreover, from the perspective of the vergence of alternating optimization, the loss function is non-differentiable with respect to and O, so backpropagation for these variables must rely on surrogate gradients, whereas the optimization of B can directly use the true gradients. Therefore, intuitivell. if h. O, and B are optimized simulta- neously, the errors introduced by the surrogate gradients will propagate the updates of B, thereby amplifying the overall optimization error and reducing the stability of convergence.
Table 6 We believe
Table 6: Results joint optimization and alternating optimization. "PPL." denotes the perplexity on Wikitext2.
Model
T/Grain
Method
WinoGrande AreC
ArcE
PiQA
PPL
Joint
70.09
45.22
73.99
77.86
6.58
8/2
Alter
70.56
46.16
73.99
77.97
6.71
LLaMA-2-7B
Joint
69.85
45.65
73.78
77.69
6,41
8/3
Alter
70.96
46.08
74.33
77.86
7.10
Joint
72.38
47.18
71.38
76.55
7.61
6/2
Alter
73.16
47.87
73.74
77.64
8.04
LLaMA-3-8B
Joint
71.67
47.18
72.39
74.54
7.52
6/3
Alter
73.24
49.23
73.82
76.82
8.53
Decoupling h and 0 from Each Other: In conventional formulations, h and 0 are often tied (e.g., h 0), which reduces the degrees of freedom of the neuron dynamics. In our setting, once the discrete values (determined by B) fixed, the neuron still needs enough flexibility mapping from continuous activations to these values. By allowing h and O to vary independently. we can increase the expressive power the neuron dynamics and enable finer adjustment of the mapping between continuous activations and discrete values. Empirically, shown in Table 7 observe that this extra flexibility helps reduce the approximation error between the SNN neuron output and the original ANN activation.
shape the
Weight Quantization. We apply weight quantization to the LLaMA-2-7B and recompute ac- tivation distributions under 8-bit and 4-bit weights. As shown in Figure 5 observe that while quantization slightly changes the exact shape of the distributions, the activations remain highly non- motivation of our distribution- uniform and layer-dependent, so the muulti-ganuulrityy phase coding under quantized weights, in- We also evaluate our distribution-awarr multi cluding 8-bit and 4-bit settings. The results in & that approach maintains competitive
design still holds.
21

---

Published as a conference paper at ICLR 2026
Table 7: Results on whether decouple h and 0. "PPL" denotes the perplexity on Wikitext2.
Model
T/Grain Decouple WinoGrande AreC
ArcE
PiQA
PPL
No
70.24
45.65
74.03
77.86
7.62
8/2
Yes
70.56
46.16
73.99
77.97
6.71
LLaMA-2-7B
No
70.17
45.90
73.86
77.97
6.90
8/3
Yes
70.96
46.08
74.33
77.86
7.10
No
71.98
45.90
71.80
75.03
8.00
6/2
Yes
73.16
47.87
73.74
77.64
8.04
LLaMA-3-8B
No
73,40
47.78
72.26
75.35
7.63
6/3
Yes
73.24
49.23
73.82
76.82
8.53
performance under 8-bit and even 4-bit weights, demonstrating that our method is compatible with weight quantization.
Table 8: Results on LLaMA-2-7B with weight quantization. "PPL" denotes the perplexity Wiki-
tex12.
Method
T
Weight Bit WinoGrande
AreC
ArcE
PiQA
PPL
LLaMA-2-7B
N/A
16
69.06
46.33
74.54
79.05
5.47
Ours (Grain=2)
70.09
45,48
73.82
77.64
7.56
8
8
Ours (Grain=3)
70.40
45.82
74.16
77.53
7.25
Ours (Grain=2)
67.80
42.75
71.21
76.88
8.91
8
Ours (Grain=3)
68.51
43.26
71.04
77.15
8.41
Table 9: Results on LLaMA-2-133 2-MM-----1BB.. "Time Cost" denotes the training time required for the method to obtaín the SNN. "Grain" denotes the number of granularities.
PPL Perf. 1 T
Time Cost
Wikitext2
C4
Redpajama
Pile
Avg. PPL
LLaMA-2-13B
N/A
N/A
4.88
6.47
5.19
4.34
5.22
Spikel.LM
8
10h 41m
5.20
6.91
5.57
4.63
5.58
LAS
N/A
18.02
21.82
17.02
11.85
17.18
SpikedAAtention
2m 35s
8.90
12.97
10.74
8.52
10.28
8
Ours (Grain=2)
2m 35s
5.07
6.74
5.39
4.51
5.43
Ours (Grain=3)
2m 34s
5.29
7.40
5.91
4.83
5.86
LAS
N/A
5.03
6.76
5.38
4.48
5.41
SpikedAAtention
3m 05s
6.43
8.54
6.94
5.80
6.93
10
Ours (Grain=2)
3m 05s
4.90
6.54
5.23
4.38
5,26
Ours (Grain=3)
3m 06s
4.90
6.54
5.23
4.37
5.26
ACC Perf. f
T
Time Cost
WinoGrande
ArcC
ArcE
PiQA
Avg. ACC
LLaMA-2-13B
N/A
N/A
72.45
49.15
77.44
80.52
69.89
SpikeLLM
8
10h
41m
69.30
47.27
76.22
79.05
67.96
LAS
N/A
72.77
51.28
77.27
80.14
70.37
SpikedAttention
2m 35s
72.38
45.82
74.54
78.07
67.70
8
Ours (Grain=2)
2m 35s
73.24
50.43
77.31
80.47
70.36
Ours (Grain=3)
2m 34s
73.72
50.60
77.53
80.14
70.50
LAS
N/A
72.53
50.17
77.10
80.85
70.16
Spiked/ttention
3m 05s
70.40
44.88
74.20
78.24
66.93
10
Ours (Grain=2)
3m 05s
72.69
49.91
77.06
81.01
70.17
Ours (Grain=3)
3m 06s
72.77
50.00
77.10
80.74
70.15
22

---

Published as a conference paper at ICLR 2026
Number of Activation
Number of Actvvtion
Number of Actvation
043
Absolute Activation Range
Absolute Activation Range
Absolute Activation Range
Figure 5: Activation distribution after weight quantization.
D.3 RESULTS ON LARGER-SCALE LLM
In Table 9 provide comparison of our method with other baselines larger-scale LLM (LLaMA 2- 13B). Similar phenomena are observed on LLaMA-2-13BB as on LLaMA-2-7B and LLaMA-3-8B, which fully demonstrates that our method remains effective for larger-scale LLMs.
D.4 RESULTS ON MULTIMODAL MODEL
Evaluating method beyond language models further strengthen the empirical evidence for its effectiveness. To this end, we have extended our distribution-aware from LLMs multimodal model, which is structurally and functionally different from language models. Specifically, we extend our method to CLIP and evaluate the performance of the spiking CLIP model on image classification tasks. The results in Table T0 show that our method can be successfully applied in this setting as well.
culliiggranularityy phase coding
Table 10: Performance Comparison Results on ImageNet, CIFARIO, and CIFARIO 100 using CLIP model. "FP32" represents the performance of the ANN evaluated under the float32 precision.
Model
Method
ImageNet
CIFAR10
CIFARI00
Avg. ACC
FP32
N/A
57.71
89.69
64.01
70.47
LAS
55.42
89.27
66.22
70.30
ViT-B/32
Ours (Grain=2)
8
56.72
90.48
66.11
71.10
Ours (Grain=3)
56.78
90.23
65.64
70.88
FP32
N/A
63.42
90.82
67.07
73.77
LAS
58.87
84.59
59.77
67.74
ViT-B/16
Ours (Grain=2)
60.68
89.70
65.49
71.96
Ours (Grain=3)
8
61.22
89.77
65.25
72.08
FP32
N/A
71.13
95.82
76.41
81.12
LAS
8
69.71
88.82
70.29
76.27
ViT-L/14
Ours (Grain=2)
69.61
94.99
77.61
80.74
Ours (Grain=3)
8
69.63
94.87
77.17
80.56
D.5 RESULTS ON OTHER LLM
To enhance the completeness of our method, we add additional experiments on Qwen2-7B Team et al. 2024) using our proposed multi-granularity phase coding. The results in Table show our method maintains high performance on Qwen2-7B, further demonstrating its effectiveness and scalability.
show that
23

---

Published as a conference paper at ICLR 2026
Table 11: Results Qwen2-7B. "Grain" denotes the number of granularities. "PPL." denotes the
perplexity on Wikitext2.
Method
T
WinoGrande
ArcC
ArcE
PiQA
Avg. Acc
PPI.
Qwen2-7B
N/A
72.38
49.91
74.71
81.23
69.56
7.14
LAS
8
70.96
50.60
74.20
80.52
69.07
10.18
SpikedAttention
8
61.96
28.33
48.96
65.13
51.10
>100
Ours (Grain=2)
8
73.40
50.60
74.12
81.01
69.78
7.41
Ours (Grain=3)
8
72.53
50.60
74.07
80.85
69.51
7.42
D.6 MORE RESULTS LOWER TIMESTEP
To further demonstrate that our method is scalable to a lower timestep, we also include experiments with T=6 on Llama-2-7B, and the results are Table Our method significantly reduces perplex- ity compared to all baselines without sacrificing accuracy.
Table 12:
Results LLaMA-2-7B with T 6. "Grain" denotes the number of granularities. "PPL"
denotes the perplexity on Wikitext2.
Method
T
WinoGrande
AreC
ArcE
PiQA
Avg. Acc
PPL
LLaMA-2-7B N/A
69.06
46.33
74.54
79.05
67.25
5.47
LAS
67.96
44.28
72.52
77.86
65.65
45.50
Spiked/Attentinn
66.69
41.64
70.03
76.77
63.78
50.05
Ours (Grain=2)
67. 67.6
45.31
72.26
77.58
65.70
12.19
Ours (Grain=3)
68. 88.98
44.37
72.52
77.86
65.93
10.79
D.7 MORE ENERGY ANALYSIS
For the energy comparison with other Spiking LLM,, we report the energy consumption data for SpikeLLM on LLaMA-3-8B. For the energy consumption calculations of both SpikeLLM and our Spiking LLMs, we employ identical configurations and perform a statistical analysis of the MACs and ACS generated by the same components. The results in Table 13 demonstrate that our method achieves lower energy consumption compared to SpikeLLM.
Table 13: The calculation count and the energy cost of ANN, SpikeLLMM, and method LLaMA-3-8B.
Method
Calculation Count
Energy Cost (J)
ANN
3912.08G MACs 0.17G ACs
18.00
SpikeLLM
2.79G MACs 14507.31G ACs
13.87
Ours (Grain=2)
15.87G MACs + 11521.88G ACs
10.44
Ours (Grain=3)
15.87G MACs 11539.14G ACs
10.46
The memory access and data movement are the primary sources of energy consumption (which we refer to as the read/write cost) on existing hardware (Dampthoffer et al. 2022). In order to further validate the effectiveness of our method, we expand our energy analysis to explicitly include the costs associated with read/write. Specifically, for the calculation of energy consumed by reading and writing weights and activations, we refer to the energy estimation approach for both ANN and SNN models presented in Hwang al. (2024). We set the 32-bit read/write energy for weights and activations, Eread and Eurites to 5 p./ and conducted comparison of the energy consumption between ANN and spiking L.L.M The total energy equations for ANN and SNN are given by:
24

---

Published as a conference paper at ICLR 2026
-2eed Countread + Ewrite Countwrite -MMA CountMAC EAC Count AC
total
ESNN
Enearon + (1 1/32) Σ
Count + 1/32 Ewrite
Σ
Count write
EMAC CountMac + EAC Countac
(59)
Where Eneuron represents the energy consumption of neuron, and Eread and Ewrite denote the energy consumption for read and write operations, respectively. Countread and Countwrit. denote number of read and write operations. The factors 1/32 in the SNN formula are due to fact that activations in SNNs are represented using the 1-bit spike. Table 14 and Table 17 show that memory access accounts for at least 68% the total energy consumption both ANN and current SNN models, making it the primary source of energy consumption. Nevertheless, thanks to the sparsity inherent in SNN computations, our results show that the total energy consumption of method remains over 12% lower than that of the ANN.
Table 14: Energy consumption of LLaMA-3-8P under ANN and our method, including both read and write operations.
Method
Read/Write Cost (J)
MAC & AC Cost (J)
Total Energy Cost (J)
Total Energy Cost Relative to ANN
ANN
38.47
18.00
56.47
100.00%
Ours (Grain=2)
38.35
10.44
48 84
86.48%
Ours (Grain=3)
38.93
10.46
49.43
87.53%
Table 15: Relative energy consumption of LLaMA-3-8B under ANN and our spiking LLMs, includ- ing both read and write operations.
Method ANN
Proportion of Read/Write Cost
Proportion of MAC & AC Cost
68.12%
31.88%
Ours (Grain=2)
78.52%
21. 31.38%
Ours (Grain=3)
78.76%
21.16%
As supported by Dampfhoffer al. (2022), the energy consumption of SNNs is closely tied the spike firing rate. To address this issue, we propose a masking mechanism. Specifically, we exploit the characteristic of phase coding, where the encoding value decreases as the timestep increases. Consequently, spikes from neurons that fire early can be considered redundant, and those occurring at later timesteps can be discarded. This strategy effectively reduces the spike firing rate by elim- inating redundant spikes without significantly impacting performance. As result, the increased activation sparsity leads to a substantial reduction in the overall energy consumption of the SNN. In energy estimation, upper-bound this cost by assigning the mask the same energy as a full neuron-level computation. Nevertheless, even under this assumption, the mask-related cost still accounts for only a small frac- tion of the total SNN energy, as neuron computation contributes relatively little compared with data movement and memory access. The lower spike firing rate resulting from the masking operation ultimately yields lower value for Count than for Count in the energy calculation. The results of energy consumption with mask are in Table 16 and Table I7
also include the cost of the masking operation. To be cautious,
-2 Eneuron + (1 1/3a) Σ Count + 1/32 Ewrite Σ Count
total
+ EMAC Count + EAC Count
(60)
Where Countread and Counturite represent the number of read and write operations, and Count AA and Ac represent the number of MAC and AC operations, all after reducing the spike firing rate.
25

---

Published as a conference paper at ICLR 2026
Table 16: Energy Consumption of LLaMA-3-8B for ANN and spiking LLMs with mask, in- cluding both read and write operations. "PPL" denotes the perplexity on Wikitex12.
Method
Avg. ACC
PPL
Read/Write Cost (J)
MAC Cost (J)
Total Energy Cost
Total Energy Cost Relative to ANN
ANN
71.19
6.14
38.47
18.00
36.47
100.00%
Ours (Grain=2)
66.18
8.82
31.02
6.08
10.13
71.03%
Ours (Grainm3)
66.50
9.55
34.35
6.14
40.53
71,77%
Table 17: Relative energy consumption of LLaMA-3-8B under ANN and our spiking LLMs with mask, including both read and write operations.
Method
Proportion of Read/Write Cost
Proportion of MAC & AC Cost
ANN
68. 12%
31.88%
Ours (Grain=2)
84.77%
15.15%
Ours (Grain=3)
84.75%
15.15%
E USE OF LLMS
In this work, LLMs are employed solely for polishing or grammar checking text that is originally written by us.
26


Reviews:
[{"review_id": "GgJ9FO7WyP", "reviewer": "Reviewer_fm6A", "summary": "To address the issue of mismatched non-uniform activation value distribution when converting LLM to SNN, this paper introduced Distribution-Aware Multi-Granularity Phase Coding with multiple learnable bases and proposed the Alternating Optimization Neuron Training Algorithm to optimize the parameters of SNN neurons, thereby reducing the conversion time.", "questions": "The problem is as described above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The structure and writing of this paper are good and easy to understand. In terms of reducing conversion time, it improves efficiency and may be a promising method.", "weaknesses": "1. This paper claims that spiking large language models (LLMs) have significant advantages on neuromorphic hardware, but the final conclusion is that the energy consumption is reduced by 42% on GPUs. Moreover, the way energy consumption is presented is extremely unscientific and rigorous, which seriously undermines the technical contribution and persuasiveness of the paper. First, the paper cites Horowitz's 2014 proposal that \"1 MAC ≈ 4.6 pJ and 1 AC ≈ 0.9 pJ under 45 nm CMOS technology\" as the basis for energy consumption conversion, but there is no specific model of \"commodity GPUs\". The energy consumption designs of different GPUs vary greatly. The mainstream process of current \"commercial GPUs\" has entered 3~8 nm, and there is a technical generation gap of more than several decades between the two. The deviation between the calculation results and the actual energy consumption of the hardware is at least an order of magnitude. The methodology of estimating the actual energy consumption of contemporary GPUs using the energy consumption coefficient of 45 nm is completely invalid [1][2]. Secondly, the energy consumption of LLM does not only come from \"computational operations (MAC/AC)\". In contemporary GPUs, the energy consumption of memory access (i.e., HBM, SRAM read and write), data movement (data transmission in PCIe), and control unit scheduling often exceeds the computational energy consumption. Moreover, LLM inference is memory bound, and memory energy consumption accounts for the vast majority of the total energy consumption. The paper only estimates the total energy consumption based on the number of MAC/AC, completely ignoring key energy consumption items such as memory access and data movement, which has no correlation with the \"end-to-end energy consumption\" of the actual GPU operation.\n\n2. The core problem pointed out by the authors of this paper is not new: conversion error arising from distribution misalignment. This problem does not only exist in the conversion of LLM to SNN, but is a long-standing inherent problem [3], and it still exists in the conversion of CNN to SNN.\n\n3. The Distribution-Aware Multi-Granularity Phase Coding in this paper actually adjusts the parameters such as h(t) and d(t) in FS neurons to multi-granularity learnable parameters, while the parameters of the original FS neurons are also learnable and adjustable, and there is no essential difference [4].\n\n4. The ablation study in Table 5 shows that when the granularity increases from 1 to 2 or 3, the performance gain is very limited, which means that the core contribution of the paper is actually the learnable B, rather than \"multi-granularity\". Compared with a carefully optimized \"single-granularity distribution-aware phase coding\", what is the real advantage of multi-granularity? It can be seen from Tables 1, 2, and 6 that when the time step is large (T=10), the performance gain by increasing the Grain is negligible, while when the time step is slightly smaller (T=8), the change in the result by changing the grain is slightly larger. Does this mean that the time step T is also an influencing factor? Therefore, an ablation experiment of time step T and grain should be added.\n\n5. The meaning of the formula is ambiguous or even ambiguous: In formula (3), the author writes \"Typically, $h(t)$, $d(t)$, and $\\theta(t)$ are specified as $B^{-t}$ as follows:$v(t + 1) = v(t) - B^{-t}s(t),\\ O(t) = B^{-t}s(t),\\ s(t) = \\Theta\\left(v(t)-B^{-t}\\right).$\" But in formula (8), it is claimed that \"we denote $\\{h(t)\\}_{t=1}^T$, $\\{\\theta(t)\\}_{t=1}^T$, $\\{d(t)\\}_{t=1}^T$, $\\{B_i\\}_{i=1}^n$ as $h$, $\\theta$, $d$, and $B$\" So what is the final expression?\n\n6. There is a lack of comparison with existing works. Only SpikeLLM, SpikedAttention, and LAS are compared, and more extensive control experiments are lacking.\n\n[1] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-nvidia-us-2188504-web.pdf?utm_source=chatgpt.com\n\n[2] https://resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306?utm_source=chatgpt.com\n\n[3] Can Deep Neural Networks be Converted to Ultra  Low-Latency Spiking Neural Networks\n\n[4] Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yV1MovcwgT", "reviewer": "Reviewer_xCuD", "summary": "This paper proposes an Alternating Optimization Neuron Training algorithm for directly training Spiking Large Language Models (Spiking LLMs) based on multi-granularity phase coding. The core innovation lies in decoupling the spiking threshold θ(t) and reset mechanism h(t) from the traditional decoding factor d(t), treating them as independent, learnable parameters. By alternately optimizing these learnable parameters with the network weights, the method aims to minimize information loss during the ANN-to-SNN conversion, thereby achieving high accuracy while significantly reducing energy consumption. The paper claims state-of-the-art results in both performance and energy efficiency.", "questions": "1.In the alternating optimization process, how are the convergence speed and stability of the two steps—optimizing weights B with fixed thresholds o, and optimizing thresholds o with fixed weights B? Is one step dominant? Have you tried joint optimization instead of alternating optimization?\n\n2.How are the learnable bases B in \"multi-granularity phase coding\" initialized? What is their evolution trend during training?\n\n3.The paper assumes a fixed total number of timesteps T. In practical inference, can true \"event-driven\" or \"adaptive timestep\" computation be achieved?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed \"multi-granularity phase coding\" framework and the idea of decoupling θ(t) and h(t) from d(t) are innovative.\n\n2.The paper provides a convergence proof for the proposed alternating optimization algorithm.", "weaknesses": "1.The paper does not sufficiently demonstrate the necessity of the proposed decoupling mechanism. It lacks critical ablation studies to show how much performance would degrade if θ(t) and h(t) were not decouple.\n\n2.The paper mentions a 42.0% energy reduction but does not specify the baseline or the evaluation methodology. Is it an estimate based on spike count in a simulator? Or is it actual power consumption measured on specific hardware?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QqQep8taBU", "reviewer": "Reviewer_qqbp", "summary": "The paper propose a ANN to SNN conversion method that accounts for the distribution of activation values.", "questions": "1. How does the time step size affect the conclusions?\n\n2. Does it work for other models and data sets?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is shown to improve accuracy at a reasonable cost in the experiments.", "weaknesses": "The opening claim that activation distribution was not considered in previous work is clearly incorrect. The following are some examples that do just that - and not compared or even cited in the paper:\n\n. Li, C., Ma, L., & Furber, S. (2022).\nQuantization Framework for Fast Spiking Neural Networks (QFFS). Frontiers in Neuroscience.\nAvailable at: https://www.frontiersin.org/articles/10.3389/fnins.2022.918793/full\n\n2. Moser, B. A., & Lunglmayr, M. (2023).\nQuantization in Spiking Neural Networks. arXiv preprint arXiv:2305.08012.\nAvailable at: https://arxiv.org/abs/2305.08012\n\n3. Guo, Y., et al. (2023).\nRMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networks. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).\nAvailable at: https://openaccess.thecvf.com/content/ICCV2023/papers/Guo_RMP-Loss_Regularizing_Membrane_Potential_Distribution_for_Spiking_Neural_Networks_ICCV_2023_paper.pdf\n\nThe time steps of 8 is also relatively large compared to recent work. \n\nOverall, while I can see the approach seems intuitively on the right path, the evaluation needs to be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qQortB61rw", "reviewer": "Reviewer_wbNY", "summary": "This paper addresses the potential conversion error caused by activation distribution mismatch when converting pre-trained ANN LLMs to SNN LLMs. The authors propose a distribution-aware multi-granularity phase coding approach, which aims to achieve a reasonable discrete value allocation by minimizing conversion error relative to the activation distributions.", "questions": "See weakness above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Distribution-aware multi-granularity phase coding is a novel and well-motivated method (derived from quantization distortion in information theory), providing a new approach to solve the distribution mismatch problem in ANN-SNN conversion.\n\n2. Experimental results (on LLaMA-2 and LLaMA-3 models) show that this method achieves SOTA conversion performance (close to the ANN's PPL and accuracy), and the time cost for conversion (i.e., training the neurons) is extremely low (within minutes).", "weaknesses": "I like the distribution-aware conversion ideas and appreciate the authors' effort in fine-tuning SNNs on LLMs within minutes, but my main concern lies in the energy claims：\n\n 1. End-to-end energy saving claim: In both the abstract and Section 6.3, the authors declare that \"On commodity GPUs,\" their SNNs achieve an \"end-to-end\" reduction compared with their ANN counterparts, which is a very strong claim. However, upon checking Table 3, I found that the energy saving calculation relies only on the count of MACs and ACs. The energy for memory access, data movement, and control logic is not considered. SNNs may suffer more from these overheads due to their long spike train representations and multiple weight accesses across the time window. \n2.  Furthermore, on GPUs, QNNs benefit from regular memory access patterns that enable efficient data reuse opportunities, whereas SNNs suffer from sparse and irregular memory access patterns. Can the authors explain how their SNNs running on GPUs solve these problems?Thus, if the authors want to claim the \"end-to-end\" \"42%\" energy consumption reduction, I recommend that they either: Consider these hardware overheads and discuss how to implement SNNs efficiently on GPUs; Measure the actual power when running the SNN and ANN on their GPU and multiply it by the running time to show the actual energy consumption on GPUs; Run their models on neuromorphic chips like Loihi and compare the energy.\n3. Also, the spiking Softmax and RMSNorm (Appendix B) seem to still have complex multiplication or exponentiation operations. Is there any way to do a simulation of these functions and also apply it to the ANN-to-SNN conversion paradigm? If not, on GPUs, was the energy consumption of these exp and sqrt operations, which run at every timestep T, included in the total energy calculation in Table 3?\n\nMinor Points:\n\n1. In Figure 1a, the meanings of the x and y axes should be labeled on the figure to be more intuitive.\n\n2. In Figure 2, the dimensions of Q, K, and V should include the time domain (e.g., n*d*T).\n\n3. For the energy comparison, the paper should also include a comparison with other SpikingLLM papers, like those mentioned in Table 1.\n\n4. If the authors tried weight quantization, does the distribution of each layer change? What are the results for, e.g., 4-bit or 8-bit weights for Spiking LLMs, since the baseline SpikeLLM can reduce the weight to 4-bits?\n\n5. Potential complexity of multi-granularity tuning: Although multi-granularity phase coding improves representational capacity, learning and optimizing multiple bases may be more complex than the traditional method with a single base. Can author discuss more about the energy overhead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)