Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression
Authors: Yifan Huang
Wei Fang
Zecheng Hao
Zhengyu Ma
Yonghong Tian
Venue: ICLR 2026 Poster (2026)
Keywords: Spiking Neural Network
Training Memory Optimization
Gradient Checkpointing
Primary area: applications to neuroscience & cognitive science
TL;DR: A widely applicable automatic pipeline combining spatio-temporal gradient checkpointing and spike compression that achieves up to 8× memory savings for SNN training while preserving BPTT-level accuracy and speed.

Abstract: Deep spiking neural networks (SNNs) hold immense promise for low-power event-driven computing, but their direct training via backpropagation through time (BPTT) incurs prohibitive memory cost, which limits their scalability. Existing memory-saving approaches, such as online learning, BPTT-to-BP, and reversible networks, compromise accuracy, training speed, or applicability. In this work, we propose a novel and broadly applicable pipeline for memory-efficient SNN training that preserves BPTT's accuracy. Our pipeline integrates layer-wise gradient checkpointing with lossless spike compression to eliminate internal state storage and reduce the memory cost of per-layer input spikes. We also introduce a multi-stage checkpoint adjustment strategy that adaptively refines checkpoint placement based on profiling results to further optimize memory usage and improve training speed. Wrapped in an optimization pass, the pipeline automatically restructures the computation flow before training with minimal user effort. Extensive experiments on diverse architectures and tasks demonstrate up to $8\times$ memory efficiency gains with $\le 20\%$ speed reduction and no accuracy loss. Our method provides a practical solution for efficient and scalable SNN training. Code is available at https://github.com/AllenYolk/snn-gradient-checkpointing.

Published as a conference paper at ICLR 2026
TOWARDS LOSSLESS MEMORY-EFFICIENT TRAINING OF SPIKING NEURAL NETWORKS VIA GRADIENT CHECKPOINTING AND SPIKE COMPRESSION
Yifan Huang Wei Fang".,, Zecheng Hao Zhengyu Ma², Yonghong Tian 12,334, School of Computer Science, Peking University, Beijing, China
Beijing Key Laboratory of Brain-inspired Spiking Large Models, Beijing, China SPeng Cheng Laboratory, Shenzhen, China
"Scoool of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China
5 Sttate Key Laboratory for Multimedia Information Processing, Beijing, China
(yfhuang, fwei, ha haozecheng, yntian] @pku. edu, cn, waazzy@pcl.ac..o
ABSTRACT
Deep spiking neural networks (SNNs) hold immense promise for low-power event-driven computing, but their direct training via backpropagation through time (BPTT) incurs prohibitive memory cost, which limits their scalability. Existing memory-saving approaches, such as online learning, BPTT-to-Bi BP, and reversible networks, compromise accuracy, training speed, or applicability. In this work, we propose novel and broadly applicable pipeline for memory-efficient SNN training that preserves BPTT's accuracy. Our pipeline integrates layer-wise gradient check- pointing with lossless spike compression to eliminate internal state storage and reduce the memory cost of per-layer input spikes. We also introduce a multi-stage checkpoint adjustment strategy that adaptively refines checkpoint placement based on profiling results further optimize memory usage and improve training speed. Wrapped in an optimization pass, the pipeline automatically restructures the com- putation flow before training with minimal user effort. Extensive experiments on diverse architectures and tasks demonstrate up 8x memory efficiency gains with I 20% speed reduction and no accuracy loss. Our method provides a practical solution for efficient and scalable SNN training. Code is available at https: ht//a//tt....ccaaaaaaa///////aaa
INTRODUCTION
Inspired by the dynamics of biological neurons (Gerstner al. 2014), spiking neural networks (SNNs) have emerged as the third generation of neural network models (Maass] 1997). SNNs transmit information via discrete spikes rather than continuous activations in conventional artificial neural networks (ANNs). Their sparse and event-driven nature makes them ideal for deployment on neuromorphic chips (Merolla et al., 2014 Akopyan al., 2015] Davies et al. 2018 Pei et al.. |
2019) for inference, offering significant potential for low-power edge computing Yao et al. 2024). To train a deep SNN end-to-end, the temporal dimension discretized into T time steps so that the SNN can be considered
binary-activated recurrent neural network (RNN) (Fang et al. 2023a Eshraghian et al.) 2023). Then, backpropagation through time (BPTT) [Werbos] 1990) is adopted compute parameter updates, with surrogate gradient (SG) tackling the non-differentiable spike emission process (Neftei et al. (2019) Wu et al. 2018 Shrestha & Orchard 2018). With the BPTT- based framework, low-latency deep SNNs can be directly trained using powerful graphics processing
units (GPUs) [Chetlur et al. 2014) and yield competitive performance (Yao et al. 2025 Wang et al.
2024
Lv et al.
2024a| Chen et 2025).
Despite its high accuracy and broad applicability, BPTT imposes intensive memory overhead (Meng al. 2023). For an L-layer SNN unfolded over T time steps, BPTT requires (LT) memory to store intermediate states, compared to (LL) for structurally similar ANN. Consequently. SNN direct training is more likely to exceed the memory capacity of computational devices. The scaling of SNNs to deeper architectures and more time steps is thus severely hindered.
"oorrsponning authors.

---

Published as a conference paper at ICLR 2026
Nighest
Case Forword
CosN Beoksaard
Forword
Comp
Farmoord
Conts Backoaard
portition
spedial
Bil
Neuren Foruard
Neuron (eedwaard
Neuron Forward
Haturon Forward
Neurns Bacloaard
Naurzn
Ggst
Gars
highwst
x
Cone Porourd
Coow Bucknward
Porword
Comp
Poroard
Cont Bockwaadd
Bii
Neuron
Neuron
Neuron
Neuron Porward
(Sacoward
Forward
Dis
Forward
Badkward
temporal partition
Cy
segment
recmputatiio
C2
Updara
Uptttt
Torward pass
pass
pass
pass
Figure 1: Comparison of (a) BPTT and (b) gradient checkpointing with spike compression. We use grey boxes with dashed borders to denote gradient checkpointing segments. (c) Spatial segment partitioning. (d) Temporal segment partitioning.
Several approaches have been explored to reduce the memory demands of BPTT-based SNN training, including online learning (Bellee et al., 2020 Xiao et al. 2022, Meng et al. 2023 Yin al. 2023
Jiang al. 2024), BPTT-to-BI (Xiao al. [202]] Wu et al. 2023 Kheradpisheh
al. 2022 Yu et al., 2024), and reversible networks (Zhang & Zhang 2024] Hu et al.) 2024). However, these methods compromise training speed, accuracy, or generality across SNN models (see Section 2.2
and T5 for details). Also, their implementations require manual architectural modifications or training code rewrites, which are Prror-prone and cumbersome. These limitations highlight the need for a broadly applicable and user- seer-friendly solution that improves the memory efficiency of SNN direct training while preserving training speed and performance.
In this work, we propose an automatic pipeline that combines gradient checkpointing (Chen et al. 2016) and spike compression address the challenge (Figure Our analysis identities internal states and per-layer input spikes as the dominant memory consumers in SNN training. To this end, we employ layer-wise gradient checkpointing to eliminate internal state storage, and losslessly compress the input spikes before saving them to reduce their memory footprint. To further optimize peak memory usage, we insert additional checkpoints spatio-temporally into high-cost layers. Check- pointed segments with no benefit peak memory then greedily reverted to standard BPTT segments to accelerate training. The entire process is encapsulated an optimization pass that automatically reconfigures the computation flow before training, requiring minimal user intervention. The proposed method obtains up to 8x memory efficiency gains with an affordable training speed drop and preserved accuracy extensive experiments. Our main contributions
(1) Memory cost analysis. We analyze the memory cost of SNN direct training and identify input spikes and internal states primary memory consumers.
(2) An automatic pipeline. We propose a broadly applicable pipeline that integrates gradient checkpointing with spike compression for memory-efficient SNN training.
(3) Efficiency and Accuracy. We obtain substantial memory savings on diverse SNN models and task settings with acceptable speed trade-off's and maintained accuracy.
2 RELATED WORK
2.1 BPTT-BASED SNN DIRECT TRAINING
If simulated discrete time steps, SNNs can be trained end-to-end as binary-activated RNNs through BPTT (Werbos] 1990), with SG addressing the non-difie differreniaability the spike firing process (Neftci
et al. 2019 Wu al. 2018 Shrestha & Orchard 2018). Compared to ANN-to-SNN conversion
2

---

Published as a conference paper at ICLR 2026
(Cao et al. 2015 Bu et al.
2022 Hu al. 2023 Hao et al. 2023bla), a). this approach enables low
inference latency (Wu et al.
2019) and broader task applicability. thus attracting increasing attention. Recent advancements have improved the performance of SNN direct training by adapting ANN architectures like ResNet (He et al., 2016) and Transformer (Vaswani et al.
2017 Dosovitskiy et al.
(02T) to spiking ResNets (Fang et al. 2221aa] Hu et al. (025) and spiking Transformers (Zhou et al.
2023 Yao et al.
222 Ib 2023; Zhou et al. (024). Other works enhance neuron models (Fang ct al,
Yao al.
2022; Fang al. 2023b Huang et al. 2024a] Li et al. 2024b Huang et al. 2024b). For
instance, the parallel spiking neuron (PSN) family (Fang et al. 2023b) models neuronal dynamics as
a linear projection of the input over time, enabling temporal parallelization and efficient capturing
of long-term dependencies. Despite the advances in performance, the memory overhead of BPTT
remains a key bottleneck.
2.2 MEMORY-EFFICIENT SNN DIRECT TRAINING
BPTT's OLT)) memory complexity motivates methods to reduce SNN training memory usage.
Online learning (Bellec et al., 2020 Xiao et al.) 2022 Meng et al. 2023 Yin et al. 2023 [Bhhsttigg] et al. 2023] Tiang et al.] (024) truncates temporal gradients and stores only the intermediate results at
the current step. However, the gradient mismatch results in severe performance drop temporal
tasks. Its step-wise running mode undermines its compatibility with widely adopted temporal
parallelization techniques like PSN (Fang al. 2023b) BPTT-to-BP approximation (Xiao et al.)
2021: Wu et al. 2023; Kheradpisheh et al. [022] Yu et al.] 2024) trains an SNN by backpropagating through static proxy based on firing rates, effectively removing the temporal gradient dimension. Despite its memory and time efficiency, BPTT-to-BP can hardly handle sequential data due to the neglect of temporal information, thus limiting its applicability. Last but not least, reversible networks (Gomez et al. 2017 Zhang & Zhhang 2024 Hu et al. (024) reconstruct intermediate features reversely during backward pass rather than storing them. preserves BPTT-level accuracy. but imposes strict architectural constraints and significantly slows training. In conclusion, existing methods trade off accuracy, speed, or applicability; they also require manual modifications on model architectures and training codes. In contrast, pipeline reduces memory usage with an affordable extra time cost, maintains accuracy and broad compatibility, and demands minimal user effort.
3 PRELIMINARIES
3.1 SPIKING NEURAL NETWORKS
SNNs can be regarded as ANNs augmented with bio- inspired spiking neuronal dynamics (Li et al., 2024a). To train an SNN directly, its dynamics are simulated on T dis- crete time steps, and the spike signals represented binary activations. For example, the discrete-time dynam- ies of a L-layer SNN composed of leaky integrate-and-fire (LIF) neurons (Gerstner et al. 2014) can be described as Equation (T). Here, / E .I,..., D} is the layer index, and / E (1...., , is the time step index. X is the input current, H and V are the membrane potentials before and after spike emission, and S is the
x'[[] of (1): w"),
H'[ = Xv 1] + x
(1)
s'((] = @ Vnn,,
V'[4 = H H1[1[( - ()
output spike (a.k.a. activation). X can be computed from the previous layer's output S [(] via a
linear transformation g with weight W (bias is omitted). X E (0, 1) is the decay factor, Vtn > 0 the firing threshold, and (xx) is the Heaviside step function (yields if x 2 and otherwise). The elements of S (1 0) are either 0 (no spike) or 1 (spike), while the network input S is not necessarily binary (Rathi& & Roy| 2023). Notice that the second to fourth lines of Equation (I) are eles and secondary neuronal parameters like the reset and resting potentials are omitted for simplicity. We use LIF as the default neuron model throughout this work.
3.2 GRADIENT CHECKPOINTING
Gradient checkpointing (GC) (Chen et al. 2016) was originally proposed for ANN training to trade computation for memory. Standard backpropagation stores all intermediate results for gradient computation, as Figure [a) shows. By contrast, GC stores merely a subset of activations (a.k.a checkpoints) and discards the others; the network is thus divided into several GC segments, each
3

---

Published as a conference paper at ICLR 2026
Aboolute Memoryy (MI5
HelaEve Compoditoo
Memoory Usage (MD)
PURt SISLLP
94521
BPTT
20000
CC-p (Oh
2000
97.2%
3230
76.356
76.8%
2e30
1000
1224
2121
3030
Roodee-S4
Spiidonmee
BosNee-54
SEW RaNa 34
Spiifieter
Boa 34
Layar Index
weights
pradilent
The memory cost evolu- tion when training a Spiking VGG on
Figure 3:
Figure 2: The memory cost breakdown of ANNs and SNNs when peak memory consumption is reached during training on ImageNet (see Appendix A).
Dashed lines indi- cate peak memory consumptions.
saving only its input. During backward pass segment, the forward computation is rerun from the segment's checkpointed input to restore the dropped activations needed for calculating gradients, illustrated in bb). Since forward pass is far less costly than backward pass, GC's extra time cost is affordable. GC has been successfully applied to temporal models like recurrent neural networks (Gruslys al.) 2016) and neural ordinary differential equations (Zhuang al. 2020 to
reduce training memory cost.
Previous studies have explored applying GC along the temporal dimension of SNNs (Singh et al. 2022 Bencheikh et al. (2024), achieving notable memory savings on shallow networks with large TT 2 100). However, these approaches do not consider the spatial dimension, and have not been evaluated on advanced larger-scale SNNs that typically adopt short time horizons (T < 16). In addition, they lack an automated, user-friendly GC workflow, which limits ease of use in practice.
METHODS
4.1 MEMORY COST ANALYSIS OF BPTT-BASED SNN DIRECT TRAINING
In BPTT-based SNN direct training. memory usage primarily stems from: (1) model parameters, (2) gradients, (3) optimizer states, (4) intermediate features, including each layer's input and internal states, that stored during forward pass for backward gradient computation, and (5) temporary runtime variables dynamically allocated and immediately freed. An upper bound for the peak memory can be formulated as:
Mppk BPTT S
Σ (Mw + Mar + +M + +M + +Ma) + max MRs
where Mwi, MMg, My Aig--,,Mar, and Mg are the memory consumptions of the weights, gradients, optimizer states, inputs, internal states, and runtime variables layer respectively.
key feature SNN direct training is that intermediate features (inputs and internal states) dominate memory usage. As shown in Figure 2 for ResNet-34 (He al.] 2016) and ViT (Dosovitskiy et al.) 2021) trained on ImageNet (Deng et al. 2009), intermediate features occupy about 77% of the memory peak usage. In contrast, for their SNN counterparts with T = the ratios rise over 96%. This is because SNNs' T time steps scale intermediate feature sizes by O while the sizes of weights, gradients and optimizer states stay unchanged. Therefore, memory optimization for SNN direct training should prioritize reducing internal states and input spike storage at each layer.
4.2 LAYER-WISH GRADIENT CHECKPOINTING
In standard BPTT, all internal states must be stored, resulting in memory cost of up to Mgy To reduce this cost, we apply GC (Chen et al. 2016) to each layer / {1...., , L). During the forward pass layer I, only the input ST and weight W are stored. In the backward pass, internal states Q! are reconstructed through an extra local forward pass given S and W With s!-1 Wi and Q¹ can propagate the gradients back through layer 1, as Figure shows.
4

---

Published as a conference paper at ICLR 2026
With GC, Ω is allocated and freed during layer I's backward pass. Thus, at most one layer's internal states are stored in memory at any given time. The peak memory's 's upper bound then becomes:
M&ck S (M + Ma + MM Ms-- 1 + May) + max (Mg) .MR).....
Since internal states in SNNS consume far more memory than in ANNs (Figure 2. GC's effectiveness will be more pronounced in SNNs compared to ANNs.
4.3 LOSSLESS INPUT SPIKE COMPRESSION
Input spikes ( must be stored even if GC is applied. For most SNN programming frameworks (Fang et al., 2023a, Eshraghian et al. 2023), spikes are represented 32-bit floats (or 16-bit with automatic mixed precision) for compatibility with arithmetic operations. However, 32-bit storage redundant for binary values. Therefore, instead of storing S' as floats during forward pass, we store its compressed form $1-1, as Figure shows. Š is decompressed S when needed in backward pass (Algorithm
D
The peak memory's 's upper bound then becomes:
peeale GC+Comp
Σ (Mw + Mo + MM + MMgu++ + Mg: 1) + max (May + MR))
(4)
The spike compressor must be lossless to ensure computational equivalence with standard BPTT. For instance, bit representation uses I bit per binary value, achieving up 32x compression over 32-bit floats. Alternatives include sparse representation that records the indices of non zen-zero elements, and lossless bit stream compressors like Zstandard (Collet & Kucherawy 2018 and asymmetric numeral systems (ANS) (Duda 2013). While bit representation cannot benefit from spike sparsity, is faster and more memory-saving than the alternatives in most cases (see Appendix [M). Hence, we choose it by default. To further accelerate compression and decompression, we handcraft Triton kernels (Tillet et al. 2019). Notice that compression skipped for non-binary inputs (e.g., ).).
4.4 ADJUSTING GRADIENT CHECKPOINTING STRUCTURE
Figure depicts the memory evolution during a training iteration of a Spiking VGG on CIFARIO-DVS (see Appendix B for explanations). For standard BPTT (blue), the peak occurs in deep layers during backward pass. Layer-wise GC (green) reduces deep-layer memory, shifting the peak to shallower layers, and spike compression (orange) further lowers deep-layer cost. After these optimizations, the global peak memory MPeak achieved the critical layer far exceeds the local peaks elsewhere. Notice that model trainability adjust the GC structure to further enhance global efficiency by allowing slightly higher memory usage in non-critical layers. We propose three strategies accordingly and summarize them in Algorithm 2
specific devices depends only this global peak. This motivates to
Spatial Segment Partitioning To reduce MPeak, we first identify the GC segment 1" with the largest peak memory and then insert spatial checkpoint within In other words, we split /* along the layer dimension into two spatial subsegments ! and 12 as Figure [c) shows. The spatial max(MM3' May' J. a
partition point is defined by the user (see Appendix [}. Since Ma reduction of max; Mg is guaranteed. However, MPeea may not drop due to the added checkpoint. This process repeats until Mpeak cannot further decrease.
Temporal Segment Partitioning Temporal partitioning similarly finds the critical segment 1* and splits it along the time axis into sequential temporal subsegments, as shown in Figure dd. Each temporal subsegment checkpoints both its inputs and initial hidden states to enable recomputation during backward pass. Users should set the temporal partitioning factor k and define the state transition function (see Appendix [}. The procedure repeats until MPeak cannot be further reduced. Temporal partitioning is applied conservatively after spatial partitioning as complementary strategy, since splitting segments along time disables temporal parallelism and limits temporal kernel fusion, resulting in restricted applicability and slower training.
Greedy Segment Restoration For GC segments whose local memory cost is well below MPeak can safely revert them to standard BPTT blocks (i.e., storing all intermediate features) without
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 One iteration of SNN training with layer-wise GC and spike compression.
Algorithm 2 GC structure adjustment.
Input: A list of GC segments V [seg']
Input: parameters Whis network input
Output: the adjusted GC segment list.
S compressor C-);; other hyperparameters. Output: trained parameters (W)/
1: // spatial partitioning
2: while True do
1: // forward pass
3: Find 1"
arg max, maxi (.MP
2: for 1.2..... do
Spatially split: seg" + { i
3: S' +- layer (S--1, autograd = False);
5: if global MPpeak doesn't 't decrease then Revert the split; break;
4: if S' is binary then
5: Compress: S C(S'-1);
7: end if
6: Save S'-1, and free Si-1;
8: end while
7: else
9: // temporal partitioning
8: Save S'-1;
10: while True do
11: Find 1" = arg max, (MMP
9: end if
end for
12: Temporally split: seg" + {seg") Mi
11: Compute the loss L and the gradient ac
13: if global M doesn't t decrease then
8S
// backward pass
14: Revert the split; break;
13: for =
do S' is compressed then
15: end if
14:
16: end while
Decompress: S + CSI(
17: // greedy restoration
16:
end if
18: sort descendingly by forward time cost;
17:
layer' Is = True);
19: for seg' in V do
18:
ac Compute #W7, Ø by BPTT;
20: Restore seg' to BPTT segment;
21: if global Mypeak increases then
19:
Free the saved tensors of layer I;
22: Re-enable GC for seg';
20:
end for
21: Update the parameters (W"). Ti-
23: end if
24:
end for
increasing MPeak, Since GC segments require an extra forward pass for recomputation, restoring them accelerates training. Specifically, we first profile the forward time cost each GC segment, and then greedily restore the segments with the largest time cost. The change is kept only if MPeak does not increase. This process terminates after all segments are considered.
4.5 AUTOMATIC PIPELINE
net memnory_optinization net, (ConvidBMWeuron, Conv2dBWeuron, @KACore, SSACore), dunny_ sunny_inputstte capttstorch.rand 3, 224, 224), compress_xeTee level=4, verbose-True,
To minimize user intervention, we wrap all the above strategies into automatic pipeline. Users can set the level parameter to specify the applied strategy set. At level O₁, only layer-wise GC and spike compression are enabled; O₂ addi- tionally applies spatial segment partitioning: O3 further incorporates temporal partitioning: O4 additionally activates greedy segment restora-
cemporal_split.ce spllit_factor2,,
Figure The pipeline's user interface.
tion. Default settings cover most cases, while advanced users customize spatio-temporal partition schemes. This design balances simplicity and extensibility.
4.6 MEMORY-EFFICIEN LIF KERNEL
Beyond the optimization pipeline, kernel-level improvements can bring further efficiency gains. We therefore design Triton kernel (Tillet al. 2019) for the widely adopted LIF neuron. The BPTT formulation of LIF can be derived from Equation as:
ac
ac
ac
H [t] 0v
0gg (H'
Vaa) + Øviμ (1 s()).
[
(5)
OC
8v

---

Published as a conference paper at ICLR 2026
Table I: Comparison of training speed and memory cost. The throughput and memory cost ratios
relative to "SJLIF, BPTT" are shown in parentheses.
Batch Size
Throughput (sample s)
Peak Alloc.
Task
Network
LIF impl.
Method
t
Mem. (MB) +
SJLIF
BPTT
4872.23
1317.23
Sequential CIFAR-10
32
128
SCNN
PTLIF
BPTT
1054.68
1264.97
MELIF
O₂
5138.76 (1.05x)
474.98 (0.36x)
SILIF
BPTT
114.52
8984.02
DVS128
16
7B-Net
PTLIF
BPTT
36.52
8067.41
Gesture
MELIF
O4
120.04 (1.05x)
4213.86 (0.47
SILIF
BPTT
290.26
6131.07
CIFAR10- DVS
Spiking VGG
10
32
PTLIF
BPTT
150.69
5889.44
MELIF
O4
270.79 (0.93x)
2349.39 (0.38x)
SJLIF
BPTT
309.04
8821.28
SEW ResNet-34
PTLIF BPTT
202.83
7140.09
MELIF O₂
281.39 (0.91x)
2004.14 (0.23x)
SJLIF BPTT
116.70
34264.76
ImageNet
32
Spikformer (8-512)
PTLIF BPTT
71.03
28779.13
MELIF O1
93.58 (0.80x)
7640.68 (0.22x)
SJLIF
BPTT
86.15
44571.33
QKFormer (10-512)
PTLIF
BPTT
55.65
37375.9
MELIF
O₂
76.51 (0.89x)
5219.93 (0.12x)
where L is the loss and O is the surrogate gradient function. Accordingly, BPTT LIF layer requires storing only {H' [1] and (S' [)) during forward pass. We further avoid storing
(S' [t] by reconstructing during the LIF layer's backward pass through S (HH Vaa) In this way, the floating-por point spikes can be dropped once their compression at the subsequent layer
is done. We name the kernel as memory memory-efficient LIF (MELIF) and use it by default.
EXPERIMENTS
In this section, we evaluate the proposed method's 's memory efficiency, well as training speed, compatibility, and accuracy. We also conduct case studies to highlight the importance our method.
5.1 MEMORY COST AND TRAINING SPEED
Table 2:
Ablation study of LIF implemen- tation and optimization levels on CIFARIO- DVS.
We assess the memory and time cost of our method on Sequential CIFAR- 10 (Fang et al. (2221b) DVS128 Gesture (Amir et al. 2017). CIFARIO-DVS (Li et al.
2017), and ImageNet (Deng et al.
2009). For Ima-
LIF impl.
Opt. Level
Throughput (sample /
Peak Alloc. Mem. (MB)
geNet, try three architectures: SEW ResNet-34 (Fang et al. 22221a), Spikformer (Zhou et al.] (023).
SILIF
290.26
6131.07
and FKFormer (Zhou al. 2024). See Appendix for more details. As Table shows, our memory op- timization pipeline at O combined with the Triton- based LIF kernel (MELIF) reduces the peak memory consumption to 0.12x 0.47x of SNNs trained with standard BPTT using Spikinglelly's CuPy-based LIF (SJLIF). This great reduction in memory footprint is achieved with no or only a slight training slowdown << 20%; see Appendix K for more detailed runtime decomposition). Table shows that the proposed Triton kernel significantly more memory- and time-efficient than SJLIF and the LIF in pure PyTorch (PTLIF). Moreover, layer-wise GC (O1) and spatio-temporal GC segment partitioning (Oa) further reduce memory, while greedy restoration (O4) mitigates the recomputation overhead of GC. A fine-gan grained ablation study three GC adjustment strategies is provided in Appendix [.
PTLIF
150.69
5889.44
331.30
4865.06
O
246.81
2887.75
MELIF
Os
247.83
2349.39
O
270.79
2349.39
7

---

Published as a conference paper at ICLR 2026
Compressed
Not Compressed
Memory Usage (MB)
Op D
Posk: 2349.39
Perk: 2530.60 66 MB
M
1000
BP
FP
Layer Index
Layer Index
Figure 5: Spiking VGG memory evolution on CIFARIO-DVS under different optimization levels.
Table Compatibility with temporally parallel SNNs.
Table 4: Compatibility with AMP and LOMO. Condition: ImageNet, QKFommer, MELIR O4.
Peak Alloc. Mem. (MB)
Task
Network
Neuron
Method
Peak Alloc,
Sequential CIFAR-10
Sliding PSN
BPTT
1302.69
AMP?
LOMO?
Mem. (MB) +
SCNN
O₂
599.34 (0.46x)
X
5219.93
SEW
BPTT
7602.64
/
5190.60
ImageNet
ResNet-34
PSN
X
3158.02
O4
2544.28 (0.33x)
/
3142.86
Finally, Figure demonstrates that spike compression brings memory saving by providing more free space for GC structure adjustment (see Appendix for detailed discussion).
5.2 COMPATIBILITY WITH OTHER METHODS
Beyond LIF neurons, method is compatible with other spiking neuron models. Table 3 shows that our approach effectively reduces memory usage for SNNs built with PSNs and Sliding PSNs (Fang et 2023b). Note that temporal parallelism is not compatible with BPTT-to-Bi or online learning. Moreover, Aeble/ddemontrrall
that our method can be seamlessly combined with prevalent memory-saving techniques, such as automatic mixed precision (AMP) (Micikevicius et al. 2018) and low-memory optimizer (LOMO) (Ly et al. 2024b) (see Appendix for introductions).
5.3 MATHEMATICAL EQUIVALENCE WITH CONVENTIONAL BPTT
To verify that pipeline produces unbiased gradients with respect to standard BPTT, we compare Sequential CIFAR-10 accuracies in Figure 6) The MELIF curves with and with- out O4 optimization (green and orange) exactly overlap, showing that GC and spike compres- sion do not introduce gradient bias. Their minor difference from the baseline (SJLIF, blue) stems from the different numerical behavior of Triton and CuPy. This gap is negligible, the orange curve lies almost entirely within the baseline's error band. Additional results and discussion on numerical discrepancies are provided in Appendices F and G Overall, our pipeline preserves BPTT-level accuracy, which is its main advantage over other efficient training approaches.
Acc. (%)
A
10
Ad MM
40
STLIE
SttiL
10
200
300
300
200
Epochs
Epoche
Figure 6: Sequential CIFAR-I0 accuracies. SJLIF shows mean = std over three runs, while the other two curves are single runs with a fixed seed.
5.4 COMPARISON WITH OTHER EFFICINNT TRAINING METHODS
Table compares throughput, memory usage, gradient fidelity, and applicability constraints of representative efficient training methods. All methods use the same Spiking VGG model, except reversible networks, whose architectures are adjusted to match the VGG in parameter count (9.2 M)
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of SNN efficient training methods. Throughput and memory are tested on CIFARIO-DVS. 'Grad. Bias' indicates additional gradient approximation beyond surrogate gradients.
Throughput t
Peak Alloc.
Grad.
Category
Method
+
Constraints
(sample s)
Mem. (MB)
Bias
Vanilla
BPTT
290.26
6131.07
X
x
SLTT
297.45
736.63
Online
OTTT
216.78
969.21
/
step-wise only
Learning
NDOT
168.48
1467.90
Tandem SNN
551.96
1706.68
no temporal dependency
BPTT-to-BP
Rate-based
497.07
1540.65
Reversible
RevSResNet
157.46
3198.78
reversible models only
X
Network
T-RevSNN
191.36
1089.43
Ours
O4
270.79
2349.39
X
layer-wise only
ORCFonner, ImtageNet
(b)
DHI-SFNN,
anad intraal
(c) SpiacVideoFomes, Kinetics-20
Pook Men. (GB)
S.I.IF MELLI O2
Men.. (MB)
10000
Baseline
aned
Pokk Mon. por CPU (CBS)
80
7200
500 31
layer
NVIDIA
20-
61.65
$8.29 samplees
5000
4617.51
layer
RTX 4000's eupucily
sxmples//
2000
hyyer
11.17
20
60
250
800
750
1000
SILU MELIP O Figure 7: Case studies. The proposed pipeline enables (a) larger batch size, (b) finer temporal resolution, and (c) training large-scale SNNs on more accessible devices. The heatmap in (b) shows which intermediate features are saved during forward pass after O₂ optimization when T = 1000.
Batch
Time Stepe (TT)
and feature-map resolution. Online learning methods like SLTT (eeg et al. 2023). OTTT (iao al., 2022) and NDOT (Jiang al., 2024) achieve the lowest memory cost but require step-wise execution, prohibiting techniques like temporal parallelism (Fang et al. 2023b) that are common in modern SNNs. BPTT-to-BP, such as Tandem SNN (Wu et al.] 2023) and Rate-based BP (Yu et al. 2024), shows higher throughput but introduces substantial gradient bias, making it unsuitable for tasks with rich temporal dependencies. Reversible networks like RevSResNet (Zhang Zhang 2024) and T-RevSNN (Hu et al. 2024) reduce memory cost but significantly slow down training and impose strict architectural constraints. In contrast, our method balances speed and memory while maintaining mathematical equivalence to BPTT and supporting generic layer-wise wiser-wise SNNs.
5.5
CASE STUDIES
QKFormer ImageNet Take QKFormer trained ImageNet (T 4) example. With
pipeline, the batch size can be increased by nearly 8x without consuming more memory. Enlarging the batch size from 64 yields about 1.43x training speedup, shown in Figure a
DH-SENN SHD We evaluate our method Spiking Heidelberg Digits (SHD) (Cramer et al.) 2022) using DH-SFNN, fully connected SNN (700 - 1024 - 1024 - 512 - 20) with dendritic heterogeneity LIF (DH-LIF) neurons (Zheng et al. 2024). Each DH-LIF contains four dendritic branches and a soma, resulting in five internal states per neuron. Batch size is set to 128. Existing efficient training approaches can hardly work here: online learning and BPTT-to-BP struggle with SHD's rich temporal dynamics, while reversible network is infeasible due to architectural constraints. In contrast, as Figure 7b) shows, our method enables 4x increase in T with negligible extra memory cost, allowing finer temporal resolution and potentially better sequence modeling quality.
?
Spike VidkeeVdeoFFeeee Former on Kinetics-400 We train Spike VideoFoiderrr (Zou et al. 2025) (55.9 M parameters) on Kinetics-400 (Kay et al. 2017) with T = 32 frames and 224 x 224 mppt resolution. Training with batch size of 4 per GPU requires 54.43 GB of memory per device, restricting experiments to high-end hardware. Indeed, the original work uses eight A6000 GPUs, which is not affordable for many researchers. With our method, the peak memory per GPU is reduced to 11.17 GB,

---

Published as a conference paper at ICLR 2026
enabling its training widely accessible GPUs (e.g., 4090, 24 GB), as Figure 7 shows. This demonstrates that our approach can lower hardware barriers for cutting-edge SNN research.
CONCLUSION
In this work, presented automatic memory optimization pipeline for SNN direct training. The pipeline integrates layer-wise GC with lossless spike compression to reduce the memory footprint of intermediate features. We then adaptively adjust GC structure by spatio-tempora segment partitioning and greedy restoration to further reduce memory demand and GC"s recomputation overhead. Experi- ments show that our pipeline achieves high memory efficiency while maintaining acceptable training speed, BPTT-level accuracy, and broad compatibility. This work provides practical approach for efficiently training large-scale SNNs. Limitations and future directions are discussed Appendix N
ACKNOWLEDDGMENTS
This work was supported by the National Natural Science Foundation of China (62425101, 62332002), and Beijing Key Laboratory of Brain-inspired Spiking Large Models.
REFERENCES
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, Brian Taba, Michael Beakes, Bernard Brezzo, Jente B. Kuang, Rajit Manohar, William Risk, Bryan Jackson, and Dharmendra Modha. Truenorth: Design and tool flow of 65 mw I million neuron programmable neurosynaptic chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 34(10):
1537-1557, 2015.
Aron Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7243-7252, 2017.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 1(():3622, 2020.
Wadjih Bencheikh, Jan Finkbeiner, and Emre Neftci. Optimal gradient checkpointing for sparse and recurrent architectures using off-chip memory. arXiv preprint arXiv:24/2. 2024.
Thomas Bohnstingl. Stanislaw Wožniak, Angeliki Pantazi, and Evangelos Eleftheriou. Online spatio- temporal learning in deep neural networks. IEEE Trunsactions on Neural Networks and Learning Systems, 34(11) 8908, 2023.
Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ANN- SNN conversion for high-aceuracy and ultra-lowllttency spiking neural networks. In The Tenth International Conference on Learning Representations, 2022.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 1 113(1):54-66, 2015.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprimi arXiv: 160.0067446 2016.
Xinyi Chen, Jibin Wu, Chenxiang Ma, Yinsong Yan, Yujie Wu, and Kay Chen Tan. Pmsn: A parallel multi-compartment spiking neuron for multi-scale temporal processing. arXiv preprint arXiv:2408. 14917, 2024.
Zehao Chen, Zhan Lu, De Ma, Huajin Tang, Xudong Jiang, Qian Zheng, and Gang Pan. Evhdr- gs: Event- geeide hdr video reconstruction with 3d gaussian splatting. Proceedings of the AAAI Conference Artificial Intelligence, 39(3):2367-2375 2025.
10

---

Published as a conference paper at ICLR 2026
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint 1r1v:01400.0759, 2014.
Yann Collet and Mark Kucherawy. Zstandard compression and the application/zstd media type. RFC 8478, Internet Engineering Task Force, 2018.
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transuctions on Neural Networks and Leurning Systems, 33(7):2744-2755) 2022.
Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, S. Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonscok Yang. and Hong Wang. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In The Ninth International Conference Learning Representations, 2021.
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, volume 35, pp. 34377-34390, 2022.
Jarek Duda. Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding. arXiv preprint arXiv: 2013.
Jason K. Eshraghian, Max Ward, Emre O. Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D. Lu. Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 11 1110))1016-108 2023.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems, volume 34, pp. 21056-21066 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2661-2671, 2021b.
Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for baike-based intelligence. Science Advances, (440)ceadi1480 2023a.
Wei Fang, Zhaofei Yu, Zhaokun Zhou, Ding Chen, Yanqi Chen, Zhengyu Ma, Timothée Masquelier, and Yonghong Tian. Parallel spiking neurons with high efficiency and ability to learn long-term dependencies. In Advances in Neural Information Processing Systems, volume 36, pp. 53674- 53687, 2023b.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press, 2014.
Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual net- work: Backpropagation without storing activations. In Advances in Neural Information Processing Systems, volume 30, pp. 2214-2224, 2017.
11

---

Published as a conference paper at ICLR 2026
Audrunas Gruslys, Remi Munos, Ivo Danihelka, Mare Lanctot, and Alex Graves. Memory-efficient backpropagation through time. In Advances in Neural Information Processing Systems, volume 29, pp. 4125 4133,,43330000 2016.
Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion eπror through residual membrane potential. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1):11-22) 2023a.
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between ANNs and SNNs by calibrating offset spikes. In The Eleventh International Conference Learning Representations, 2023b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Jiakui Hu, Man Yao, Xuerui Qiu, Yuhong Chou, Yuxuan Cai, Ning Qiao, Yonghong Tian, Bo Xu, and Guoqi Li. High-performance temporal reversible spiking neural networks with O(I) training memory and (1) inference cost. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 19516-19530, 2024.
Yangfan Hu, Qian Zheng, Xudong Jiang, and Gang Pan. Fast-snn: Fast spiking neural network by converting quantized ann. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (12):14546-14) 2023.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 36(2): 2353 2367, 2025.
Yifan Huang, Wei Fang, Zhengyu Ma, Guoqi Li, and Yonghong Tian. Flexible and scalable deep den- dritic spiking neural networks with multiple nonlinear branching. arXiv preprint arXiv:2412.06:555 2024a.
Yulong Huang, Xiaopeng Lin, Hongwei Ren, Haotian Fu, Yue Zhou, Zunchang Liu, Biao Pan, and Bojun Cheng. CLIF: Complementary leaky integrate- and--ired neuron for spiking neural networks. Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 19949-19972, 2024b.
Haiyan Jiang. Giulia De Masi, Huan Xiong, and Bin Gu. NDOT: Neuronal dynamics- online training for spiking neural networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 21 21806-21888 1833, 2024.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint /005.0/770.66999 06950, 2017.
Saced Reza Kheradpisheh, Maryam Mirsadeghi, and Timothée Masquelier. Spiking neural networks trained via proxy. IEEE Access, (4270769-70777 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The Third International Conference Learning Representations, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
Guoqi Li, Lei Deng, Huajin Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass. Brain-inspired computing: A systematic survey and future trends. Proceedings of the IEEE, 112 (6):544-584, 2024a.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dve: An event-stream dataset for object classification. Frontiers in Neuroscience, 11:309, 2017.
12

---

Published as a conference paper at ICLR 2026
Yang Li, Yinqian Sun, Xiang He, Yiting Dong, Dongcheng Zhao, and Yi Zeng. Parallel spiking unit for efficient training of spiking neural networks. In 2024 Interaational Joint Conference on Neural Networks, pp. 1-8, 2024b.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromorphic data augmentation for training spiking neural networks. In European Conference Computer Vision, pp. 631-649, 2022.
Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128x 128 120 db 15 jus latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2):566 576. 2008
Changze Lv, Yansen Wang, Dongqi Han, Xiaoqing Zheng, Xuanjing Huang, and Dongsheng Li. Efficient and effective time-series forecasting with spiking neural networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 33624-33637, 2024a.
Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo, and Xipeng Qiu. Full parameter fine- lin-ttunig for large language models with limited resources. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 8187-8192 2024b.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):1659-1671 1997.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Towards memory- and time-efficient backpropagation for training spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6166-6176, 2023.
Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven K. Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron D. Flickner, William P. Risk, Rajit Manohar, and Dharmendra S. Modha. million spiking-neurols integrated circuit with a scalable communication network and interface. Science, 345(6197):6688
2014.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. The Fifth International Conference on Learning Representations, 2018.
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmenta- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 774-782, 2021.
Emre O. Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 1-63, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, pp. 8026-8037, 2019.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-111 2019.
Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 3446)3317--318,, 2023.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, volume 31, pp. 1419-1428, 2018.
13

---

Published as a conference paper at ICLR 2026
Sonali Singh, Anup Sarma, Sen Lu, Abhronil Sengupta, Mahmut T. Kandemir, Emre Neftci, Vi- jaykrishnan Narayanan, and Chita R. Das. Skipper: Enabling efficient snn training through activation-checkpoiin and time- skipping. In 2022 55th IEEE/ACM International Symposium Microarchitecture, pp. 565-581, 2022.
Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Lunguages, pp. 10-19, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pp. 5998-6008, 2017.
Kexin Wang, Jiahong Zhang, Yong Ren, Man Yao, Di Shang, Bo Xu, and Guoqi Li. SpikeVoice: High-quality text-to-speech via efficient spiking neural network. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 7927-7940, 2024.
Paul J. Werbos. Backpropagation through time: What does and how to do Proceedings of the IEEE, 78 18510): 1990.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 94(1):4464466 2023.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. Proceedings of the AAAI Conference on Artificial Intelligence, 33 (01):1311-1318, 2019.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation the equilibrium state. In Advances in Neural Information Processing Systems, volume 34, pp. 14516-14528, 2021.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. In Advances Neural Information Processing Systems, volume 35, pp. 20717-20730, 2022.
Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. In Advances in Neural Information Processing Systems, volume 36, pp. 64043-64058, 2023.
Man Yao, Ole Richter, Guangshe Zhao, Ning Qiao, Yannan Xing, Dingheng Wang, Tianxiang Hu, Wei Fang, Tugba Demirci, Michele De Marchi, et al. Spike-based dynamic computing with asynchronous censing-computing ncuromorphic chip. Nature Communications, 15(1):4464, 2024.
Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling ddikendriven transformer with efficient spike firing approxima- tion training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4):2973-2990, 2025.
Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. Glif: A unified gated leaky integrate-and -ire neuron for spiking neural networks. In Advances in Neural Information Processing Systems, volume 35, pp. 32160-32171, 2022.
Bojian Yin, Federico Corradi, and Sander M. Bohté. Accurate online training of dynamical spiking neural networks through forward propagation through time. Nature Machine Intelligence, 5(5): 518-527, 2023.
Chengting Yu, Lei Liu. Gaoang Wang, Erping Li, and Aili Wang. Advancing training efficiency of deep spiking neural networks through rate-based backpropagation. In Advances in Neural Information Processing Systemx, volume 37, pp. 115786 115786-115815, 2024.
14

---

Published as a conference paper at ICLR 2026
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6023-6032, 2019.
Hong Zhang and Yu Zhang. Memory-efficient reversible spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15):6733-16667, 2024.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In The Sixth International Conference on Learning Representations, 2018.
Hanle Zheng, Zhong Zheng, Rui Hu, Bo Xiao, Yujie Wu, Fangwen Yu, Xue Liu, Guoqi Li, and Lei Deng. Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics. Nature Communications, 15(1):277, 2024.
Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Qkformer: Hicrarchical spiking transformer using q-k attention. In Advances in Neural Information Processing Systems, volume 37, pp. 13074-13098, 2024.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Juntang Zhuang, Nicha Dvoreek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ODE. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp.
11639-11649, 2020.
Shihao Zou, Qingfeng Li, Wei Ji, Jingjing Li, Yongkui Yang, Guoqi Li, and Chao Dong. Spikevideo- former: An efficient spike-driven video transformer with hamming attention and Smmtthcal ( (0) (()S complexity. In Proceedings of the 42th International Conference on Machine Learning, 2025.
15

---

Published as a conference paper at ICLR 2026
A DETAILS OF MEMORY COST BREAKDOWN
Figure 2 illussrates the memory breakdown of stored feature maps (input tensors and internal states of all layers), model weights. gradients, optimizer states, and temporary runtime variables when training SNNs or ANNs on ImageNet (Deng et al., 2009). We evaluate SEW ResNet-34 Fang et al. 202 la and Spikformer (Zhou al. 2023) using the same settings as our main experiments (Appendix with ResNet-34 (He et al.] 2016) and ViT (Dosovitskiy et al. (021)) mirroring the settings of SEW ResNet-34 and Spikformer, respectively. SEW ResNet-34 and Spikformer are implemented using SpikingJelly (Fang et al. 2023a), and the LIF model with CuPy backend is adopted; For ResNet-34 and ViT. we use torchvision implementations (Paszke et al. 2019). We run the experiments on a single NVIDIA A100 GPU (80 GB, CUDA 12.2).
The memory usage for weights, gradients, and optimizer states can be easily computed by summing the sizes of all tensors of these kinds. To quantify the size of stored feature maps, we measure the allocated memory after the forward pass before the backward pass starts, and subtract the sizes of weights and optimizer states from the value. For runtime variables, first identify the critical layer with the highest peak memory. In other words, the network-level peak memory occurs during backpropagation layer' 1" The difference between the peak allocated memory layer I" and the allocated memory at the start of the layer's backward pass reflects runtime variable costs. Note that gradient sizes are slightly overestimated, as not all gradients ready when global peak memory is reached. Detailed results are shown in T
Table 6: Detailed memory breakdown of different networks when trained on ImageNet. Memory costs are measured in MB.
Network
Stored Features
Weights
Gradients
Optimizer States
Runtime Variables
ResNet-34
936.49
83.15
83.15
83.15
38.25
ViT
1629.20
100.05
100.05
200.11
91.19
SEW ResNet-34
8373.98 33372.02
83.15
83.15
83.15
40.50
Spikformer
113.26
113.26
226.52
496.00
B MEMORY EVOLUTION CURVES
Figure 3 and Figure 5 demonstrate the memory cost evolution within training iteration of Spiking VGG on CIFARIO-DVS. To get these curves, we record the allocated memory at the start, peak, and end of each target layer's forward pass (FP) and backward pass (BP). The resulting sequence. arranged in the temporal order of events, is
MPeak [Mpert : Fp1 , MMpp,, Mpter Fp2
Merd pp2.....
(6)
Matart
Peak BP=
Mend Matart
peak
M end BP
BP
where FP' and BP' denote the forward and backward pass of layer respectively. The global peak memory can be defined Mpeak max ((Mpp))) h U(Mpeal)) )1).
C DETAILS OF THE MAIN EXPERIMENTS
The main experiment is implemented using PyTorch (Paszke et al. 2019 and Spikinglelly (Fang al. 2023a),
Sequential CIFAR-10 Sequential CIFAR-10 (Fang et al. 2023b Chen al. 2024 is sequence classification task derived from the standard CIFAR- 10 benchmark (Krizhevsky 2009). It is widely used for evaluating SNNs' capability to learn long-term temporal patterns. In this task, the CIFAR-10 images are fed into the model column by column, mimicking the way humans sean pictures from left right. Each sample is sequence with T 32 elements, and each element contains 32 RGB pixels.
16

---

Published as a conference paper at ICLR 2026
There are 50, 000 training samples, 10, 000 test samples, and 10 classes. Following the practice in PSN (Fang et al. 2023b), we augment the training data with random mixup (Zhang et al.) (018)). random cutmix Yun et al., 2019), random horizontal flipping, TrivialAugment (Müller & Hutter 2021), predefined data normalization, and random erasing. An 8-layer ID convolutional SNN is employed (SCNN) (Fang et al. 2023b). Hyperparameters and running environment are listed in Table
DVS128 Gesture DVSI28 Gesture (Amir et al. 2017) an event-based gesture recognition dataset recorded by a DVS128 camera. It contains 11 gesture classes performed by 29 subjects under 3 illumination conditions with spatial resolution 128 x 128. For experiments, we follow the standard split provided in SpikingJelly (Fang al. 2023a): 1, 1,176 training samples and 288 test samples. frames, and no extra augmentations are applied. We use hyperparameters and the running environment.
Each recording is integrated into T
7B-Net, small-scale SEW ResNet (Fang al. 2021a as the backbone. See Table 7 for other
CIFARIO-DVS CIFARIO-DVS (Li al. 2017) neuromorphic vision classification task cre- ated by recording CIFAR-10 images (Krizhevsky 2009) through Dynamic Vision Sensor (DVS) (Lichtsteiner al. 2008). The dataset is composed of 10, 000 samples, each represented as an event stream with 2 channels and 128 x 128 resolution. Following the protocol of temporal effective batch normalization (TEBN) (Duan et al. 2022) and PSN (Fang et al. 2023b), we partition the dataset into 9, 000 training samples and 1, 000 test samples, downsample the resolution to 48 x 48, and integrate each event stream into T 10 frames. The data augmentation pipeline incorporates random resized cropping, random horizontal flipping, and Neuromorphic Data Augmentation (NDA) (Li et al. 2022). We adopt a Spiking VGG11 architecture, following the practice of TEBN (Duan et al. 2022) and f
PSN (Fang al. 2023b). Refer to Table for hyperparameters and running environment.
ImageNet ImageNet-Ik (Deng et al. 2009) is large-scale visual recognition benchmark containing about 1.28 million training samples and 50,000 validation samples across 1,000 classes. Training on the entire ImageNet dataset is computationally expensive, so we use its 32 subset instead, whose samples are evenly distributed across all 1000 classes. Since the peak memory cost during training is independent of the sample size, the memory footprint we report can faithfully reflect full-dataset training conditions. Each image resized to 224 X 224 resolution. We utilize SEW ResNet-34
Fang et al. a0 Spikformer (Zhou et al. 2023) and QKFormer (Zhou et al. 2024 architectures. The SEW residual connections in these architectures bring non-binary integer activation values compress them into 8-bit unsigned integers (uint8)
(Fang et al. rather than bits to avoid accuracy loss. For experiments using SEW ResNet-34, we use the same data augmentation pipeline in the original work (Fang et al. 2021a); for both Spikformer and QKFormer, we augment data using the procedure in the original QKFmmeer work (Zhou et al. 2024). Hyperparameters and running environments are provided in Table
2021a); for these activations,
Table Hyperparameter settings and running environment configurations for the main experiments.
Sequential CIFAR-10
DVS128 Gesture
CIFAR1O- DVS
ImageNet
SEW
Transformer
À
0.5
0.5
0.25
0.5
0.5
Ven
1.0
1.0
1.0 SGD(0.9)
1.0
1.0
Optimizer
SGD(0.9)
SGD(0.9)
SGD(0.9)
Adam W
L2 Reg.
0
5 10
0
5 x 10-2
Init. LR
0.1
0.1
0.1
0.1
0.001
Scheduler
Cosine
1te 64)
Cosine
Cosine TET
Cosine
Loss Batch Size
CE
CE
TET
Smooth CE
128
32
32
T
32
16
10
4
k
2
CUDA Version
12.3
12.3
12.3
12.2
12.2
Device
1x 4090
1x 4090
1x 4090
1x A100
1x A100
17

---

Published as a conference paper at ICLR 2026
D EXPERIMENTS OF MULTI-GPU TRAINING
The experiments in Table of the main are conducted on single GPU. To further validate the scalability of the proposed framework, we conduct multi-GPP training experiments on ImageNet (Deng et al., 2009) using QKFormer (Zhou al. 2024). The experimental setup follows Appendix C except that 1, 2, or 3 NVIDIA A100 GPUs are used for distributed data parallel (DDP) training. We set a per-device batch size of 32. Table X
reports the time and memory costs. Here, the hatch time cost refers to the average time per training iteration for a single GPU. Throughput accounts for all GPUs, measured as the total number of training samples processed per second. The peak allocated memory the maximum of peak allocated memory across all devices. Generally, in multi-GPU settings, method achieves substantial memory efficiency improvements while incurring a moderate increase in training time, which is consistent with the single- cases.
Table 8: Time and memory efficiency when training QKFormer ImageNet using multiple GPUs.
Throughput (samples s)
Peak. Alloc. Mem. (MB)
#GPUs
Neuron
Method
SULIF
BPTT
86.15 76.51 (0.89x)
44571.33
MELIF
O4
5219.93 (0.12x)
SJLIF MELIF
BPTT
168.43
44679.28
2
O₂
151.54 (0.90x)
5323.13 13 (0.12x)
SILIF
BPTT
235.45
44679.28
3
MELIF
O₂
211.01 (0.90x)
5323.13 (0.12x)
E ADDITIONAL MEMORY OPTIMIZATION TECHNIQUES
Low-Memory Optimizer (LOMO) Low-memory optimization (LOMO) (Lv al. 2024b) reduces the memory cost of gradients by updating W once its gradient G is computed, instead of waiting until all gradients are available. Unlike the stateless original LOMO (Lv al. (024b), we retain opti- mizer states (e.g., those of Adam (Kingma & Ba 2015)) to match the baseline cases. LOMO ensures that at most one gradient tensor resides memory a time, reducing Σ, Mg Equation to max, Me..
Automatic Mixed Precision (AMP) Automatic mixed precision (AMP) training (Micikevicius
et al. (018) can be optionally enabled to reduce overall memory usage and accelerate training by utilizing 16-bit floats for activations and gradients. The loss scaled to prevent underflow and ensure numerical stability.
F ACCURACY RESULTS
We report additional validation accuracy results in Table Note that these experiments are designed validate the mathematical equivalence of method with standard BPTT rather than maximize performance, so we do not apply advanced training tricks like random temporal delete (Fang et al. 2021a). We train 300, 192, and 100 epochs for Sequential CIFAR-10, DVS128 Gesture and CIFAR1O- DVS, respectively. For 32 ImageNet, we report validation accuracy at the fifth epoch to reduce training cost, which is sufficient to demonstrate the equivalence. The results show that MELIF attains accuracy nearly identical to SJLIF across all benchmarks, with minor discrepancies arising only from different backend numerical behaviors. In most cases, the optimization pipeline itself does not affect accuracy. However, O₃ and O₂ for Spikformer and QKFormer slightly influence accuracy due to the temporal segment partitioning on weight layers. Appendix G discusses this issue in detail. These small deviations stem purely from numerical computation rather than from any approximation in the gradient computation. The gradients produced by our method remain free from systematic bias.
1
18

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of validation accuracy (%). For 1 ImageNet, we report the validation accuracy 32 epoch 5. For SJLIF conditions, report mean + std over three runs. For MELIF conditions, report the results on a single run using a fixed seed.
SJLIF
MELIF
Task
Network
BPTT
BPTT
O₁
O
O
O4
Sequential CIFAR-10
SCNN
82.53:10.25
82.36
82.36
82.36
82.36
82.36
DVS128 Gesture
7B-Net
95.08
95.14
95.14
95.14
95.14
95.14
CIFARIO-DVS
Spiking VGG
85.98:10.25
86. 30.10
86.10
86.10
86.10
86.10
SEW ResNer-34 Spikformer
3.46-1-021
3.50
3.50
3.50
3.50
3.50
ImageNet
1.03--0.16
0.90
0.90
0.90
1.10
1.10
QKFormer
1.06-+-.12
1.20
1.20
1.20
1.10
1.10
G POTENTIAL SOURCES OF NUMERICAL DISCREPANCIES
Numerical discrepancies in gradients may arise when temporal GC segment partitioning is applied to layers with learnable parameters. Without temporal partitioning, the gradient first computed for each time step / E (1....,7) T) and batch sample {I,..., N). and then summed over the temporal and batch dimensions:
T
N
G
Σ t=1n=1
G
where G.,, denotes the gradient contribution at time step from sample n. In contrast, when the temporal dimension is partitioned (k = 2 for example), the accumulation is performed in two stages:
N
T
(
G
G (2)
G
(8)
Zm mm1
mm
followed by final aggregation:
G G (2))
(9)
Although mathematically equivalent to the unpartitioned case, these operations differ in numerical practice because floating-point addition is not associative. As result, reordering the accumulation of gradient terms leads to slight deviations in the final gradient values. This explains the minor accuracy deviations observed in Table Spikformer and QKFormer at Og and O4.
H TIME COST OF MEMORY OPTIMIZATION
Table 10 reports the time cost of the memory optimization pipeline at each optimization level. For SCNN, the overhead increases moderately with higher optimization levels, reflecting the additional computations from spatial and temporal segment partitioning and greedy segment restoration. For QKFormer, the jump in time cost from O3 to O is much more pronounced, primarily due to the transformer's greater depth, which increases profiling costs and the number segments to iterate Importantly, this overhead is incurred only once before training and is negligible relative the total training time.
Table 10: Time (in seconds) spent by the memory optimization pipeline at each optimization level.
O
Oz
O
O4
SCNN
1.08
26.32
30.63
75.41
OKFormer
1.13
44.91
78.58
564.26
19

---

Published as a conference paper at ICLR 2026
I TUTORIAL
We provide a brief tutorial on using the proposed automatic memory optimization pipeline, taking the training Spiking VGG on CIFARIO-DVS as an example. The model can be defined using PyTorch PPaszke et al. 2019) and SpikingJelly (Fang et al. 2023a) as shown in the code below.
class VGGBlock (nn.Module) def iiiiiii self, in_plane, out_plane, T, neuron_type, preceding_avg_poolffll **kwargs
):
super ()n_))___iit___ ()
proj_bn []
if secceding_vvg_ooe
proj_bn.append (nn (nn.AvgPool2d )2))
proj_bn +=
nn.Conv2 (inppaae,, out_plane, 3, 1, 1),
12
14
self.proi_br = See[o (4proi
"wargs["T]] [I"]] T
16
self nelroneuron = get_neuron (neuron_type, **kwargs)
def forward x_seq):
return nelfon (self, projfffproj_.... (x (x_seq)
21
class CIFARIODVSVGG (nn (oddul))))))
def _nnit__(sell т, necron_type, dropout=0.25, **kwargs):
23
()per()._iiit...
fall.feetures. = nn. Sequential
25
VGGBlock (2, 64, T, neuron_type,, False, **kwargs),
VGGBlock (64, 128, T, neuron_type, False, **wargg))
27
VGGBlock (128, 256, T, neuron_type, True, **kwargs),
VGGBlock (256, 256, T, neuron_type, False, **kwargs),
29
VGGBlock (256, 512, T, neuron_type, True, **kwargs),
VGGBlock (512, 512, T, neuron_type, False, **kwargs),
31
VGGBlock (512, 512, т, neuron_type, True, **kwargs),
32
VGGBlock (512, 512, T, neuron_type, False, **kwargs), layer.
33
step_mode= mode="m"),
d = int (48 / / 2)
36
1 [nn. Dropout (dropout) if dropout > 0 else [] 1. a.append Lin.aara self.classifier = nn. Sem.entnttiall (*1)
d d, 10))
38
for in self melf.modules
40
if isinstance (m, cn.Convdd)
nn.init kaiminggnormal___
42
weigith mode=' fan_ rell
43
44
45
def forwardsself, input): # imputtshapp = [N, T, C, H, W]
47
input traut.transsose (0, 1) conttiguoss ()
# [T, N, C, H, W]
49
x self features (input)
flatten (x, 2) # [T, N, D]
51
x self.classifan (x)
return
20

---

Published as a conference paper at ICLR 2026
Users can define spatial partitioning rules by implementing the spatial split method, which returns a tuple of submodules corresponding to the spatial subsegments a layer. For instance, VGG block can be split into convolution-plue patriioo--pluuu--a--ooooooo batch-norm segment and spiking neuron segment.
class VGGBlock Monuleel) def ssppppal_spltt______e_l__ (selft)
return self.proj_bn, self welf.now neuron
To define temporal partitioning rules, users should implement the tc_ aocinit.___ thitt_sttees_____ and ttc_oorwarr______.. methods. _tt__nit__tttt__eee______ tastt_s_tse_____ returns list of initial hidden states, while wt__t_ffoowrrrrc takes a chunk of input tensors along with the initial hidden states, and then returns the corresponding outputs and updated hidden states. The stateless layer container BeqToANNContaing is the simplest case, where no hidden states are required and the temporally chunked forward pass just the same as the container's original forward pass.
class SeqIoAMWContainer (layer SegJoANNContainer)
"""Stateless layer container that supports temporal chunking"'" def _te_init Siiit_sttts________ (self, x_seq)
return []
def _tt__forrrrdoocee dscforward__seeff xc) return [self. (oreard (xc),
A more complex example is the NeuronMaxPool block:
class NeuronMaxPool (nn Module))
def __iiitttssss_ seef,, neuron_type, **kwargs):
super .)._______..___ ______)_________ init ()
neuron get_neurop neuron (neuron_type, **kwargs)
self.pool = SecToAMCCotaiine(
nn. Mr.MaxPPol220
pearnel_size-i stride=2, padding=1)
def forward (self, x_seq)
10
return self. (self neuron (x_s)q)
12
def to itt_iiitssttee___st (self,,, x_seq)
device, dtype x_seq.deviee device, x_seq.dtype
14
return (torch.zeros ([), device atype=dtype) ]
def tc ttt_oorwrdd (self, xc, v):
17
se, yc self woo. (sc) return yc, v
sttpeppsstte__eeee uptee (xc, v)
19
which means that the hidden state (the neuron's membrane potential) is initialized zero, and the temporally forward pass consists of a multi-step state update of the neuron followed by max pooling. In this example, there is only one input, hidden state, and output. However, multiple inputs, hidden states, and outputs are also supported. Finally, the memory oot zaiiition function can be called to apply the automatic pipeline.
net = CIFAR1 ODVSVG (T, neuron_ dropout, **kwargs) net = memory memoryy_pttimization
net,
instance= (VGGBlock,), tummy_ingut=torrch.r rand (32, т, 2, 48, 48),
21

---

Published as a conference paper at ICLR 2026
cmmpress_=-rruee
1evel=4,
verbose=Irue,
sytporal_split_fcttte
where instann ance specifies the layer types to apply gradient checkpointing, dunmy_input is sample input tensor for profiling, compress indicates whether to compress input spikes, level sets the optimization level, and temporal_splae _as____ppt___ttt___ granularity of temporal partitioning. After optimization, the model can be trained using standard procedures without further modification.
the k factor that controls the
Note that the pipeline will automatically check whether spike compression is applicable each GC segment based on the input distribution. Users can also manually specify spike compressors by setting the module's s x compressoo attribute. For instance, for layer in SEW residual block (Fang et al. (222a)) whose input is non-binary integer tensors, we can compress the input to 8-bit unsigned integers (uint8):
class SEXBlock (nn Module)
def _niii_nns____ (self, c_in, c_mid, neuron_type, **kwargs):
super super()._init_ init_nn ()
self.conv = nn Sengeutttia
Conv3x3 (c_in, s_mid, neuron_type, **kwargs),
Conv3x3 (c_mid, c_in, neuron_type, **kwargs),
self.conv [0] .o__0____o___ x_cooprresoor = "dimtBppieecoopeessor"
def forward (self, out self conv (x)
torch. Tensor)
11
out out + x return out
13
THE EFFECT OF SPIKE COMPRESSION ON TRAINING MEMORY
Table reports the peak memory usage corresponding Figure 5 The majority of memory saving comes from layer-wise GC (BPTT vs. O₁, compression disabled), while spike compression alone only provides marginal memory savings (O,, compression disabled vs. O₁, compression enabled).
However, shown in Figure 5 spike compression reduces the memory footprint of activations, thus substantially lowering the instantaneous memory usage of deeper layers. This reduction creates the headroom for stronger spatio-temporal partitioning, leading larger memory savings at higher optimization levels (O3 and O4). In summary, spike compression not the main source of memory efficiency, but enabling factor that allows spatio-temporal partitioning to further reduce peak memory.
Table 11: Peak allocated memory (MB) of Spiking VGG when training on CIFARIO-DVS with spike compression enabled or disabled (T = 10, batch size is 32). See Figure
Compression
BPTT
0, (+GC)
Og (+partitioning)
O4 (+restoration)
2887.75
2349.39
2349.39 2530.66
x
6131.07
2892.63
2530.66
K DETAILED RUNTIME PROFILING
Table reports the the forward and backward runtime for each layer in Spiking VGG. Note that the fully connected classification head is omitted since change is applied to it across all optimization
22

---

Published as a conference paper at ICLR 2026
levels. GC introduces additional backward computation roughly equivalent single extra local forward pass. Spike compression and decompression further add small overheads to both forward and backward passes, but the increase is negligible relative to the total runtime, demonstrating the efficiency of bit compression; note that ConvO do not apply input spike compression. In this example, spatial partitioning is applied only to Conv1, while temporal partitioning is skipped since it does not yield additional memory benefits (see Algorithm 2). As result, virtually no extra computational cost. Finally, greedy segment restoration significantly reduces the computation load of both passes. Conv3 and Conv5 are reverted to standard BPTT blocks, and their forward and backward runtimes return BPTT level.
Table 12: Layer-wise runtime profiling for Spiking VGG on CIFARIO-DVS (MELIF, T 10, batch size is 32). Results are averaged over 200 iterations with 10 warmup iterations and reported in milliseconds.
Condition
Stage
ConvO
Conv
Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
fwd 2.48
5.96
4.16
5.22
2.80
3.98
1.04
0.91
BPTT
bwd
4.28
12.82
8.57
10.13
5.58
8.29
2.26
1.95
fwd 2.52
5.99
4.20
5.26
2.81
4.01
1.07
0.92
GC
bwd
6.80
18.86
12.78
15,42
8.45
12.33
3.27
2.89
fwd 2.51
6.19
4.51
5.46
3.04
4.11
1.18
1.02
O
bwd 6.81
19.08
13.21
15.63
8.71
12.46
3.41
3,04
fwd 2.49
6.15
4.48
5.38
3.05
4.09
1.17
0.99
Os
bwd
6.72
19.02
13.24
15.62
8.56
12.22
3.39
2.98
fwd
2.50
6.16
4.49
5.18
3.03
3.99
1.16
1.00
O₂
bwd
6.76
19.01
13.23
10.08
8.55
8.24
3.34
2.97
We further investigate how training time cost scales with the number of checkpointed layers and temporal splits. Since GC performs one fixed-cost recomputation per segment, the total overhead increases as the number GC segments grows. However, the scaling is not strictly linear, since recomputation cost varies across layers. As shown in the left plot of Figure 8 the overhead grows as more GC segments are added, but the increments are uneven. From Table 12 we know that spatial partitioning has almost effect training speed. In contrast, temporal partitioning actually reduces temporal parallelism, thereby slowing down training (cspecially for models with a large T). To illustrate this effect, measure per-batch training time cost of DH-SFNN SHD. Training time per batch increases nonlinearly with the the temporal partitioning factor k.
317.72
Training Time (ms batch)
Training Time (ms batch)
120
120.62
122.50
123.29
310
303.41
110
112.81
106.72
285.30
100
99.63
280
2775.5
90
92.49
270
272.02
88.20
0
1.2
4.5
10
20
25
Figure 8: Left: taining time per batch (forward backward) of Spiking VGG on CIFARIO-DVS as a function of the number of GC segments. The first L layers adopt GC, where L 0 corresponds to standard BPTT. = 10, and batch size is 32. Right: training time per batch of DH-SFNN on SHD function of the temporal partitioning factor à under Og. k I indicates temporal partitioning. = 100, and batch size is 128. Experiments are run on a single 4090 (24 GB).
23

---

Published as a conference paper at ICLR 2026
L FINE-GRAINED ABLATION STUDY
Table 2 shows the ablation results of optimization levels on CIFARIO-DVS, To better understand the impact of each GC adjustment strategy, we conduct fin-graineed ablations on QKFormer for ImageNet. As shown in Table [3], spatial partitioning provides substantial memory reduction with almost no impact on training throughpur. Temporal partitioning also reduces memory usage, though introduces a slight slowdown due to reduced temporal parallelism. When combined, the two strategies complement each other effectively, lowering peak memory to 5219 MB. Greedy restoration further improves throughput while preserving memory savings. With all three strategies jointly applied, throughput reaches 76.51 samples/s, the highest among all variants; notably, it even exceeds the condition without temporal partitioning because there are more GC segments restored to standard BPTT blocks. Overall, these ablations clarify the individual and collective contributions of the three components to memory and computational efficiency, confirming the intended synergistic effect of the full pipeline.
Table 13: Ablation study of QKFormer for ImageNet on the impact of spatial partitioning, temporal partitioning, and greedy restoration. All conditions adopt MELIF, GC and spike compression.
Spatial Partition
Temporal Partition
Greedy Restoration
Throughput (sample/s)
Memory (MB)
Annotation
66.13
7726.55
O
66.02
6834.48
O2
63.82
6920.25
73.17
7725.87
64.01
5219.93
Os
73.07
6833.87
70.24
6920.25
76.51
5219.93
O1
M LOSSLESS SPIKE COMPRESSORS
As discussed in Section 4.3, adopt bit representation as the default lossless spike compressor due to its superior speed and memory efficiency. To validate this choice, we compare it with two alternatives: sparse representation (storing indices of non-zero elements) and lossless bit-stream compressor (e.g., ANS from nvCOMP Experiments are performed on Sequential CIFAR- 10 using SCNN (T 32, batch size is 128) on NVIDIA GeForce RTX 4090. The results in Table 14 show that bit compression consistently achieves the lowest memory footprint and highest throughput under both O and O₁. Sparse representation yields slightly higher memory consumption and lower speed, while ANS provides moderate compression gains but is substantially slower.
For more direct comparison, we evaluate compressed size and cost across a range of firing rates p. using float32 spike tensor of 10" elements (38. 14 MB) as input. Time costs are averaged over 100 trials following 20 warm-up runs. As Table 15 and Table 16 show, bit compression consistently produces fixed-size 1.19 MB representation regardless of sparsity, whereas sparse representation's memory efficiency decreases rapidly as p increases. ANS achieves small compressed sizes at low sparsity but is over order of magnitude slower. Notably, firing rates in modern activation-basee based SNNs typically fall within 0.02 to 0.35 (Zhou et al. 2024). a regime in which bit representation performs effectively. These results confirm that bit representation is both efficient and effective.
time
N LIMITATIONS, FUTURE WORK, AND SOCIAL IMPACTS
While the proposed memory optimization pipeline achieves significant memory reduction with broad compatibility and preserved accuracy, several limitations remain. First, GC inevitably introduces computational overhead due to the recomputation of intermediate features during backward pass, and spike compression also slightly adds computational burden. Although we alleviate these by greedily
24

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of lossless spike compressors (bit, sparse, ANS) on Sequential CIFAR-10
(SCNN, T = 32, batch size is 128).
Throughput (sample /
Memory (MB)
Compressor
01
O₂
O1
O₂
bit
496.79
474.98
4768.11
5138.76
sparse ANS
527.53
516.99
4454.24
5020.27
509.42
497.71
2029.53
2542.50
Table 15: Compressed memory (MB) of spike compressors across firing rates
Compressor p 0.01
== 0.1
p 0.2
p p= 0.5
p p= 0.8
p == 0.9
bit
1.19
1.19
1.19
1.19
1.19
1.19
sparse
0.76
7.63
15.27
38.14
61.03
68.66
ANS
0.30
1.68
2.82
5.14
6.61
6.95
Table 16: Compression time cost (sec) of spike compressors across firing rates
Compressor p 0.01
p= 0.1
p p= 0.2
p p= 0.5
p p= 0.8
p p= 0.9
bit 0.1234
0.1257
0.1262
0.1216
0.1250
0.1215
sparse
0.1538
0.1568
0.1616
0.2210
0.2965
0.3119
ANS 2.6097
2,4416
2.4407
2.5761
2.6305
2.6603
restoring Now-memory-ippas GC segments to standard BPTT segments and implementing efficient Triton kernels for compression and decompression, the overall training speed can be reduced to about 0.8x that of the baseline in the worst case. Second, audio classification benchmarks, which is a common practice in SNN research. While the pipeline theoretically applicable to other modalities, its effectiveness tasks such language modeling remains to be validated. Thirdly, the proposed method is specially designed for layer-wise training setting (Table(5). which common for modern medium- large-scale SNNs. Its applicability to strict online learning settings, where model parameters are updated immediately at each time step, is limited. Fourth, the current pipeline follows user-specified partitioning schemes to reduce the search space. Automatic generation of partitioning rules are not yet supported.
experiments mainly focus visual and
Future work can address these limitations several directions. One direction is to evaluate the framework on large-scale SNNs for language tasks, and to design optimization strategies tailored to language backbones. This would broaden the applicability and further demonstrate the generalizability of the pipeline. Another promising direction is to automatically generate spatio-temporal partitioning rules, enabling fully automated memory optimization without manual intervention. To this end, more principled optimization approaches, such as dynamic programming (Gruslys al. 2016). could be employed to search for efficient partitioning schemes.
By reducing the memory cost of SNN training while retaining BPTT-level accuracy and broad compatibility, our method lowers the hardware barriers for scaling up SNNs. The pipeline facilitates the deployment of energy-efficient SNNs on resource-constrained platforms, including mobile and edge 1oT devices. Such advances can democratize access to neuromorphic computing, promote sustainable AI solutions, and ultimately contribute to reduced energy consumption in intelligent systems. We do not see any negative societal impacts from this work.
USE OF LARGE LANGUAGE MODELS
We utilized large language models (LLMs) refine phrasing, correct spelling and grammar, and enhance the clarity of expressions. Additionally, LLMs were employed to assist in result visualization, such providing initial code templates or optimizing figure layout suggestions. However, the core ideas, methodological design, code framework development, and key contributions of this paper were independently conceived and completed by the authors, without relying on LLMs for substantive support.
25


Reviews:
[{"review_id": "pzXTlDBVK7", "reviewer": "Reviewer_cB7e", "summary": "This paper investigates memory compression mechanisms in large language models (LLMs), focusing on achieving lossless or near-lossless memory efficiency. It systematically analyzes the relationship between attention redundancy, representation compression, and retrieval fidelity, and proposes a new compression architecture intended to retain semantic fidelity while reducing activation and KV cache memory.", "questions": "1. How exactly is “lossless” defined in your experiments? Is it equivalent to maintaining identical outputs, or bounded semantic deviation?\n2. Have you measured mutual information or entropy changes before and after compression?\n3. How does your method compare to dynamic memory eviction or entropy-based key selection methods?\n4. Does the compression framework support online adaptation — e.g., varying compression ratio during inference?\n5. Are the proposed memory structures compatible with GPU-efficient implementations (e.g., FlashAttention or PagedAttention)?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The paper addresses an important issue in modern LLM systems — the scalability of context memory and cache compression. This is a crucial bottleneck for both open-source and proprietary systems. The authors provide a solid overview of various strategies (quantization, pruning, clustering, key-value cache compression, etc.), framing them under a unified memory-entropy perspective. This synthesis is useful for the community. The experiments, particularly those on retrieval fidelity and downstream reasoning, demonstrate that small amounts of information loss can lead to disproportionate accuracy degradation, supporting the motivation for “lossless” compression.", "weaknesses": "1. The “lossless” definition used in the paper lacks a rigorous mathematical formulation. The text refers vaguely to “semantic equivalence under compression,” but no formal mapping (e.g., bijection between original and compressed latent space preserving mutual information) is provided. This makes it hard to assess the conceptual contribution.\n2. While the paper cites several existing compression methods (e.g., KV cache quantization, low-rank adaptation, memory token merging), it is unclear what aspect is fundamentally new. The proposed framework seems to integrate known techniques rather than introduce a distinctly novel algorithm.\n3. The experiments appear to be conducted on small- to medium-scale models and limited benchmarks. It’s unclear how the proposed method scales to models beyond 7B parameters or generalizes to instruction-following and multimodal tasks. In addition, most results lack variance reporting and ablation analysis.\n4. The metrics primarily evaluate perplexity and accuracy but not information preservation. A more appropriate evaluation would involve mutual information, reconstruction error, or similarity metrics on attention distributions.\n5. The claim that “compression can improve reasoning stability by removing noise” is speculative and not well supported by experiments. Correlation is shown, but no causal analysis or controlled experiments are presented.\n6. The paper repeatedly claims that “redundant activations lead to inefficiency” and “compression improves generalization,” but it only shows correlations between compression ratio and performance metrics (e.g., perplexity).\nThere is no controlled experiment isolating causal factors — for instance, whether observed gains arise from reduced redundancy, implicit regularization, or simple noise filtering. Without causal validation, these claims remain speculative.\n7. Memory compression may cause unstable behavior, especially in autoregressive decoding where small perturbations can cascade.\nThe paper does not evaluate output stability under repeated sampling, sensitivity to compression noise, or robustness across different temperature settings. Without such analyses, it is unclear whether the approach is reliable for real-world use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQdIS9zIA9", "reviewer": "Reviewer_pEn5", "summary": "Training spiking neural networks (SNNs) is highly memory-intensive, typically requiring O(LT) memory. This paper addresses the issue by taking advantage of the binary nature of spikes and adopting a checkpointing strategy. During training, the computational graph is not constructed; instead, the input data is passed through the network while intermediate activations, if they are spikes, are compressed to reduce memory usage. Once the loss at the final layer is computed, a local computational graph is reconstructed at each layer, starting from the final layer and proceeding backward to the first layer, using the stored intermediate representations as inputs. This strategy significantly reduces the memory footprint during training.", "questions": "1. Is it correct that the main source of memory efficiency arises from the fact that spike activations can be compressed, rather than being stored as 32-bit floating-point representations?\n\n2. This method appears to rely on having access to the entire sequence of T input spikes during training. In an online setting, where input spikes arrive sequentially, how would the model behave? Could the authors comment on its applicability and performance in such scenarios?\n\n3. The checkpointing strategy has also been successfully applied to train Neural ODEs [1]. Given the conceptual similarities between SNNs and NODEs from the ODE perspective, the authors may consider citing this related work for completeness.\n\n4. Could the authors provide results obtained after training the models for the full number of epochs (e.g., on DVS-CIFAR10 or other datasets)? In the paper, only partial training results are presented to demonstrate the memory advantage. It would be valuable to also show that the accuracy remains unaffected when the model is trained to convergence.\n\n[1] https://proceedings.mlr.press/v119/zhuang20a/zhuang20a.pdf", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Paper is easy to follow. \n\nThe proposed method can be easily adapted to train SNN models in a memory-efficient way.", "weaknesses": "I guess it's not very efficient if the SNN models are trained online, here it is assumed that the entire spiking data is available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZ0zs6gi1G", "reviewer": "Reviewer_eX6a", "summary": "This paper presents an automatic, lossless memory optimization pipeline for training spiking neural networks (SNNs) using backpropagation through time (BPTT). The core idea is to combine layer-wise gradient checkpointing with lossless spike compression, targeting the two major memory bottlenecks in SNN training—internal state storage and per-layer spike activations.", "questions": "1. Checkpoint adaptation: How sensitive is the memory efficiency to the chosen “level” parameter (O1–O4)? Could adaptive tuning be integrated dynamically during training? How does the system decide spatial vs. temporal split thresholds? Could these be learned or auto-tuned?\n2. The paper reports ≤20% slowdown, but could the authors provide more detailed runtime decomposition? Does the time overhead scale linearly with the number of checkpoints or show nonlinear interactions with spatial/temporal splits?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n1. Introduces a lossless and automatic memory-saving approach for SNNs, avoiding the accuracy compromises typical in online learning or reversible architectures.\n2. The combination of gradient checkpointing with binary spike compression is novel and elegantly leverages SNN characteristics (binary activations).\n\nQuality\n1. Provides comprehensive theoretical analysis (Equations 2–5) for memory cost and correctness.\n2. Extensive empirical validation on multiple architectures and datasets.\n3. Clear comparisons with other efficiency-oriented methods (online learning, BPTT-to-BP, reversible networks).\n\nClarity\nWell-structured paper with good logical flow. Figures illustrate the differences between BPTT, checkpointing, and the proposed compression. The language is clear and technically rigorous.\n\nSignificance\n1. Addresses one of the most critical barriers in SNN research: high memory cost during training.\n2. Enables scaling SNNs to large architectures and long sequences, potentially democratizing access to neuromorphic research on commodity GPUs.", "weaknesses": "1. Limited exploration of trade-offs: Although the paper claims ≤20% slowdown, more fine-grained runtime profiling across levels would strengthen the argument for scalability. The additional computational cost of spike compression/decompression could be analyzed more quantitatively.\n2. Sparse ablation: The adaptive checkpoint adjustment (spatial vs. temporal) is key but not deeply evaluated in isolation; Ablations showing how each component (spatial partitioning, temporal partitioning, greedy restoration) contributes to performance would enhance interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "okshrturIW", "reviewer": "Reviewer_XHsK", "summary": "The paper proposes to use gradient checkpoint (GC), possibly combined with compression, to reduce the peak memory required for training of SNN. The advantage of the proposed methods is that they do not introduce mathematical discrepancies (except for possible numerical ones), thus maintaining the training accuracy. The paper introduces a simple yet effective heuristic to minimize the peak memory by combining spatial and temporal GC segment partitioning. The paper is well written and easy to read.\n\nOverall, the training performance penalty remains very limited (>0.9x on average) compared to the gain in peak memory (<0.4x on average), making the proposed method relevant and applicable to real world models training. However, the novelty and innovation of the paper remains limited: GC is a well-known technic, as well as compression.", "questions": "- Do the authors intent to release their method source code?\n- It could be interesting for the authors to bring more insight about the different compression methods explored? In particular, by providing some figures on how they compare in terms of performances and peak memory?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to read;\n- The peak memory reduction technic and heuristic proposed in the paper is relatively generic and applicable to many SNN models;\n- The is a mathematical equivalency with the original model (despite possible numerical discrepancies);\n- The impact on learning performance is very limited, compared to the gain on peak memory.", "weaknesses": "- The scope and impact of the paper remains limited;\n- The novelty limited: GC optimization and compression are well-known technics;\n- It is not clear how the proposed method will benefit the community: while some code snippets are provided in the supplementary materials, the possible release or diffusion of the source code is not mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)