Spiking Neural Networks Need High-Frequency Information
Authors: Yuetong Fang
Deming Zhou
Ziqing Wang
Hongwei Ren
ZeCui Zeng
Lusong Li
shibo zhou
Renjing Xu
Venue: NeurIPS 2025 poster ()
Keywords: Spiking Neural Networks
Vision Transformers
Primary area: applications
TL;DR: The paper reveals that the performance gap between SNNs and ANNs stems not from information loss caused by binary spike activations, but from the intrinsic low-pass filtering of spiking neurons.

Abstract: Spiking Neural Networks promise brain-inspired and energy-efficient computation by transmitting information through binary (0/1) spikes. Yet, their performance still lags behind that of artificial neural networks, often assumed to result from information loss caused by sparse and binary activations. In this work, we challenge this long-standing assumption and reveal a previously overlooked frequency bias:  **spiking neurons inherently suppress high-frequency components and preferentially propagate low-frequency information.** This frequency-domain imbalance, we argue, is the root cause of degraded feature representation in SNNs. Empirically, on Spiking Transformers, adopting Avg-Pooling (low-pass) for token mixing lowers performance to 76.73% on Cifar-100, whereas replacing it with Max-Pool (high-pass) pushes the top-1 accuracy to 79.12%. Accordingly, we introduce **Max-Former** that restores high-frequency signals through two frequency-enhancing operators: (1) extra Max-Pool in patch embedding, and (2) Depth-Wise Convolution in place of self-attention. Notably, **Max-Former** attains 82.39% top-1 accuracy on ImageNet using only 63.99M parameters, surpassing Spikformer (74.81%, 66.34M) by +7.58%. Extending our insight beyond transformers, our  **Max-ResNet-18** achieves state-of-the-art performance on convolution-based benchmarks: 97.17% on CIFAR-10 and 83.06% on CIFAR-100. We hope this simple yet effective solution inspires future research to explore the distinctive nature of spiking neural networks. Code is available: https://github.com/bic-L/MaxFormer.

Spiking Neural Networks Need High-Frequency Information
Yuetong Fang', Deming Zhou', Ziqing Wang', Hongwei Ren',
Zecui Zeng', Lusong Li³, Shibo Zhou?: Renjing Xu"
The Hong Kong University of Science and Technology (Guangzhou) Brain Mind Innovation INC IJD Explore Academy "Northwestern University
yfang8700ccomect. hwust-gz. edu. en, bob@brain-mind com. en, renjingxudhkust-ga www.gu....dudddu. edu.
Abstract
Spiking Neural Networks promise wrain-inspired and energy-efficient computation by transmitting information through binary (0/1) spikes. Yet, their performance still lags behind that of artificial neural networks, often assumed to result from information loss caused by sparse and binary activations. In this work, we chal- lenge this long-standing assumption and reveal previously overlooked frequency bias: spiking neurons inherently suppress high-frequency components and preferentially propagate low-frequency information. This frequency-domat imbalance, we argue. is the root cause of degraded feature representation in SNNs. Empirically, on Spiking Transformers, adopting Avg-Pooling (low-pass) for to- ken mixing lowers performance to 76.73% on Cifar-100 whereas replacing it with Max-Pool (high-pass) pushes the top-1 accuracy to 79.12%. Accordingly, we introduce Max-Former that restores high-frequenny signals through two frequency- enhancing operators: (1) extra Max-Pool in patch embedding, and (2) Depth- Wise Convolution in place of self-attention. Notably, Max-Former attains 82.39% top- accuracy on ImageNet using only 63.99M parameters, surpassing Spikformer (74.81 1,,,,,.6668 66.34M) by +7.58%. Extending insight beyond transformers, achieves tfatcooftthe-a performance on convolution basedd bench- marks: 97.17% CIFAR- 10 and 83. 83.06% on 100. We hope this simple yet effective solution inspires future research to explore the distinctive nature spiking neural networks. Code is available: https///tthbb.coommbe///atttffot
S-MLP
Spiking Neuron CONY
Embedding
Patch Embedding
Aug- -Pol
Max-PPol
S.LPP
S-MLP
5 Spiking Neuron CONY
Perfomance: 76.73 %
Performance: 79.12 %
(a)
(b)
(c)
Figure 1: Spiking Transformer architectures: (a) Avg-Pool vs. (b) Max- Maol for token mixing, with (c) detailed implementation the Spiking MLP (S-MLP) block. mainstream (non-spiking) Vision Transformer research, Avg-Pool that captures global low-frequency patterns is a more common token mixing strategy than Max-Pool thigh-pass) 2. Surprisingly, Spiking Transformers, replacing Avg-Pool PPol with Max-Pool yields a +2.39% improvement on Cifar-100.
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
ReLU
5-Neuron
Relatve Log Amplitude
ReLU
S-Neuron
ReWU
Spectum
-2
GradC
-4-
6-
0.0 Frequenty (x)
1.0
(a)
(b)
(c)
(d)
Figure 2: Comparison between ReLU and spiking neuron (S-Neuron): (a) Input images; (b) Fourier spectrum analysis of output features processed as input-+ activation+ weighting, with high-frequency regions marked (red dashed boxes: regions >0.55x max amplitude) and (c) the corresponding relative log amplitude; (d) GradCAM comparison with identical architectural setting following (9. with the converted Spiking Transformer using 256 timesteps. Spiking neurons cause the rapid dissipation of high-free feghffrequency components, which consequently leads to the degradation of feature representations.
1
Introduction
Spiking neural networks (SNNs) are emerging as an energy-efficient alternative to conventional artificial neural networks (ANNs) 4. Their efficiency arises from spiking neurons that utilize spatiotemporal dynamies to mimic biological computation in the human brain [5]. In ANNs, all neurons within the same layer must await the complete processing of real-valued, dense tensors before any information can flow to the subsequent layer. SNNs, however, transmit information asynchronously, with spiking neurons consuming energy only when receiving or emitting spikes ("1"), otherwise remaining inactive 7 This binary activation pattern enables SNNs replace the energy-intensive muutiply--nd-accumee simpler spike-based accumulation. Leveraging the energy-efficiency benefits, modern SNN variants, such Spiking Transformers that integrate the powerful Transformer architecture with -pike- computing, have gained growing attention [
(MAC) operations that are essential in ANNs with much
Despite their energy efficiency, the discrete nature of spike-based computation presents both opportu- nities and challenges. major obstacle for SNNs remains their performance gap relative to ANNs. This gap is often attributed to the so-called "representation error" [2) which argues binary spike trains inherently limit the precision of feature representations compared to continuous activations. However, this seems inconsistent with the established consensus in the standard deep learning lit- erature, where low-bit and even binary networks can still achieve comparable accuracy [13 [4]. Further, it should be noted that SNNs operate on temporal sequences: while spiking neurons strictly transmit binary signals at each individual time step, a train of spikes spanning ri simulation timesteps can encode activation values with at least log(n)-biz -bit precision [15] [6]. The conflicting observation reveals an unexplored dimension in understanding SNN performance limitations.
It is natural to think from the frequency domain. Spiking neurons produce discrete, pulse-like activa- tions, fundamentally distinguishing their frequency response from continuous activation functions commonly used in standard networks (e.g., ReLU (17). Prior works have suggested that spiking neurons may enrich signals with local details (high frequencies)
[8]] 19 However, in Figure 2 (b-c), we observed a surprising phenomenon: when examining the end-to-end information flow of input -> activation >> weighting, rather than focusing solely on the property of activation functions, spiking neurons tend to propagate low-frequency information more prominently than ReI.U.
Feature degradation observed in SNNs may instead originate from the rapid dissipation of high- frequency components, which prevents the network from effectively capturing local, fine-grained information (Figure 2 (d)). To support this finding, we perform a simple experiment in which non-paun non-parametric pooling operators. i.e., Max-Pool and Avg-Pool, serve as token mixers in Spiking Transformers, shown in Figure From the frequency domain perspective, Max- excels at capturing local figh-frequency details (e.g., local edges/textures), whereas Avg-Pool favors global low-frequency patterns. Intriguing, Spiking Transformers exhibit opposite preferences to ANNs in token mixing: while standard Transformers typically employ Avg-Pool for token mixing D 2 replacing with Max-Pool Spiking Transformers yields a +2.39% improvement on CIFAR-10G making it surpass the well-tuned Spikformer [8] bascline by 0.97%.

---

Overall, this work provides further theoretical and empirical evidence supporting the view that high-frequency information is essential for SNNs:
We provide the first theoretical proof that spiking neurons inherently act as low-pass filters the
network level, revealing their tendency to suppress high-frequency features.
which restores high-frequency information in Spiking Transformers via two lightweight modules: extra Max-Pool in patch embedding and Depth- Wise Convolution (DWC) in place of early-stage stage self- attention.
We propose
Restoring freh-freuencyy information significantly improves performance while saving energy cost. On ImageNet, Max-Former achieves 82.39% top-1 accuracy (+7.58% over Spikformer) with 30% energy consumption and lower parameter count (63.99M vs. 66.34M).
Extending the insight beyond transformers, Max-ResNet-18 achieves state-of-the-art performance on convolution- based benchmarks: 97.17% on CIFAR-10 and 83.06% on CIFAR-100.
We believe this straightforward yet powerful solution will motivate future research to explore the unique properties of SNNs, beyond the established practice in standard deep learning.
2
Preliminary and Related Works
2.1 Spiking Neuron Models
SNNs implement spike-driven processing through biologically-insprre neuron models for non-linear activations. The Leaky Integrate-and-Fire (LIFF) model is a widely adopted abstraction of this behavior, offering an effective balance between biological plausibility and computational efficiency [20]. The discretized LIF model under each simulation timestep can be formulated as:
Unn] = f(V/n (m))), S|n] H (m Va.),
(I)
[U[m] [ UUn),
Vah.
S(n] 1, S|n] = 0,
Vin] [
(3)
(2)
where 3 is the decay factor, Veh the firing threshold, and (.). refers to the Heaviside step function that determining spike generation: S[n] H [U[]] Vin)) when U [m]] 2 Vin, otherwise remains
inactive (S[n] = 0). The charging process of the LIF neuron is determined by f(-):
f(V/n 11, [m)) / 1] + (1 [m]
(4)
At each timestep n, the current membrane potential [[n] is updated by integrating the time-domain signal [m] corresponding to input data intermediate operations like Conv and MLP. If U [m]
exceeds the threshold Vín. the neuron fires spike (S[n) = 1). V [n]] records membrane potential over
time given the decay factor 3 and output spike activity. If the neuron does not fire, then V'n] Um]. [m] Notably, the LIF model simplifies to integrate-ann and-fire (IF) neurons when eliminating the membrane potential decay process between timesteps. Its charging process be formulated as:
f(V/n 1), [m]) V [n- 1] [[2]
(5)
2.2 Spiking Neural Networks
Drawing inspiration from biological neurons, SNNs extend conventional ANNs by incorporating temporal dynamics and discrete spike-based communication (5). Leveraging this spike-driven mechanism, neuromorphic chips implement computation through event-driven spike routing and accumulation, which substitutes energy- mnergy-nntensive matrix-vecto multiplications (7.6.. This facilitates high parallelism, scalability, and exceptional power efficiency, with power consumption typically in the range of tens to hundreds of milliwatts 21).
Recently, the development modern SNNs, c.g., Spiking Transformer, has demonstrated both attractive performance and reduced energy consumption 88 9 22). Spikformer [8] pioneered the baske-based self-attention mechanism called Spiking Self Attention (SSA) that utilizes sparse spike- form Query, Key, and Value vectors to eliminate the need for energy-intensive softmax operations. Following its success, many works endeavor to enhance Spiking Transformers by adapting advanced devising complicated spike coding mechanisms to reduce representation error (e.g., tolti--trrehold /25/ multi-spike neurons [26] (7]).
ANN Transformer architectures 23 24
Here, we instead address fundamental question: What truly limits SNNs' performance compared to ANNs? Our investigation reveals that the answer lies frequency properties - specifically, that spiking neurons function as low-pass filters, impeding the propagation of freuurry detail
3

---

within the network. In addition to theoretical proof, we probe the importance of information through our Max-Former, which features two frequency-enhaneinn operators: Max-Pool in patch embedding and DWC in place of early-stage self-attention. We further validate this principle convolutional architectures with our proposed Net-Re
3 Methods
In this section, we first present theoretical analysis of the frequency properties of spiking neurons. We show that, although the raw output spike trains of spiking neurons appear spectrally all-pass due to their impulse-shaped spike waveform, the resulting nigh-frequency components are merely superficial and cannot be propagated through the network. fact, spiking neurons act as low-pass filters at the network level. This is fundamental problem that has been overlooked in previous works. Building on this insight, we probe the importance of high-frequency information in SNNs through Max-Former, which strategically employs high-pass operators (Max-Poo Mool and DWC) to details and avoid feature degradation.
restore
Activation (Time)
Activation (F req.)
Weighting (F (er.).
1.0
x(t)
[(()
[XII]
r(t)
[()
Log Magnitude 6
(*([]]
Log Magnitude
Amplitude
0.5
s(t)
(f)
S
0.0
4
-0.5
0
0.00
0.01
0.02
200
400
200
400
Time (s)
Frequency (Hz)
Frequency (Hz)
(a)
(b)
(c)
Figure 3: frequenyy analysis of ReLU and spiking neurons. (a) Time-domain signals of input
al() 3 (2i 100() + sin(2 200() + sin(2x :300)) (blue), ReL.U-roceeseed: r(t) (red), spiking output of LIF neuron with the 3 = 0.25: s(t) (green). (b) Fourier analysis of x=t). r(t). and s(t). (c) Fourier analysis of linear transformed (CONV/MLP) activations, where ReLU expands the frequency bandwidth of the input signal, while the spiking neuron shows high-frequency attenuation.
3.1 Spiking Neurons are Low-pass Filters
We begin with an intuitive time-frequency analysis using an input (t)) = (sin 100t) + sin(2m sin -300f)) shown in Figure The results reveal three key observations: (1) In the time domain, the ReLU output (r(t)) perfectly tollows (t) > O while spiking neurons selectively respond 100Hz (Figure 3 (a)): (2) However, the spiking outputs' spectral response IS(() still appears nearly all-pass, which contradicts the low-frequency behavior observed in the time domain. These spurious high-frequency components actually arise from the impulse-shaped spike waveform itself rather than from genuine (Figure 3 (a-b)); (3) The waveform-induced high-frequency components cannot network- level low-pass behavior. When considering the whole process from input - activation + linear transform, ReLU expands the frequency bandwidth of xz (t) (28) whereas the spiking neuron exhibits strong high-frequency attenuation (Figure (c)).
be propagated across layers, resulting
We first examine the charging process of spiking neurons to theoretically analyze their frequency- selective properties. Given Equ. (3) and Equ. (4) this can be formulated as:
V [n]] = 3V [n 1] + (1 B) (m))
Applying the Z-transform with 2(V(n [r- - 1]] = = V (=) yields:
V[=] -== V[ + (1 [[]]
which can be rearranged to formulate the transfer function from input current to membrane potential:
V(=) = 1 B2-1
1-3 3
H(2) = [(2)
0 < 5 <1
(8)
Equ. 8 exactly the form of a first-order infnite-impulls-rspoonee (IR)) low-pass filter with
single pole at B. Accordingly, as 3, i.., the pole, approaches 1, LIF neurons exhibit stronger

---

low-frequency selectivity. Notably, the decay factor 3 correlates to the membrane time constant T as z. with T ranges from to +o0, a smaller yields a smaller B. From intuitive standpoint, a shorter time constant allows the membrane potential to respond within narrower temporal windows, rendering the neuron more sensitive to higher-frequency inputs.
From the individual neuron to the network-level information transmission, the average membrane potential is positively and consistently correlated with the possibility of spike firing during operation. We approximate the inherently nonlinear spike-generation process as linear around the firing threshold Vin. Denoting the firing rate by fr(V), we define the local gain and approximate the Z domain spike train (z) by:
a fr
S(z) = kV(2).
(10)
av V=
When output spike trains are weighted by causal synaptic kernel w(n]] the Z-transformed output current (y/n] w(n] s(n)) can be denoted Y(2) = ((2) S(z). The overall input-to-output
transfer function is obtained by combining this with ( and (10p:
(=))) = (()) T(=)
1-8
= S(=) = l (=)
(11)
1 1-1--
The first-order low-pass IIR characteristics of H(2) make the system Y(z) inherently favor low- frequency signal components, regardless of whether the synaptic kernel W process S(2) changes the gain or phase response. The low-pass term ( (1-32-1 ) further amplifies the system's s frequency selectivity when the process (²))) is cascaded L times (layers). The complete formula as follows:
((2) or the spike coding
(=) (=(=)
II (.((s) (:())) [()]
(II ke (.()))
(1--6--
( 32-1
1 1-8
(12)
T(=)
:=1
In the special case of the non-leaky IF neurons, which obey the charging process in (5) H()) is formulated as:
H(z)
(13)
This corresponds to an idcal discrete- time low-pass filter with a pole at 2 = 1, which can yield a consistent conclusion with previous analysis.
3.2 Max-Former
It remains unclear whether high-frequency information is truly important for SNNs and whether restoring can improve performance. Therefore, we systematically investigate the low-pass filtering characteristics of spiking neurons through Max-Former. To decouple frequency effects from model complexity, we: (1) replace self-attention with high-frrequncyy frequuency-preeeeving DWC in the early stages, and (2) add Max-Pool in patch embedding to compensate spiking neurons' low-pass preference. Notably, compared to the quadratic computational complexity of self-attention. DWC and Max-Pool only require linear complexity relative to the sequence length and are more parameter-effiiient. We consistently adopt the LIF neuron model throughout this work.
3.2.1 Overall Architecture
Figure 4 illustrates the overall framework of Max-Former. The architecture consists of 3 stages H W H W. 16 TE
with T. T. x 5 and H W tokens respectively, where H and W denote the height and width of the input image. Critically, MaxFormer processes information through discrete spikes over time. This spike-driven computing paradigm supports two types of input:
(1) Event Streams: Asynchronous events [z. y, pl containing spatial coordinates (x, y), times- tamp f, and polarity are converted to event frames through temporal binning. Given original resolution dt, and target dt adt,, events aggregated over consecutive bins:
a(t+1)-1
1
Σ Amat
S& E G 2xhx
(14)
where Sa denotes raw event data. The whole process denoises raw events and converts them to frame sequences target temporal resolutions.
(2) Static Images: Conventional images are converted to spike sequences by: 1) Repeating static frames T times, 2) Encoding pixel intensities to spikes using spiking neurons. The resulting input Embed((4e)7 (1e) which contains identical information across timesteps.
formulated as: 1 Spiking Embed(
5

---

D,IxcCx #x W
Dex "x W
Dj:Tx4CX H W
(a)
16
S.DW CONV
S-DW DOWC
Dx:Tx W
Embedding
Embrodinn
Patch Embeddang
Input Images
Palch
BN
Patch
BN
S-MLP
SSA
S-MLP
Classilication Head
(b)
Embed-Orig
Embed-Max
Embed-Max+
Embed
Max-Embed
Embed
Tax-Embed
Max-E Embed
CONV
BN
LIF
CONV
BN
MaxP
Embed
Embed
Embed
Max-F mbed
Max-Embed
Embed
Figure 4: (a) Overview of Max-Former: we restore high-frequency signals by using lightweight DWCs instead of self-attention in the early stages. Following the hierarchical design of [29], Max- Former adopts 3-stage architecture. D,: feature dimensions of stage-i. (b) In Max-Formen patch embedding stage, propose three configurations (Embed-orig, Embed-Mat and Embed-Maxe to enhance high- seggffreueency components.
3.2.2 Patch Embedding
To transform the imput into a tokenized representation, given imput {S} E RTxCXH*W the process of patch embedding is formulated as:
Y (G))) G2((5)), Y YE RTxC'X'
(15)
where C² = 2C, and H' [H/P], W" = [W/P] with the patch size P = 4. To address spiking neurons' inherent frequency preference, we present three patch embedding configurations as shown in Figure 4b:
Embed-Orig 92) = (Embed, Embed)
(16)
Embed-Max 92) (Max-Embed, Embed)
(17)
Embed-Max+ :G1, G2) (Max-Embed, Max-Embed)
(18)
where: Embed = (L.F- - BN], and Max-Embed -L.F- BN MaxPool).
3.2.3 Token Mixing
In Transformers, lower layers typically require more high-frequency details, while higher layers benefit from more global information [30 31]. Like biological vision, high-frequency details enable early stages learn low-level features while progressively building local-to-global representations. Accordingly, we replace carly-stage self-attention with DWC to preserve high frequencies essential for local feature learning. Given input embedding Y RTxCx HXC the spiking DWC defined as:
ZY(Y = LIF( Σ Wes Y.D)
(19)
JEN(s)
where 12))) denotes the local neighborhood of position i, We, represents learnable convolution weights for channel c, and Xe, Ze x is the input and output slice for channel c. For the final stage, we implement token mixing via Spiking Self-Attent (SSA) [8. The SSA computation follows:
L=F)(B)(YW)) Z (Q,
(20)
SSA(Q, V) = LIF(QK"V s)
(21)
where Q K, V x are spike-fore form tensors generated by learnable linear layers, is scaling factor. SSA eliminates floating-poin multiplications, ensuring spike-driven compatibility.
3.3 Membrane Shortcut
Residual learning and shortcuts enable very deep networks to be trained effectively by providing identity shorteuts that preserve information flow and mitigate vanishing gradients [32]. For SNNs, crucial consideration is maintaining compatibility with the spike-driven computing paradigm throughout all operations. As Fig. shows, existing SNN shorteut implementations fall into three categories: (1) Vanilla Shortcut
33 (2) Pre-Spike Shortcut (34), and (3) Membrane Shortcut (35).

---

Vanilla Shortcut
Pre-S Spke Shortcut
Membrane Shortcut
Membrane
Spiking Neuron
Spiking Neuron
pike
Spike
Spiking Neuron
CONV-BN
CONV-BN
CONV-BN
M Membrane
piking Neuron
Membrane
S pike
Spiking Neuron
Spiking Neuron
pike Membrane)
pSpike Spike)
SN(Membrane +Membrane)
Figure 5: Shortcut connection in SNNs. (Lefft) Vanilla Shortcut that combines spike and membrane potential. (Middle) Pre-Spike Shortcut that adds spike signals before neuron charging. (Right) Membrane Shortcut that directly connects membrane potentials, ensuring identical potential mapping while strictly preserving the spike-driven computing paradigm throughout the network.
The Vanilla Shortcut scheme
directly connects spikes (binary) to membrane potentials (continu- ous), leading to a distribution mismatch that inherently violates the identity mapping principle. The Pre-Spike Shortcut [3]] adds spike signals before neuron charging, resulting in summation values that range from to 2. This disrupts the binary spike representation and spike-driven flow in SNNs.
In this work, we adopt the Membrane Shortcut 35 mapping by directly connecting membrane potentials while maintaining binary spike outputs that remain inherently compatible with spike-driven computation. Unlike Vanilla Pre-Spike Shortcuts, this approach ensures both mathematical consistency with residual learning principles and seamless compatibility with spike-driven operations. We provide detailed analysis of its impact model performance and energy costs in Appendix
for its dual advantages: it preserves identity
4 Experiment
As shown Figure 6 Max-Forma restores feggufrryeecyy information by grafting the merits of rregeecc-eehaancinn operators: Max-Pool and DWC. To empirically probe the importance of high- frequency information in Spiking Transformers, we evaluate Max-Former through comprehensive experiments static datasets (CIFAR-10 10 (36. CIFAR- 100 377 and ImageNet (38) ) and neuromor-
phic datasets (CIFARIO-DVS [39, DVS128 Gesture (40)), with architecture configurations detailed in Table I In addition, design Max-ResNe to further investigate the effect of high-frequency restoration in convolutional architectures (model implementation is detailed in Appendix A.3).
Experimental settings and energy analysis methods are detailed in Appendix A and B
Spiking Neuron
Max-Pooling
Depth-Wise Conv
Self-Attention
Figure Fourier spectrum of Spiking Neurons, Spiking Max-Pool, Spiking Depth-Wir Wiith-WWie Convolution and Spiking Self-attention.
Table Max-Former architecture configurations for different classification tasks. Notation DWC-N denotes depth-wise convolution with kernel size N x N. For block settings: CIFAR-10/100 3 stages (1/1/2 blocks); ImageNet: stages (1/3/7 blocks); Neuromorphic: 2 stages (1/1 blocks).
Stage
Stage
Stage
Dataset
Patch Embed
Token Mix
Patch Embed Token Mix Patch Embed
Token Mix
Cifar10 [36 /10
Embed-Orit Orig
Identity DWD-7
Embed-Man
DWC-3
Embed-Man Embed-Max
SSA
ImageNet 38
Embed-Orig
Embed-Maa
DWC-5
SSA
Neuromorphic 39 40
Embed-Masee
DWC-3
Embed-Max
SSA

---

4.1 Results on CIFAR and Neuromorphic Datasets
As shown Table 2 Max-Former delivers performance improvements across both static datasets (CI- FARIO/CIFARI00) and neuromorphic datasets
Notably, for CIFARI0/100 classification, its first stage only uses identity mapping for token mixing (Table yet still attains attractive results. Max-Former achieves 97. 97.04% accuracy CIFARIO 10 with only 6.57M parameters at T=4, surpassing Spikformer (95.51 35551% %, 9.32M), S- S-Transformer (95.60 60%, 10.28M), and QKFormer (96.18%, 6. 6.74M). Similarly, on CIFARION Max-Former attains 82.65% aceuracy, significantly outperforming Spikformer (78.21% %). S-Transformer (78.40%). and FKrmme (81.57%).
Max-Formet and QKFormer share similar hierarchical architecture, though QKFormer originally employs pre-spike shortcuts (24). For a fair comparison, we additionally implemented QKFormer with the Membrane Shortcut (denoted MS- IlS-QKFormer in the table) using identical training configurations. Max-Forma still outperforms MS- MS-QKSSQOOO Former by 0.2% on CIFAR 77.004% vs. 96.84%) and by Mofoormmmss-O-o
81.57%), while requiring slightly fewer parameters (6.57M vs. 6.74M). For neuromorphic datasets, Max-Former maintains this performance advantage. On DVS128, achieves 98.6% accuracy, matching MS-QKFormer with the membrane shortcut. On 10-DV, Max-Former reaches 84.2% accuracy, exceeding MS-QKForrmrr (82.3%) by 1.9% and surpassing other spike-driven models like S-Transformer (80.0%) and a SWformer (83.9%).
1.08% CIFAR 100 (82.65%
Table 2: Performance Comparison CIFARIO 36). CIFARI00 37. DVS128 40 and CIFARIO-
DVS [39]. Param.; Parameter (M); Acc.:Top-1 Accuracy (%); T: Simulation Timestep. "Models trained from scratch using identical configurations.
Method
CIFARIO
CIFARI00
DVS128
CIFARIÓ-DVS
Membraae
Param,
T
Acc.
Param.
T
Acc.
Param.
Acc.
Param.
T Ace.
Shortcut
ResNet-19 (ANN)
24
12.63
94.97
12.63
75.35
Max-Former (ANN)
6.57
96.82
6.60
82.41
Spikformer [
9.32
4
95.51
9.32
78.21
2.57
16
98.3
2.57
16
80.9
S.Transformer
2.57
SWfomme
19
7.51
4
96.10
7.51
79.30
2.05
16
83.9
QKFormer 24
6.74
4
96.18
6.74
81.15
1.50
98.6
1.50
84.0
MS-QKFomer
6.74
96.84
6.74
81.57
98.6
1.50
82.3
Max-Former*
6.57
4
97.04
6.60
4
82.65
1.45
16
98.6
1.45
16
84.2
Table 3: ImageNet performance comparison. Notation: A2S: ANN-1o-SNN conversion; Model-L-D:
models with L blocks and D channels. Input resolution: 224x224, "Identical training configurations.
Methods
Type
Architecture
Param
Power
Time
Top-1
Membranc
(M)
(mJ)
Step
Acc (%)
Shortcut
ANN
ViT-L/16
304.3
80.96
79.70
DeiT
ANN
DeiT-B
86.6
80,50
81.80
PVT
ANN
PVT-Large
61.4
45.08
81.70
MST
A2S
Swin Transformer-T
28.50
512
78.51
Spikformer-8- 16.81
7.73
70.24
Spikformer S
SNN
Spikformer-8-
66.34
21.48
74.81
Transformmr--- 16.81
3,90
72.28
S S'Tansformer
41
SNN
S- Transffommr----Pe
66.34
6.10
4
76.30
Meta-Spikffrmer
45
SNN
31.3
7,80
75.4
31.3
32.80
77.2
SWformer
[19
SNN
SWformer-S 8-512
27.6
5.08
75,43
HST- HST-10-34 10-384
16.47
15.13
78.80
Former
24
SNN
HIST- 10-768
64.96
8.52
81.69
IIST 10-3-40-884
16.47
5.52
4
76.48
MS-QKFormer*
SNN
HST- 10-768
64.96
6.79
77.78
Max- 10-384
16.23
4.89
4
77.82
Max- 10-512
28.65
2.50
75.47
Max-Former"
SNN
10x-10-512
28.65
7.49
4
79,86
Max-10-768
63.99
5.27
78.60
10--10-768
63.99
14.87
4
82,39
8

---

4.2 Results on ImageNet Classification
performance on ImageNet classification, demonstrating its effectiveness for complex visual tasks. Max-Former-10-777 (T=4) achieves 82.39% accuracy (+7.58% over Spikformer) with 30% lower energy (14.87mJ vs 21. 21.48mJ), despite using only lightweight for early- stage token mixing. It also outperforms the ANN-to-SNN MST model (78.51 %) that requires 512 timesteps. Training/inference speed and memory usage are analyzed in Appendix
Thowes
Our analysis focuses on models using the Membrane Shortcut, which eliminates the energy-ineffiieen ternary spike transmissions ({0, 1, 2) in Pre-Spike Shortcut) while maintaining full compatibility with standard neuromorphic hardware without additional hardware overhead (see Appendix D). For fair comparison, we implemented membrane potential variant of QKFormer (denoted MS- MS--QFormmer) where MS- ASFQmerrmmer shows 64% lower energy (5.52mJ vs 15.13mJ) in HST- 10-344..
Max-Former-10-365 (16.23M, T=4) achieves 77.82% accuracy, outperforming MS-QK MS-QKFormer- (66.34M, 76.3%), and Meta-Spikformer (31.3M, 77.2%). For energy efficiency under identical settings, Max-Former- 10-384 consumes 4.89mJ, signif- (5844(5.22m)) S-Transformer(6. 10mJ), and Meta-Spikfformer (32.8mJ). Compared to conventional ANN models, Max-Former demonstrates concrete advantages in energy efficiency while maintaining competitive accuracy. Specifically, when compared to PVT- Large (a representative hierarchical ANN), (82.34% vs. 81.70%) with 67% lower energy consumption (14.87mJ vs. 45.08mJ.. These results confirm the importance of figh-frequency information in Spiking Transformers: replacing energy- intensive self-attention with lightweight DWC in early stages actually produces hetter performance.
10-384 (16.47M, 76.48%), S-Transformer-8
icantly lower than MS- MS-QKFormer-
10-768 (T=4) achieves comparable accuracy
Table 4: Ablation of Batch-Embedding/ Token-Mixing Strategies on CIFAR100 and CIFARIO-DVS.
CIFAR100
CIFARIO-DVS
Patch Embed
Token Mix
Acc (%)
Patch Embed
Token Mix
Acc (%)
Orig/Max/Max
Identity/DWC-3/SSA
82.65
Max+/Max
DWC-3/SSA
84.2
Orig/Orig/Orig
Identity/WCC3/SSSA
81.63
Orig
DWC-3/SSA
79.2
Orig/ Orig/Max/Orig
Tdentity/DWC-3//SS
81.88
Orig/ Max
DWC-3/SSA
81.5
Orig/Max/Max
Identity/ddentity/SSAA
81.28
Max+/Max
DWC-1/SSA
81.2
Orig/Max/MMa
Ideniit//DCC-5/SSA
82.02
Max+/Max
DWC-5/SSA
82.7
Ori//Max/Max
DWC-7/DWC5S/SSA
82.42
Max+/Max
DWC-7/SSA
82.1
Orig/Max/Max Orig/Orig/Orig
SSA/SSA/SSA SSA/SSA/SSA
82.23
Max+/Max
SSA/SSA
83.9
81.43
Orig/Orig
SSA/SSA
79.8
4.3
Ablation Study
We provide direct evidence for the critical role of high-frequency information in Spiking Transformers through in-depth ablation studies.
(1) Ablation on Patch Embedding Strategies: Proper patch embedding strategies help to unlock the performance limit of Spiking Transformers. On CIFAR 100, changing Max-Former's patch embedding strategy from the proposed Limbed-Orig/Emne Orig/mmedd-ax/EEmeeer Max/Embed-Maal Max to the default Embed-Orige gaabed-Orrg/Embeee Orig/Embed- -rig configuration reduces performance from 82.65% to 81.63%. Neuromorphic datasets exhibit stronger dependence on high-frequency components. On CIFAR 10-DS, replacing the first stage's Embed-Max- with Embed-Orig causes significant performance drop from 84.2% to 81.5 11.5% Incorporating feehfrryyency information through patch embedding proves effective. When purely using SSA token mixing, optimizing the patch embedding strategy can improve accuracy by +0.8% on Cifar 100 and +4.1% % on IVVSS. highlighting its critical role in spiking architectures.
(2) Ablation on Token Mixing Strategies: Max-Formee further restores figh-frequency information through the token mixing strategy. Despite SSA taking higher energy/parameter costs, Max-Formed achieves better performance by simply replacing early-stage self celf-attention with lightweight DWC. On 000, this substitution leads to +0.42% performance gain (All SSA: 82.23% vs. Max-Former: 82.65%). On CIFARIO-DVE Max-Former with hybrid token mixing (DWC-3 + SSA) achieves 84.2% accuracy, outperforming the full-SSA variant (83.9%) by 0.3%. The kernel size selection for high- frequency preservation also of great significance. Larger kernels (DWC-5/7) degrade performance (-0.63% CIFAR100, -2.1% CIFAR (FARRO-DVS) due excessive feature smoothing. Insufficient filtering

---

with DWC-1 also underperforms (81.2% vs 84.2% on CIFARIO-DVS), This highlights the necessity of balancing high-frequency and low-frequency components in Spiking Transformers.
Overall, Max-Former's s performance gains stem from effective high-frequency information propa- gation, independent of parameter count, which is evidenced by: (1) Max-pooling patch embedding Max++) consistently outperforms original versions despite similar parameter budgets. (2) Larger DWC kernels (DWC-5/DWC-7) increasing parameters but degrading accuracy (-0.63% on CIFARI00 -2. % on CIFARIO-DVS vs Max-Former with DWC-3). See Appendix D for visualizations and E for limitations discussion.
4.4 Generality across Convolutional Architectures
We further extend the effectiveness of Max-Forme to convolutional architectures by proposing Max-ResNet. The key modification in our Max-ResNet lies in the inclusion of only two additional max-pooling operations compared to MS-ResNet [46]. The detailed implementation of Max-ResNe is provided in Appendix AA.3] Training settings are listed in Appendix A
information is essential for SNNs. As shown in Table Max-ResNet achieves remarkable performance improvement over the baseline MS-ResNet, despite having identical model sizes. Specifically, with the block configuration [2, 2, 2. 2]. Max-ResNel improves the CIFAR- 10 accuracy by +2.41 % (from 94.4% to 96.81 ()) and the CIFAR- 100 accuracy by +6.48%. Similarly, under the [3, 3, 2] configuration, the accuracy increases by +2.25% and +6.65% on CIFAR-10 and CIFAR-100, 100, respectively.
In short, Max-ResNet sets oftee-ofthhearr benchmarks across convolutional baselines, with only a moderate model size and an extremely straightforward high-freug gigh-freueecc restoration strategy. Therefore, preserving high-frequency information is fundamental to effective feature representation in SNNs, regardless of architecture.
Table 5: Comparison of different ResNet architectures on CIFAR- 10 and CIFAR- 100.
Training Method
Block Config.
Params (M)
Time Step
CIFAR-10 Acc. (%)
CIFAR-100
Architecture
Acc. (%)
Knowledge
KDSNN-ResNet-18 47
22, 2,
11.22
4
95.72
78.46
Distillation
[2, 2,
11.22
4
94.40
75.06
MS-ResNet-18
46
Direct Training
[3,
12.50
4
94.92
76.41
MS- MS-RcsNct-34
46
[2, 22, 2, 2]
21.33
4
94.69
75.34
Direct
[2, 2, 2, 2] 3,
11.22
96.81
(+2.41)
81.54 (+6.48)
Max-ResNet-18
Training
12.50
4
97.17 (+2.25)
83.06 (+6.65)
5
Conclusion
This work challenges the prevailing assumption that binary activation constraints are the primary cause of SNNs' performance gap. Through theoretical analysis and empirical validation, we demonstrate for the first time that spiking neurons inherently function low-past filters the network level, resulting in the rapid attenuation of high-frequency components that critically degrade feature representation. We demonstrate that high-frequency information is crucial for effective spiking computation: Max-Former (63.99M parameters) achieves 82.39% top-1 accuracy on ImageNet, outperforming Spikformer (74.81% 66.34M parameters) by +7.58% while reducing energy consumption by 30% at comparable model size; Max-ResNet-116 further achieves state-of-the-art performance among convolutional baselines 97.17% on CIFAR-10 and 83.06% on CIFAR-I00. Notably, all these improvements are achieved through extremely simple modifications, even slightly reducing the overall model size. We believe this simple yet effective solution will motivate future research to explore the unique properties of SNNs, beyond the established practices in ANN studies.
6 Acknowledgements
This work is supported by the Major Science and Technology Innovation 2030 "Brain Science and Brain-like Research" key project(No.20) 2021100002220201111 405), the Guangzhou- HUSTT(G Joint Funding Program (Grant No. 2023A03J0682), the National Natural Science Foundation of China (Grant No. 62405255), GuangDong Basic and Applied Basic Research Foundation (No. 2023A15151 10679), and partially supported by the collaborative project with Brain Mind Innovation, Inc
10

---

References
[1] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 10819-10829, 2022.
[2] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):896-9) 911669912222 2023.
[3] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444,
2015.
[4] Chunming He, Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang, Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, and Xiu Li. Diffusion models in low-level vision: A survey, arXiv preprint arXiv:2406.111 11106. 2024.
[5] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784))607--61,, 2019.
[6] Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, and Gi-Joon Nam. Truenorth: Design and tool flow of 65 mw I million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 11110:1537711555 2015.
[7] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1)82299, 2018.
[8] Zhaokun Zhou, Yuesheng Zhu, Chao He. Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425. 2022.
[9] Ziqing Wang. Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, and Renjing Xu. Masked spiking transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1761-1771, 2023.
[10] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34:23426-23439 2021.
[11] Yufci Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im-loss: information maximization loss for spiking neural networks. Advances Neural Information Processing Systems, 35: 15:166616, 2022.
[12] Jianhao Ding. Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint arXiv:2105.. 1654, 2021.
[13) Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in neural information processing systems, 35:34451-34463, 2022.
[14] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xeor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pages 525-542. Springer, 2016.
[15] Edgar D Adrian and Yngve Zotterman. The impulses produced by sensory nerve endings: Part
3. impulses set up by touch and pressure. The Journal of physiology, (1):4465, 1926.
[16] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ANN: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pages 6316-6325. PMLR, 2021.
11

---

[17] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))227--233 1998.
[18] Jaldert Rombouts and Sander Bohte. Fractionally predictive spiking neurons. Advances in Neural Information Processing Systems, 23, 2010.
[19] Yuetong Fang, Ziqing Wang, Lingfeng Zhang, Jiahang Cao. Honglei Chen, and Renjing Xu. Spiking wavelet transformer. In European Conference on Computer Vision, pages 19-37. Springer, 2024.
[20] Wolfgang Maass. Networks spiking neurons: the third generation of neural network models. Neural networks, 10(9):165--1671, 1997.
[21] Arindam Basu, Lei Deng, Charlotte Frenkel, and Xueyong Zhang. Spiking neural network integrated circuits: A review of trends and future directions. In 2022 IEEE Custom Integrated Circuits Conference (CICC), pages 1-8. IEEE, 2022.
[22] Yuetong Fang, Ziqing Wang, Deming Zhou, Hongwei Ren, Shibo Zhou, and Renjing Xu. Dynamic token masking in spiking neural network. 2025.
[23] Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan. Yonghong Tian, Bo Xu, and Guoqi Li. Spike- driven transformer. Advances in neural information processing systems, 36, 2024.
[24] Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Okformer: Hierarchical spiking transformer using qk attention. arXiv preprint 16555:2403.16552, 2024.
[25] Zecheng Hao, Xinyu Shi, Yujia Liu, Zhaofei Yu, and Tiejun Huang. Lm-ht smn: Enhancing the performance of snn to counterpart through learnable multi-hierarchical threshold model. 2024.
arXiv preprint arXiv:2402.0044
[26] Ziqing Wang, Yuetong Fang, Jiahang Cao, Hongwei Ren, and Renjing Xu. Adaptive calibration: unified conversion framework of spiking neural networks. Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 1583-1591, 2025.
[27] Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling spike-driven transformer with efficient spike firing approximation training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[28] Christodoulos Kechris, Jonathan Dan, Jose Miranda, and David Atienza. De is all you need: describing relu from signal processing standpoint. arXiv preprint arXiv:2407.16556 2024.
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[30] Maithra Raghu, Thomas Unterthiner, Simon Korblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 14-12216- 2021.
[31] Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. In- ception transformer. Advances in Neural Information Processing Systems, 35:23495-23509, 2022.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Aision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 1-14, 2016, Proceedings, Part IV 14, pages 645. Springer, 2016.
[33] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11062-11070, 2021.
12

---

[34] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks, Advances in Neural Information Processing Systems, 34:21056-2106) 2021.
[35] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEFE Transactions Neural Networks and Learning Systems, 2024.
[36] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, (6(11):2278-23) November 1998.
[37] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
[38] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[39] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CIFARIO-DVS: An Seent-Stream Dataset for Object Classification. Frontiers in Neuroscience, 2017.
[40] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-basee gesture recogmition system. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7243-7252, 2017.
[41] Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike- driven transformer. arXiv preprint arXiv:2307.01694 2023.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
[44] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 068-578,777 2021.
[45] Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo XU, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next- gext--enneatton neuromorphic chips. In The Twelfth International Conference Learning Representations.
[46] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks towards deep residual learning, arXiv preprint arXiv:2112.066 1899954, 2021.
[47] Qi Xu, Yaxin Li, Jiangrong Shen, Jian K Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7886-7895, 2023.
[48] Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Timothée Masquelier, Yonghong Tian, and other contributors. Spikingjelly. httpp://gii //t.tggttttttttt com/ 2020. Accessed: 2023-02-21.
[49] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
13

---

[50] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[51] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[52] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[53] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecttre for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 188-222,,222 2016.
[54] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[55] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems,
2021.
[56] Philipp Weidel and Sadique Sheik. Wavesense: Efficient temporal convolutions with spiking neural networks for keyword spotting. arXiv preprint arXiv:2 2021.
[57] Ismael Balafrej. Soufiyan Bahadi, Jean Rouat, and Fabien Alibart. Enhancing temporal learning in recurrent spiking networks for neuromorphic applications. Neuromorphic Computing and Engineering, 2025.
[58] Ashish Gautam and Takashi Kohno. Adaptive stdp-based on-chip spike pattern detection. Frontiers in Neuroscience, 17: 2023.
(59) Shikuang Deng, Yuhang Wu, Kangrui Du, and Shi Gu. Spiking token mixer: An event- driven friendly former structure for spiking neural networks. Advances in Neural Information Processing Systems, 37: 21122888222-884 2024.
[60] Svea Marie Meyer, Philipp Weidel, Plank Philipp, Leobardo Campos-Macias, Sumit Bam Shrestha, Philipp Stratmann, Jonathan Timcheck, and Mathis Richter. A diagonal state space model on loihi 2 for efficient streaming sequence processing. In NeurIPS 2024 Workshop Machine Learning with new Compute Paradigms.
[61] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626, 2017.
[62] Zhaokun Zhou, Kaiwei Che, Wei Fang, Keyu Tian, Yuesheng Zhu, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer v2: Join the high accuracy club on imagenet with an snn ticket. arXiv preprint arXiv:2401.020200 2024.
[63] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, et al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2020.
14

---

A Analysis of High-Frequency Information in Spiking Transformers
To validate our approach, conduct experiments across three static and neuromorphic datasets. In this section, we first provide the detailed experimental setup used to obtain the results presented in our paper. We then perform additional analysis the importance of high-frequeng" information in Spiking Transformers, For complete parameter configurations, please refer to our public code repository
Table 6: Hyperparameters for image classification on different datasets.
Hyper parameters
ImageNet
CIFAR-10
CIFAR-100
Neuromorphic
Model Size
10-384 / 10-512/ 10-768
4-384
4-384
2-256
Epochs
200
400
400
106
Resolution
224 x 224
x
x
128
128
Batch Size
512 (8 x 64)
128
64
16
Optimizer
AdamW
AdamW
AdamW
Adam W
1.2 x 10 (T
Learning rate
1.50 10-3
1.50 10-3
6.00 10-3
1.35 x 10-3 (T
Learning rate decay
Cosine
Cosine
Cosine
Cosine
Warmup epochs
5 0.05
20
20
10
Weight decay
0.06
0.06
0.06
Rand Augment
0.5
9-n1 / 0.4
9-n1 / 0.4
Mixup
0.25 / 0../ 0.8
0.5
0.75
0.5
CutMix
1
0.5
0.5
Mixup prob
0.5
0.5
Erasing prob
0.0
0.25
0.25
Label smoothing
0.1
0.1
0.1
0.1
A.1
Experimental Details
Datasets: We evaluate Max-Formed through comprehensive experiments on static datasets (CIFAR- 10 (36). CIFAR-100 (37) and ImageNet (38) and neuromorphic datasets (CIFAR10-DVS (39).
DVS128 Gesture 440 The training and inference pipeline are implemented in SpikingJelly 488
Static Datasets: For static image classification, we evaluate three standard benchmarks. ImageNet- 1k (38] one of the most widely used datasets computer vision. It contains 1.28 million training, 50,000 validation, and 100,000 test images covering the common 1K classes. Both CIFAR-10 36) and CIFAR- 100 (37 include 50,000 training images and 10,000 testing images with 32x32 resolution. The main difference between them is that CIFAR-10 has 10 categories for classification, while CIFAR-100 has 100 categories.
For event-based vision tasks, we evaluate two standard benchmarks. CIFARIO-DVS [39] is an event-based version of the CIFAR-10 dataset. created by capturing moving image samples using the Dynamic Vision Sensor (DVS). It includes 10,000 event-based images (128x 128 pixels) spread across 10 classes, with 9,000 samples for training and 1,000 for testing. The DVS128 Gesture dataset 44 contains 1,342 event-based recordings of 11 different hand gesture types performed by 29 people under 3 different lighting conditions. Each gesture recording lasts about 6 seconds average.
Neuromorphic Datasets:
Hyper Parameters: Our training scheme mainly follows 24 and 41 Specifically, Mix [4)]
CutMix [50] and RandAugment [51] are used for data augmentation. The models are trained using W optimizer [52) with the weight decay of 0.05 for ImageNet- 1K classification tasks and the weight decay of 0.06 for all other datasets. Label Smoothing [53] is set O.I. Detailed training
15

---

hyperparameters are shown in Table For our ImageNet experiments, we used 8 NVIDIA A30 GPUs train most models. However, for the MaxFormer-10- 10-5ee (T=4) and MaxFormer- 10-768 (T=4) models, we used 8 NVIDIA H20 GPUs instead. For the smaller datasets (CIFAR10, CIFAR100, DVS128 Gesture, and CIFAR IO-DVS), we used a single A30 GPU for training.
A.2 Impact High-Frequency Information on Model Performance
Table 7: Patch embedding and token mixing schemes on CIFAR- 100. Acc.:Top-1 Accuracy (%). DWC-N: spiking depth-wist wise convolution with kernel size x N. SSA: Spiking Self-Attention.
Patch Embed
Token Mix
Acc. (%)
Avg-Pool/Avyy-Poll/Ay--Po
76.73
(1) Ommbedd-nog
Mas-PPol/MaxxPPo//MaaPPPoo
79.12
Embed Onnfddddonnnibbddddee///eeeeeeer
Idemity/ldentity/deenity Ideniity/lenttiylldenttt
80.1
(2)
Embed Embbed/Dnn/ZimeeeeWWWW/EEeeee--
80.46
Avg-PPal/Avy--Pool/yyyPPe
77.61
Pool
79.78
(3) Emhhel-Orj/nnneeeMMM////neee---
Seenity/Maax-PPollddeeel
79,99
Ldenity/ldentttyMax-Pooo
80.12
SSA/SSA/SSA
82.13
DWC-3/DCC-/SSA
82.36
DWC-S/DWC-3/SSA
82,46
(4)
DWC-5/DWC-7/SSA
82.45
DWC-7/DWC-S/SSA
82,42
DWC-3/DWC-3/SSA
82.59
Idenity/IDWC-3/SSA
82.65
Emmbec-Orig/Enbbeed-aa//EEEnnnn
SSH-DNCC/SSA-DDCSSSBB-DDYYC
82.09
(5)
SS+-DWCC/SSA--WCC.SSS-DDW
82.56
Embed Haabed-Orig/Emmbed Dong/EEibbdd-Ori/EEmmbe Orig
SSA+DWCCSSS.NNNWCS//SADDD
82.73
Embed Oraeee/-OriiE/meed--ee
Maaeeldeedddd///eeeebbeeeee Max
SSA/SSA/SSA
82.13
(6)
Embed- Mww./EEhhcc-M/////eeed---a
SSA/SSA/SSA
79.98
SSA/SSA/SSA
78.78
Embed- Baabeed-rri/EEmeed-lle MMax/Ebbeddd IMax
SSA/SSA/SSA
82.02
(7)
Embed- Maaieedd-Ma///mbed---aa Max//Embed----e MMx
SSA/SSA/SSA
81.86
Embed Siinedd-rri//mhhed-ee Mawieeeeddd-MMM//e/eeeeeeee
SDSA/SDSA/SDSA
81.77
(8)
Embed-OOrg/Embeed Iaax/Emhedd Maxx
SDSA/SDSA/SDSA
81.52
In Table 7 we provide additional analysis on the impact of high-frequency information. We conducted these experiments CIFAR-100 (37], using the experimental settings detailed in Table We discuss
the ablation below according to the following aspects:
(a) Extra Max-Pool in Patch Embedding/Token Mixing:
High-frequency information plays a critical role in the performance of Spiking Transformer due to the inherent low-pass filter characteristies of spiking neurons. Experimental results shown in Table (1) reveal that strategically preserving these frequencies through max-pooling operations significantly enhances model accuracy, with 2.39% improvement when replacing average pooling with max-pooling across all stages (76.73% to 79.12% 12%). In Table (2), the performance of Spiking Transformer increases progressively when extending Embed-Mat from the last patch embedding blocks (80.1 1 (%) to include the middle block (80.46%).
However, excess geeh-frequency information will instead impair model performance. For instance, in Table (3), switching from using avg-pool for all token mixing to max- pool improves top-1 accuracy from 77.61 % 79.78%. Yet found even better results with more targeted setting: when employing max- max-pooling exclusively in the middle stage increases accuracy to 79.99%, while restricting it to only the last stage further pushes performance 80..2%.
This happens because spiking neurons act like low-pass filters that naturally reduce high- gigh-freuuenc components as information moves deeper through the network. Therefore, strategically adding back
16

---

-ree-rreyency components at specific points in the network is crucial for pushing the performance limit of Spiking Transformers.
In general, for the hierarchical spiking transformer, Embed-Max can improve performance when used for patch embedding, as shown in T (6-8). However, for non-hierarchical ones, although enhancing high-frequency components can in principle be beneficial, its actual impact is limited since patch embedding is only performed once throughout the entire process, causing the improvement to diminish in deeper layers. A more promising solution lies in optimizing token mixing, as discussed in [9].
(b) Spiking Transformers Benefit from High-Frequency Information:
In biological vision, high-frequency details help early processing stages learn elementary features,, which are then gradually built from local to global representations. Similarly, in standard non-spiking Transformers, the lower layers typically need more high-frequency details, while higher layers work better with global information. Spiking Transformers follows the same design philosophy, but with an important difference: they need additional rrequency-enhanein operations (e.g., maxx-pooling and depth-wise convolution) for restoring high-frequency information that would otherwise be lost.
In Table 7 (4), we show that, proper token mixing strategy can effectively restore high-frequenge information in Spiking Transformer, resulting significant performance gains. By replacing Spiking Self- Atff-Attention (SSA) with depth-wise convolution (DWC) for token mixing, we improved perfor- mance from 82. 13% to 82.65%. Importantly, this improvement does not come from adding more parameters/ computational burden. For example, the (dentity/DWC-3/SSS better than DWC-3/DWC-5/SS even though the former one has a lower computational cost. Our further experiments in 7 confirm that these findings hold true in the full-SSA network: restoring high-frequency components can significantly optimize the performance from 82. to 82.73% (+0.6%) Cifar100. The proper high-frequency enhancement strategy is essential to unlocking the full potential Spiking Transformers.
combination works 0.29%
A.3 Max-ResNe Implementation
As shown in Figure 7, Max-ResNet introduces only minor architectural change to MS-ResNet 446 all layers are replaced with Max-ResNet layers, while the first layer remains unchanged. Code implementation is available at httpss//ghhbb.comm/bllL/aaarre
ResNet Block
RaxxRe Block
Max-ResNet Layer
Max-ResNet Block
CONV
CONV
CONV
MaxPool
CONV
BN
LIF
BN
LIF
BN
LIF
BN
ResNet Block
ResNet Block
Figure Overview Max-ResNet. A single Max-Pool operation is added per block and layer.
B Energy Analysis
To estimate the theoretical energy consumption of Spiking Transformers, follow the methodology used in previous studies 88 41 19]. is worth noting that the batch normalization (BN) layers and linear scaling transformations that follow convolution layers be combined directly into the convolution layers themselves with added bias terms during deployment. Thus, common practice 54 19), the energy consumption of BN is typically excluded when calculating theoretical energy usage. For fair comparison, our work adopts the same strategy. In Spiking Transformers, energy consumption is directly proportional to synaptic operations (SOPs), which can be calculated as:
SOPs(!) fr T FLOPs(()
(22)
where represents specific block or layer in the Spiking Transformer architecture, fr refers to the firing rate of the input spike train for that particular block or layer, and T the simulation time step of spiking neurons. Assuming the
(MAC) and accumulate (AC) operations
17

---

are implemented on the 45mm neuromorphic chip described in 77, where each MAC operation uses EMAC 4.6pJ of energy and each AC operation uses ÆAC 0.9pJ, we can estimate the total energy consumption of a Spiking Transformer by adding up the energy used by all MAC and AC operations
across all layers:
M SOP SNN
Essn = EMAC x FLOP + EAC x SOPSNN Conv
(23)
FLOP represents the floating-poin operations in the first layer, which converts non-spike inputs into spike form for static image classification tasks. Since this layer performs floating- point computations, we estimate its energy consumption using EMAC. For all subsequent layers, which process spike data, estimate energy consumption using Exc. For mainstream non-spiking Transformers, the energy consumption is estimated through:
EANN = EMAC x FLOPs
(24)
C Comparison on Train/Inference Time and Memory Consumption
Table 8: Comparison of training/inference time and memory usage between OKFormer-10-766 and Mlux-Forme--i 10-768 models. All measurements were conducted with the simulation timestep of and the batch size of 32. MS-QKFormer indicates the QKFormer variant with the membrane shorteut.
Train
Train
Infer.
Infer.
Time (s)
Memory (MB)
Time (s)
Memory (MB)
QKFormer (64.96M, T=1, B=32)
0.214
18227
0.053
5000
Formerr (64.96M, T=1, B=32)
0.208
17496
0.048
4822
Max-Former (63.99M, T=1, B=32)
0.179
15431
0.044
4354
Max-Former delivers faster training and inference speeds while consuming less memory. We compared the performance metrics of QKFormer (24), its membrane shorteut variant (MS- (MS-)K)ormer), and Max-Former on ImageNet using 224 x 224 input resolution. All tests were conducted on CentOS 7.9 server equipped with the Intel Xeon Gold 6348 CPU (2.60GHz) and the Nvidia A30 GPU. As Table [ shows, when compared to MS- MS-QKFormer with the same hierarchical architecture and shortcut configuration, Max-Former reduces training time by 14% and both inference time and memory usage by 10%. Additionally, our results indicate that the pre-spike shortcut strategy used in the original QKForeer increases both processing time and memory demands.
D Residual Connections in Spiking Neural Networks
(a) Performance and Energy Tradeofl'::
The unique asynchronous nature of spike-based computation makes implementing residual connec- tions challenging. As a result, the research community of spiking neural networks (SNNs) has not yet reached consensus on the standardized residual learning approach, either in terms of algorithm or hardware implementation. While the main focus of our work is not related to residual learning, we want to offer a detailed comparison between the pre-spike shorteut (55) and membrane shortcut [4 the two most representative residual learning methods that emerged in recent years.
As explained in Section 3.3, the pre-spike shortcut [55] implements residual connections between spiking outputs, while the membrane shortcut (54) connects membrane potentials directly. In algorithm designs, membrane shortcuts have been reported to facilitate better performance in prior works 24 41 especially on small datasets. However, our findings indicate this advantage is not universal across all scenarios. As shown in Table [ the patch embedding stages of Spiking Transformers account for the majority of energy consumption. Consequently, the multiple patch embedding stages hierarchical architectures like QKFormer (24, while enabling efficient feature learning with fewer parameters, also come at higher energy usage. This makes the choice of shorteut scheme particularly impactful on overall energy efficiency. When processing ImageNet images at 224 224 resolution, the pre-spike QKFormer consumes three times more energy than
18

---

its membrane shorteut variant, reflecting the substantially higher SOPs required. Nevertheless, this increased computational overhead does translate notable performance gains (+2.32% accuracy when comparing QKFormer to MS-QKFormen
From the hardware perspective, implementing either shorteut type on neuromorphic chips is tech- nically feasible but presents significant challenges [5). Implementing the pre-spike shortcut [5].
specifically requires the chips to support multi-spike operations, as the ternary spike transmissions (values of O, 1, or 2) can occur. This results in either higher energy consumption or increased hardware complexity [58]. Yao et al. [4] proposed implementing membrane shortcuts through the addressing function that passes the membrane potential to corresponding neurons in subsequent layers for merging. While membrane shorteuts do strictly adhere to the spike-driven computing paradigm and could possibly be supported by standard neuromorphic hardware, transmitting membrane poten- tials can create substantial communication overhead, making practical implementation non-trivial. Many discussions on hardware deployment still advocate avoiding shortcuts as the current preferred approach [59] 60] However, given the critical role of residual learning in modern deep learning, avoiding shortcuts altogether is not sustainable long-term strategy for advancing neuromorphic computing.
In our work, we primarily compare with MS-QKFormer using the membrane shortcut to ensure fair comparisons. We hope the SNN community can establish consensus on standardized shortcut implementations in the near future, considering the significant impact of shortcut schemes on both energy efficiency and model performance.
Table 9: Energy consumption comparison (in mJ) across processing stages for QKFormer, QKFormer with membrane shortcut (MS-QKFormer), and our proposed Max-Former model. * Identical training configurations.
Stage 1
Stage 2
Stage 3
Total Energy
Model
Patch
Token
Patch
Token
Patch
Token
Classifier
MLP
MLP
MLP
Embed
Mix
Embed
Mix
Embed
Mix
QKFormer (16.47M, T=4, 78.80 78%8
1.26
0.16
0.41
2.68
0.35
0.77
2.97
2.71
3.81
0.006
15.13
MS-QKFomeer* (16.47M, T=4, 76.48%)
1.19
0.052
0.15
0.96
0.11
0.25
0.88
0.88
1.06
0.007
5.52
Max-Former" (16.23M. T=4, 77.82%)
0.41
0.02
0.17
0.89
0.01
0.36
0.91
0.96
1.16
0.001
4.89
E
Visualization
We demonstrate the GradCAM visualizations [61 of four Spiking Transformers with the membrane shortcut of similar size. Compared to Spike-Driven Transformer [41] and SWFormer [19]. the hierar- chical architecture used in both FKroreer 24 and our Max-Former allows for more precise focusing on target objects. Compared to MS-QMSQQQKKK ASS-QrFrrmer, Max-Former shows more concentrated activation patterns. For instance, the polar bear image, Max-Former completely skips the background and precisely focuses on the bear's ' key features (the head, rather than the outline or fur).
F Impact and Limitation
Our work serves the theoretical foundation for many existing architectural choices in Spiking Transformers. Specifically, in [41], the authors found that directly applying known practices of MetaFormer does not achieve good results in Spiking Transformers: employing average pooling operators to replace SDSA as the token mixer surprisingly results in substantial performance degrada- tion from 61.0% to 41.2%. Similar phenomena have been discussed in earlier works. For instance, Spikformer v2 [62] discovered that removing the max- max-pooling operator in the original Spikformer [8] leads substantial performance drop. while adding convolution layers (which act high-pass filters) in the patch embedding stage significantly improves the performance. Our work reveals the underlying principles of these architectural designs: spiking transformers need to enhance high- frequency components to alleviate the feature degradation caused by their inherent low-pass activation.
We are well aware that there still much space to be explored, and we hope that our Max-Former can serve as good starting point for future research. For instance, similar to (31. Max-Former requires manually balancing frequency components, which demands considerable expertise when adapting to different tasks. Incorporating direct frequency learning approaches like Fourier-based 63
19

---

Spike-Driven Transformer
SWFormer
MS-QKFormer
Max-Former
Figure 8: GradCAM visualizations 61 comparing four Spiking Transformers of similar size: Spike- Driven Transformer-8-55 41 (29 28), SWFormer-8-511 [19] (27. 6M). QKFormer 10-512 with membrane shorteut (MS-QKFormer) (29.08M), and our Max-Former- (28.65M).
or Wavelet-based 19 methods would offer more straightforward solutions. The main challenge. however, lies in developing efficient based frequency representations. Overall, we believe our work will inspire more future research advance neuromorphic computing through exploring the unique properties of spiking neurons, rather than expending excessive effort to adapt established practices from standard non-spiking neural networks.
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction reflect the paper's contributions and scope clearly.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in Appendix
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All the theorems, formulas, and proof' are clearly stated in the main paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: We provide experiment details in the Appendix A
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is available: https///ttthbbcoom/cc//Ma//ttto
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details have been revealed in the Appendix A
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be computationally expensive.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Appendix
9. Code of ethics
21

---

Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: This research every respect with the NeurIPS Code Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This research belongs to foundational research and tied particular applications.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits creators and mentions the license and terms of use for existing assets.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, or non-standard components.
22


Reviews:
[{"review_id": "UVpgdD4e18", "reviewer": "Reviewer_foAf", "summary": "Here is a **precise summary** in bullet points:\n\n1. Issue Addressed:\n\n   Spiking Transformers underperform compared to traditional deep networks due to information loss, especially of high-frequency signals, from their binary and sparse activations.\n\n2. Proposed Solution:\n\n   Introduce Max-Former, which enhances high-frequency feature retention via:\n\n     Max-Pooling in patch embedding.\n     Depth-Wise Convolution replacing self-attention.\n\n3. Evaluation:\n\n   CIFAR-100: Improves accuracy from 76.73% (Avg-Pooling) to 79.12% (Max-Pooling).\n   ImageNet: Achieves 82.39% top-1 accuracy, a +7.58% gain over Spikformer, with comparable model size.", "questions": "Refer to Strengths And Weaknesses session", "limitations": "Refer to Strengths And Weaknesses session", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "1. Lack of Theoretical Justification for Low-Frequency Bias in Spiking Neurons:\n   While the paper claims that spiking neurons preferentially propagate low-frequency information, there is no rigorous theoretical or empirical analysis provided to support this assertion. A formal frequency-domain analysis or signal decomposition study would significantly enhance the credibility of this claim.\n\n2. Insufficient Literature Review on High-Frequency Modeling:\n   The manuscript omits discussion of several **relevant works that address high-frequency representation in vision models**, such as:\n\n   *SVT (Scattering Vision Transformer)[1], which models spectral components using wavelet-inspired mixing,\n   * HiLo Attention [2], which splits frequency bands for efficient attention computation,\n   * SpectFormer and WaveViT, which are designed to efficiently encode high-frequency textures.\n\n   Including these would provide valuable context and distinguish the proposed method more clearly from existing approaches.\n\n3. Lack of Comparison with Contemporary Spectral Mixing Techniques:\n   The proposed Max-Former does not compare or contrast with similar spectral decomposition and frequency-aware channel mixing methods, such as:\n   * Monarch Mixing, which exploits structured linear transforms for spectral control,\n   * EinFFT used in SiMBA-TS[3], which mixes channels via frequency-domain transformations.\n\n   A detailed comparison is essential to understand how Max-Former is uniquely positioned among these recent advances.\n\n4. Limited Generalizability Discussion:\n   It remains unclear whether the proposed Max-Former architecture is generalizable to non-spiking architectures or other modalities such as image captioning or multimodal tasks. The paper should explicitly discuss whether the frequency enhancement mechanisms apply only to spiking transformers or more broadly across architectures.\n\n5. Unsubstantiated Claims About Max-Former’s High-Frequency Behavior:\n   The claim that Max-Former captures high-frequency components more effectively is not backed by sufficient evidence. There is no frequency-domain visualization, filter response plot, or Fourier spectrum analysis to demonstrate that the method indeed enhances high-frequency signals across layers or tasks.\n\n6. Missing Comparison with Strong Baselines:\n   Tables 2 and 3 lack comparative results with strong recent baselines, particularly:   SVT[1], HiLo Attention[2], EinFFT-based SiMBA[3].\n\n   These omissions make it difficult to assess whether Max-Former truly offers a state-of-the-art tradeoff between accuracy, frequency preservation, and efficiency.\n\n7. Potential Overlap with Existing Work:\n   A recent preprint available at [arXiv:2505.18608](https://arxiv.org/pdf/2505.18608) appears to tackle a similar problem space of frequency-aware modeling in spiking networks. The authors should clearly articulate how their contributions differ from or improve upon this work, including differences in methodology, theoretical insight, or empirical scope.\n\n[1] Patro, Badri, and Vijay Agneeswaran. *\"Scattering Vision Transformer: Spectral Mixing Matters.\"* NeurIPS 2024.\n[2] Fast Vision Transformers with HiLo Attention, NeurIPS 2022\n[3] SiMBA-TS: Simplified Channel Mixing and Mamba for Long-term Time Series Forecasting, ICASSP-2025", "was_revised": true, "final_justification": "The authors have provided a detailed rebuttal, but unfortunately, many of my original concerns remain unaddressed or insufficiently clarified. SO I would like to keep my rating as it is.", "rebuttal": "Thank you for your time in evaluating our work and providing feedback. However, we cannot fully agree with your opinions.\n\n### **W1 & W5: Lack of Theoretical Justification and Frequency Domain Analysis**\n\nWe understand your concerns about the analytical rigor in our work. We share your focus on reliability and have performed extensive experiments to validate our claims. **Indeed, our claims are supported by comprehensive empirical evidence (especially our frequency domain analysis) and theoretical proofs throughout the paper.**\n\nWe propose and preliminarily validate our hypothesis in **lines 42-57** of the main paper. The discussion is presented along with several **empirical findings**. **Figure 2** provides a comparative visualization of **Fourier spectra, relative log amplitudes, and Grad-CAM activations** for images processed by ReLU versus spiking neurons. This shows spiking neurons propagate low-frequency information more than ReLU neurons. To further verify our hypothesis, **Figure 1** shows that simply replacing the avg-pooling with max-pooling operators for token mixing in Spiking Transformers can lead to a striking (+2.39%) performance improvement on the CIFAR-100. The results confirm that the rapid dissipation of high-frequency components is one of the factors that cause the modest performance of Spiking Transformers. \n\nIn **Section 3.1**, we use an entire subsection to prove the low-pass filtering nature of spiking neurons. Our analysis starts with an intuitive **frequency analysis** (**Figure 3, lines 111-119**), which shows that while ReLU expands frequency bandwidth, spiking neurons exhibit strong high-frequency attenuation. We then present a comprehensive **theoretical analysis** (**lines 119-141**) of the frequency-selective properties of commonly used LIF and IF neurons.\n\nThe **Fourier spectra** of spiking neurons, spiking max-pooling, spiking depth-wise convolution, and spiking self-attention are visualized to verify that Max-Former's architectural design helps to restore high-frequency components (**Figure 6**). We finalize our conclusion with the success of Max-Former, which achieves 82.39% top-1 accuracy on ImageNet (7.58% higher than the Spikformer with a similar model size), and the ablation studies in **Section 4.3** and **Appendix A1.2**, which provide direct evidence for the critical role of high-frequency information in Spiking Transformers.\n\nWe always hope to support our point of view with sufficient analysis and promote the development of the field of spiking neural networks. We hope our response can address your concerns. \n\n### **W2&W3&W6: Insufficient Literature Review and Comparison with Baselines**\n\n**We need to clarify that our work is not related to high-frequency modeling based on spectral decomposition and is not aimed at proposing a better frequency learning method in spiking neural networks.** Instead, our Max-Former only add two lightweight high-frequency operators: additional max-pooling in patch embedding process and depthwise convolution to replace early-stage self-attention. We hope to eliminate the impact of factors such as computational complexity and receptive field, thereby better supporting our core argument: the performance degradation of spiking neural networks stems not only from binary activation but also from frequency loss. Therefore, we believe that including the works you mentioned would distract the readers from our main contribution.\n\nWe also agree that direct frequency learning approaches like Fourier-based or wavelet-based methods, would offer more straightforward solutions, as discussed in our limitation section (**Appendix A5**). However, the main challenge lies in developing efficient spike-based frequency representations. For instance, existing works on spiking neural networks have attempted to adopt time-to-value mapping for accurate Fourier transforms, but at the expense of high latency [1]. A more recent work in Spiking Transformers proposed leveraging the inherent sparsity and robustness of the wavelet transform to approximate the spiking frequency representation, but relies on uncommon negative spike dynamics [2]. \n\nWe have already compared with [2] in Table 2. In the final version, we will explicitly highlight frequency learning approaches as a valuable future direction for spiking neural networks and add the results of [2] to the comparison in Table 3.\n\n[1] “*Time-coded Spiking Fourier Transform in Neuromorphic Hardware*”. In IEEE Transactions on Computers, 2022\n\n[2]  “*Spiking Wavelet Transformer*”. In ECCV, 2024\n\n### **W4: Limited Generalizability Discussion**\n\nThis is indeed an interesting question worth exploring. However, the core idea of our work is to emphasize the importance of the low-pass filtering properties of spiking neurons for model design, rather than proposing a universal frequency enhancement strategy that works for all tasks and architectures.\n\nThe frequency enhancement strategy in MaxFormer may be less effective for non-spiking Transformers that do not experience the same high-frequency attenuation problem. Here, we provide additional experimental results on CIFAR-10/100:\n\n| Dataset | Architecture | T | Acc. (%) |\n|-|-|-|-|\n| Cifar10 | **MaxFormer(ours)** | 4| **97.04%** |\n| | QKFormer* | 4| 96.84% |\n| | MaxFormer(ANN) | NA| 96.82% |\n| Cifar100 | **MaxFormer(ours)** |4 | **82.65%** |\n| | QKFormer* | 4| 81.57% |\n| | MaxFormer(ANN) | NA| 82.41% |\n\nIt is noted that the ANN version of MaxFormer consistently underperforms the spiking MaxFormer on both datasets (96.82% vs 97.04% on CIFAR-10; 82.4% vs 82.65% on CIFAR-100). Compared to QKFormer*, MaxFormer(ANN) is slightly weaker on CIFAR-10 (96.82% vs 96.84%) but stronger on CIFAR-100 (82.41% vs 81.57%, +0.83%).\n\nRegarding other modalities, we need to point out that, for spiking neural networks, the field of image captioning or multimodal processing remains in early development stages, and there is no mature evaluation framework yet. However, we believe that empirical validation across modalities will be valuable future work.\n\n### **W7: Overlap with Existing Work**\n\nThank you for the reminder. We need to clarify that our work is entirely original. We have gone through the publication and noticed that the cited paper was submitted to arXiv **one week after** the conference submission deadline. Therefore, we believe a comparison with this work is not necessary."}, {"review_id": "Zs7K65TbBo", "reviewer": "Reviewer_i2a6", "summary": "In this paper, the authors analyze the Spiking Transformer in the frequency domain. Interestingly, they found that LIF tends to pass low-frequency information and block high-frequency information. This phenomenon has been verified using an intuitive example. I am no expert in frequency-domain analysis, but I found the conclusion interesting and valuable for this area. The final resulting model, called MaxFormer, is a straightforward modification of the existing Spiking token-mixer and achieves a significant performance improvement.", "questions": "1. How do you explain the performance improvement from the pre-spike shortcut or vanilla shortcut with the frequency information? \n\n2. Except for max-pooling, is there another way to increase the high-frequency information by modifying LIF?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n+ Strong support of low-pass filter nature in LIF neuron, based on the analysis in Section 3.1.\n\n+ Impressive performance improvement over average-pooling-based baselines. However, I did not verify the code's correctness. Given that the authors have provided the implementation of CIFAR100 models, I will buy the results for now. \n\n\nWeakness: \n\n- Throughout the paper, the authors keep talking about the difference between LIF and ReLU neurons and claim that the ReLU neuron is more than just a low-pass filter. It would be interesting to see the ANN version of MaxFormer (at least on CIFAR-100). The current table only shows some trivial ANN practices, which do not fully convince the readers.\n\n- I hope the authors can fully open-source their implementation in rebuttal/after submission, or clarify the reason that only the CIFAR-100 code is provided.", "was_revised": true, "final_justification": "The authors have addressed my concerns that high frequency information is only needed for SNNs. Based on this result, I believe this work has enough take-home information and I encourage the publication on NeurIPS.", "rebuttal": "### **W1: Comparison with the ANN baseline**\n\nThank you for your insightful feedback. We agree that it is more appropriate to compare the ANN version and spike-based version of Max-Former under the same architectural configuration and training hyperparameters. However, **as the distribution of attention scores in ANNs and SNNs is very different, directly adopting the ANN-version of spiking self-attention, e.g., replacing all spiking neurons with the ReLU activation, will cause gradient explosion.** We managed to provide additional experimental results on the ANN version of Max-Former by additionally scaling the q, k, and v matrices in the ReLU-based processing:\n\n| Dataset | Architecture | T | Acc. (%) |\n|-|-|-|-|\n| Cifar10 | **Max-Former(ours)** | 4| **97.04%** |\n| | QKFormer* | 4| 96.84% |\n| | Max-Former(ANN) | NA| 96.82% |\n| Cifar100 | **Max-Former(ours)** |4 | **82.65%** |\n| | QKFormer* | 4| 81.57% |\n| | Max-Former(ANN) | NA| 82.41% |\n\nIt is noted that the ANN version of Max-Former consistently underperforms the spiking Max-Former on both datasets (96.82% vs 97.04% on CIFAR-10; 82.4% vs 82.65% on CIFAR-100). Compared to QKFormer*, Max-Former(ANN) is slightly weaker on CIFAR-10 (96.82% vs 96.84%) but stronger on CIFAR-100 (82.41% vs 81.57%, +0.83%).\n\nThis can be explained from two perspectives. First, since ANN Transformers do not experience the same high-frequency attenuation problem, the frequency enhancement strategy in Max-Former may be less effective. Second, the spiking self-attention module may not be well-suited for token mixing in ANN Transformers.\n\n### **W2: Code Implementation**\n\nWe will fully open source our implementation in the final version since we cannot upload any files during the rebuttal phase. The CIFAR-100 model file we provided actually contains the implementation of all the patch embedding strategies/token mixers used in this article. We will further organize our code to facilitate reproduction.\n\n### **Q1: Shortcuts in Spiking Neural Networks**\n\nFor pre-spike shortcut, **we found this cannot be explained simply from the frequency domain perspective as discussed in **Appendix A4(a)** of our paper.** The pre-spike QKFormer requires **3x** more synaptic operations than its membrane shortcut variant when processing the same images. **We feel that a comparison between the two is unfair given the significant difference in computational cost.** Therefore, throughout our paper, we perform analysis or experiments under the premise of binary activation transmission. Likewise, our Max-Former is primarily compared with the membrane shortcut variant of QKFormer (denoted as QKFormer*) that adheres to the binary activation rule.\n\nFor the vanilla shortcut, as explained in **lines 188-190** of our main paper, it can lead to distribution mismatch and thus limit the model performance. Therefore, this shortcut scheme has not been widely adopted in spiking neural networks and is rarely used in recent studies.\n\nBesides, we need to point out that replacing the membrane potential shortcut with the pre-spike shortcut does not always result in performance improvements. As presented in **Table 2** and **Table 3**, compared to QKFormer with pre-spike shortcut, QKFormer* that employs the membrane shortcut achieves higher performance on CIFAR10/CIFAR-100/DVS128, while showing weaker performance on CIFAR10-DVS and ImageNet. The reasons behind this will be a valuable direction for future research.\n\n### **Q2: LIF for High-Frequency Information**\n\nBased on our theoretical analysis presented in **lines 119-141** of the main paper, **reducing the value of $\\beta$ in **Equ. 6** can enable LIF neurons to capture more high-frequency information.**\n\nTo illustrate this more intuitively, we need to note that in our formula, $\\beta = 1 - \\frac{1}{\\tau}$, where $\\tau$ represents the time constant of the membrane potential, which ranges from in $1$ to $+\\infty$. Therefore, $\\beta$ becomes smaller the closer $\\tau$ is to $1$, and a smaller time constant $\\tau$ causes the membrane potential to respond to narrower time windows, making the neuron more sensitive to higher frequency signals. We will make revisions in the final version to better explain this point."}, {"review_id": "Voc77GX13G", "reviewer": "Reviewer_uUY8", "summary": "This paper reveals a profound insight about SNNs: spiking neurons act as low-pass filters at the network level, causing critical high-frequency information loss. The authors propose Max-Former, which recovers high-frequency signals through two lightweight operators: max-pooling in patch embeddings and depth-wise convolution replacing early-stage self-attention. This simple solution achieves remarkable results, 82.39% ImageNet accuracy (+7.58% over Spikformer) with 30% less power consumption, plus consistent improvements on CIFAR and neuromorphic datasets. This work challenges the prevailing assumption that binary activation constraints are the main Spiking Transformer limitation, instead identifying frequency domain characteristics as the key bottleneck. It provides theoretical proof and practical solutions that significantly improve performance and energy efficiency while preserving the spike-driven computing paradigm.", "questions": "Most of my questions are aligned with the weaknesses.\n\n1. Given that low-pass filtering in spiking neurons is a general phenomenon, would integrating high-frequency enhancement operators also improve other SNN architectures or tasks? \n\n2. The results in Tables 2-3 and Appendix A4(b) show that QKFormer* and Max-Former with hierarchical designs significantly outperform the other models. Does the hierarchical design itself help preserve high-frequency information?\n\n3. In Appendix A1.2(5), how is the DWC operation inserted into SSA? The explanation seems to be missing in the article.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "***Strength***\n\n1. **Motivation**\n- This paper challenges the common view that binary activations of spiking neurons in SNNs are the main reason for their performance limitation. The perspective is supported by both a clear theoretical foundation and empirical evidence. \n\n2. **Simple Design with Good Performance**\n- The design of Max-Former demonstrates that simply adding high-frequency enhancement operators (such as Max-Pooling and Depth-wise Convolution) can bring significant and consistent performance and energy efficiency improvements on almost all datasets, which can also inspire practical solutions for unlocking the full potential of Spiking Transformers. \n\n3. **Comprehensive Experiments**\n- The authors provide comprehensive experiments in both the main paper and the supplementary material to prove the importance of high-frequency information. Overall, the paper is well-written, clearly presented, and the insights are very interesting. \n\n\n**Weakness**\n\n1. Although low-pass filtering in spiking neurons is a common phenomenon, this paper only discusses how to improve spiking Transformers to achieve performance improvements on CV tasks based on this insight. A more general discussion will help to enhance the impact of this paper.\n\n2. As noted in the limitation section, Max-Former requires manually balancing frequency components, which the high-frequency enhancing strategy needs to be modified according to the task, rather than adaptively adjusting. However, I agree that Max-Former will inspire future architectural design in the SNN field.\n\n3. Technical details: There are typos in lines 236-237 of the main paper. Besides, in Section 3.2.3, the inputs to the DWC module appear to be non-spiking signals. A LIF neuron layer is probably missing here.", "was_revised": true, "final_justification": "After considering the author rebuttal, I am increasing my score to 5. The authors addressed my main concern about generalizability by demonstrating their high-frequency enhancement approach works beyond Transformers (showing improvements in convolutional SNNs), validating the broader applicability of their theoretical insight. The paper provides practical solutions that will likely inspire future research in spike transformers.", "rebuttal": "### **W1 & W2 & Q1 : Spiking Neural Networks Need High Frequency Information**\n\nThank you for appreciating our work. **We have investigated this aspect and successfully verified that high-frequency information is also beneficial to convolutional-based spiking neural networks.** To demonstrate this, we provide additional experimental results using the MS-ResNet18 backbone with the same experimental settings as reported in [1]. **Our Max-ResNet18 only makes minor modifications to the original MS-ResNet18 architecture**, i.e., adding a max pooling operator at the end of the downsampling shortcut module in the second ResNet-Block to enhance high-frequency information. \n\n|Dataset|Architecture|Param |T| Acc. (%) |\n|-|-|-|-|-|\n| Cifar10 | MS-ResNet[1]|12.50 |6| 94.92% |\n| | GAC-SNN[1] |12.63|6| 96.46%(+1.54%) |\n| | **Max-ResNet(ours)** |**12.50**|6| **96.67%(+1.75%)** |\n| | MS-ResNet(ANN)[1]|12.50 |NA| 96.75% |\n| Cifar100 | MS-ResNet[1] |12.54|6| 76.41% |\n| | GAC-SNN[1] |12.67|6| 80.45% (+4.04%) |\n| |**Max-ResNet(ours)** |**12.54**|6| **81.15%(+4.74%)**| \n| | MS-ResNet(ANN)[1]|12.54 |NA| 80.67% |\n\nThe results are quite promising: Max-ResNet consistently outperforms the MS-ResNet baseline with identical model size, and GAC-SNN with more parameters. **Notably, our Max-ResNet nearly matches the ANN version of MS-ResNet on CIFAR-10 (96.67% vs 96.75%) and significantly outperforms it on CIFAR-100 (81.15% vs 80.67%, +0.48%).**\n\nAt present, our conclusion is that **for static datasets, high-frequency information should be strengthened in the middle layers of the model, while for event datasets, enhancing high-frequency information in the shallow layers of the model will bring greater benefits.** This is also supported by **Section 4.3** and **Appendix A1.2** of our paper.  **We are preparing to share our analysis and results on more general cases in future work.**\n\n[1] “*Gated attention coding for training high-performance and efficient spiking neural networks.*” AAAI 2024.\n\n### **Q2: Hierarchical Design Is Not the Fundamental Reason**\n\n**In fact,this is mainly because QKFormer\\* has adopted maxpooling operators in the patch embedding stage, and, in hierarchical design, the patch embedding is used multiple times for downsampling.** Their architectural choices implicitly validate the importance of high-frequency information to the Spiking Transformer, which is also consistent with our experimental results presented in **Table 4** of the main paper, where enhancing high-frequency information through patch embedding strategies improves the performance of Max-Former from **81.63%** to **82.65%**.\n\n**Beyond that, our work provides theoretical grounding for many existing architectural choices in Spiking Transformers.** For instance, Spikformer v2[3] discovered that adding convolution layers (which act as high-pass filters) in the patch embedding stage significantly improves the performance, while removing the max-pooling operator in [2] leads to substantial performance drop. We believe our empirical and theoretical analysis offers valuable insights that will guide future research in spiking neural network architectures.\n\n[2] “*Spikformer: When spiking neural network meets transformer.*” ICLR 2023\n\n[3] “*Spikformer v2: Join the high accuracy club on imagenet with an snn ticket.*” arXiv 2024.\n\n### **Q3: Implementation of SSA with DWC**\n\nWe insert the DWC module after converting the output of the original SSA module into spikes. We will include detailed explanations in **Appendix A1.2** along with implementation code to ensure reproducibility.\n\n### **W3: Technical details**\nThank you for identifying these issues. We will correct all typos and revise the corresponding content in the final version."}, {"review_id": "tb8iFCZ9qr", "reviewer": "Reviewer_coWm", "summary": "The paper argues that the fundamental reason for the performance bottleneck of SNN in the Transformer structure is the rapid decay of high-frequency information in the network, which reduces the representation ability. To enhance high-frequency information, modules like Max-Pooling and Depth-Wise Convolution are adopted, boosting the model's performance on tasks such as CIFAR and ImageNet.", "questions": "1. The core idea of this work shares strong similarities with MetaFormer [1], which also posits that a combination of token mixer and channel mixer is sufficient for vision tasks. Additionally, several architectural choices appear closely aligned with those proposed in Spike-Driven Transformer v2 [2]. It would be valuable for the authors to discuss the distinctions between their work and these prior approaches.\n\n2. Furthermore, since this paper emphasizes the importance of high-frequency components in Spiking Transformers, could the authors elaborate on whether this frequency sensitivity also exists in convolutional SNN architectures, such as MS-ResNet [3] and its variant GAC-MS-ResNet [4]?\n\n3. Lastly, could the authors provide details on the training time of Max-Former? As training efficiency remains a key bottleneck in SNNs, it would be insightful to know whether the proposed design contributes to faster or more efficient training.\n\n[1] MetaFormer Baselines for Vision. IEEE TPAMI 2024.\n\n[2] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. ICLR 2024.\n\n[3] Advancing spiking neural networks toward deep residual learning. IEEE TNNLS 2024.\n\n[4] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.", "limitations": "See Questions before", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.It provides a fresh frequency-domain perspective on spiking networks, showing that performance degradation is not solely due to binary activations but to frequency loss. \n\n2.The proposed Max-Former is lightweight, easy to implement, and achieves superior performance and energy efficiency without requiring complex architectural changes or longer time steps.\n\n3.Well-written and organized\n\nWeaknesses:\n\n1.The core idea of this work shares strong similarities with MetaFormer [1], which also posits that a combination of token mixer and channel mixer is sufficient for vision tasks. Additionally, several architectural choices appear closely aligned with those proposed in Spike-Driven Transformer v2 [2]. It would be valuable for the authors to discuss the distinctions between their work and these prior approaches.\n\n2.Furthermore, since this paper emphasizes the importance of high-frequency components in Spiking Transformers, could the authors elaborate on whether this frequency sensitivity also exists in convolutional SNN architectures, such as MS-ResNet [3] and its variant GAC-MS-ResNet [4]?\n\n3.Lastly, could the authors provide details on the training time of Max-Former? As training efficiency remains a key bottleneck in SNNs, it would be insightful to know whether the proposed design contributes to faster or more efficient training.\n\n[1] MetaFormer Baselines for Vision. IEEE TPAMI 2024.\n\n[2] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. ICLR 2024.\n\n[3] Advancing spiking neural networks toward deep residual learning. IEEE TNNLS 2024.\n\n[4] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.", "was_revised": true, "final_justification": "Although the authors explain the differences with existing methods, I still think the main idea is similar to the previous ones, i.e., [1] and [2]. Thus, keeping the original score is suitable.", "rebuttal": "### **Q1: Clarification of the Core Idea and Architectural Design**\n\nThank you for recognizing the novelty of our work. In fact, **the design of our Max-Former does not emphasize the concept of token mixer or channel mixer as in [1] or [2].** Instead, we hope that the success of Max-Former can empirically support our core idea, as you pointed out, that the performance degradation of spiking neural networks is not only due to binary activation but also to frequency loss. **Therefore, to eliminate the influence of factors such as computational complexity and receptive field, Max-Former only makes minor changes to the existing model [3]**. Specifically, we add max-pooling operations in patch embedding and replace early-stage self-attention with lightweight depth-wise convolution. We primarily compare our Max-Former with the self-implemented version of [3] that uses the same training configurations to ensure fair comparison and an identical shortcut scheme to strictly adhere to the binary activation rule.\n\n**Our work serves as the theoretical foundation for many existing architectural choices in Spiking Transformers.** Specifically, in [4], the authors found that **directly applying known practices of MetaFormer does not achieve good results in Spiking Transformers**: employing average pooling operators to replace SDSA as the token mixer surprisingly results in substantial performance degradation from **61.0%** to **41.2%**. **Similar phenomena have been discussed in earlier works.** For instance, Spikformer v2[5] discovered that removing the max-pooling operator in [6] leads to a substantial performance drop, while adding convolution layers (which act as high-pass filters) in the patch embedding stage significantly improves the performance. **Our work reveals the underlying principles of these architectural designs: spiking transformers need to enhance high-frequency components to alleviate the feature degradation caused by their inherent low-pass activation.** We are well aware that there is still much space to be explored, and we hope that our Max-Former can serve as a good starting point for future research.\n\n[1] ”*MetaFormer Baselines for Vision*”. IEEE TPAMI 2024.\n\n[2] ”*Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips*“. ICLR 2024.\n\n[3] ”*Qkformer: Hierarchical Spiking Transformer using QK Attention*”. NeurIPS 2024.\n\n[4] ”*Spike-Driven Transformer*”. NeurIPS 2023. (rebuttal for reviewer 8FkB, Openreview)\n\n[5] “*Spikformer: When spiking neural network meets transformer.*” ICLR 2023\n\n[6] “*Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket*” arXiv 2024.\n\n### **Q2: Experiments on Convolutional SNN Architectures**\n\nYes, **we have verified that high-frequency information is also beneficial to convolutional-based spiking neural networks.** To demonstrate this, we provide additional experimental results using the MS-ResNet18 backbone with the same training settings as reported in [7]. **Our Max-ResNet18 only makes minor modifications to the original MS-ResNet18 architecture**, i.e., adding **one** max pooling operator at the end of the downsampling shortcut module in the second ResNet-Block to enhance high-frequency information.\n\n|Dataset|Architecture|Param |T| Acc. (%) |\n|-|-|-|-|-|\n| Cifar10 | MS-ResNet[1]|12.50 |6| 94.92% |\n| | GAC-SNN[1] |12.63|6| 96.46%(+1.54%) |\n| | **Max-ResNet(ours)** |**12.50**|6| **96.67%(+1.75%)** |\n| | MS-ResNet(ANN)[1]|12.50 |NA| 96.75% |\n| Cifar100 | MS-ResNet[1] |12.54|6| 76.41% |\n| | GAC-SNN[1] |12.67|6| 80.45% (+4.04%) |\n| |**Max-ResNet(ours)** |**12.54**|6| **81.15%(+4.74%)**| \n| | MS-ResNet(ANN)[1]|12.54 |NA| 80.67% |\n\nThe results are quite promising: Max-ResNet consistently outperforms the MS-ResNet baseline with identical model size, and GAC-SNN with more parameters. **Notably, our Max-ResNet nearly matches the ANN version of MS-ResNet on CIFAR-10 (96.67% vs 96.75%) and significantly outperforms it on CIFAR-100 (81.15% vs 80.67%, +0.48%)**. Beyond that, we are preparing to share our analysis and results on more general cases in future work.\n\n[7] “*Gated attention coding for training high-performance and efficient spiking neural networks.*” AAAI 2024.\n\n\n### **Q3: Training Efficiency**\n\nWe also attach importance to training efficiency. Here, we provide an additional Table regarding the full training time, supposing 8×H20 for ImageNet, with timestep=4.\n\n|Architecture|Max-10-384|Max-10-512 |Max-10-768|\n|-|-|-|-|\n| Time | 37 hours| 50 hours| 84 hours |\n\nFor CIFAR-10, CIFAR-100, and CIFAR10-DVS, we use a single A30 GPU with training times of 8h, 8h, and 1.5h, respectively. The training settings are detailed in **Appendix A1 (Table 5)**.\n\nRegarding to prior studies, we have provided a detailed comparison of training/inference time and GPU memory usage in **Appendix A3**. Compared to [3], our Max-Former **saves 16.4% time and 15.3% GPU memory in the training phase.**"}]

Author rebuttal:

Meta-review: The paper views spiking transformers through a frequency‑domain lens: LIF/IF neurons behave as low‑pass filters that dissipate high‑frequency content, contributing to SNN–ANN performance gaps. The authors propose Max‑Former, which restores high‑frequency information via (i) added max‑pooling in patch embedding and (ii) depth‑wise convolution replacing early self‑attention. Results show 82.39% top‑1 on ImageNet—well above Spikformer with similar model size—with consistent gains on CIFAR and neuromorphic datasets. The rebuttal adds ANN variants and a convolutional SNN (Max‑ResNet) showing similar benefits.

# Strengths

* Clear, actionable insight (low‑pass bias) tied to simple architectural changes that keep the spike‑driven paradigm.

* Strong empirical gains on standard SNN benchmarks (incl. ImageNet) with lightweight changes and good efficiency.

* Rebuttal meaningfully extends evidence (ANN comparison, Conv‑SNN), plus added training/efficiency details.

# Weaknesses

* Theory is suggestive rather than rigorous; arguments lean on intuition and visuals.

* Related‑work positioning vs. frequency‑aware models (wavelet/Fourier, spectral mixing) could be deeper; partial overlap with MetaFormer‑style design and Spikformer‑v2 choices remains a concern.

* Generality beyond vision SNNs is promising but not yet demonstrated.

# Reasons for decision
Two reviewers moved to accept (5) after rebuttal, one remained borderline-accept (4), and one stayed negative. I weigh the consistent empirical gains, simplicity, and the take-home conceptual message (mind the frequency loss in SNNs) more heavily than concerns about theoretical tightness or partial overlap with related lines. The added convolutional SNN experiment and ANN comparisons clarify scope: the benefit is specific to SNNs where high-frequency attenuation is salient. Overall, the paper will likely influence practical SNN design and spur deeper theoretical work.

# Discussion & rebuttal highlights

* Authors clarified the frequency-domain evidence (figures/ablations) and provided a derivation for LIF’s low-pass behavior; critics still desired more rigor, but other reviewers found the analysis sufficient.
* Comparisons & positioning: Authors argued their goal is not spectral learning per se; they added comparisons (e.g., to a spiking wavelet transformer) and discussed why broader spectral baselines might distract from the central claim.
* Generalization: New results on Max-ResNet indicate gains beyond transformers; ANN vs SNN contrasts show the benefit aligns with the hypothesized SNN frequency issue.
Decision: Accept (poster)