Title: Spiking Neural Networks Need High-Frequency Information
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spiking Neural Networks

Vision Transformers
Abstract: Spiking Neural Networks promise brain-inspired and energy-efficient computation by transmitting information through binary (0/1) spikes. Yet, their performance still lags behind that of artificial neural networks, often assumed to result from information loss caused by sparse and binary activations. In this work, we challenge this long-standing assumption and reveal a previously overlooked frequency bias:  **spiking neurons inherently suppress high-frequency components and preferentially propagate low-frequency information.** This frequency-domain imbalance, we argue, is the root cause of degraded feature representation in SNNs. Empirically, on Spiking Transformers, adopting Avg-Pooling (low-pass) for token mixing lowers performance to 76.73% on Cifar-100, whereas replacing it with Max-Pool (high-pass) pushes the top-1 accuracy to 79.12%. Accordingly, we introduce **Max-Former** that restores high-frequency signals through two frequency-enhancing operators: (1) extra Max-Pool in patch embedding, and (2) Depth-Wise Convolution in place of self-attention. Notably, **Max-Former** attains 82.39% top-1 accuracy on ImageNet using only 63.99M parameters, surpassing Spikformer (74.81%, 66.34M) by +7.58%. Extending our insight beyond transformers, our  **Max-ResNet-18** achieves state-of-the-art performance on convolution-based benchmarks: 97.17% on CIFAR-10 and 83.06% on CIFAR-100. We hope this simple yet effective solution inspires future research to explore the distinctive nature of spiking neural networks. Code is available: https://github.com/bic-L/MaxFormer.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper argues that the fundamental reason for the performance bottleneck of SNN in the Transformer structure is the rapid decay of high-frequency information in the network, which reduces the representation ability. To enhance high-frequency information, modules like Max-Pooling and Depth-Wise Convolution are adopted, boosting the model's performance on tasks such as CIFAR and ImageNet.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.It provides a fresh frequency-domain perspective on spiking networks, showing that performance degradation is not solely due to binary activations but to frequency loss. \n\n2.The proposed Max-Former is lightweight, easy to implement, and achieves superior performance and energy efficiency without requiring complex architectural changes or longer time steps.\n\n3.Well-written and organized\n\nWeaknesses:\n\n1.The core idea of this work shares strong similarities with MetaFormer [1], which also posits that a combination of token mixer and channel mixer is sufficient for vision tasks. Additionally, several architectural choices appear closely aligned with those proposed in Spike-Driven Transformer v2 [2]. It would be valuable for the authors to discuss the distinctions between their work and these prior approaches.\n\n2.Furthermore, since this paper emphasizes the importance of high-frequency components in Spiking Transformers, could the authors elaborate on whether this frequency sensitivity also exists in convolutional SNN architectures, such as MS-ResNet [3] and its variant GAC-MS-ResNet [4]?\n\n3.Lastly, could the authors provide details on the training time of Max-Former? As training efficiency remains a key bottleneck in SNNs, it would be insightful to know whether the proposed design contributes to faster or more efficient training.\n\n[1] MetaFormer Baselines for Vision. IEEE TPAMI 2024.\n\n[2] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. ICLR 2024.\n\n[3] Advancing spiking neural networks toward deep residual learning. IEEE TNNLS 2024.\n\n[4] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.\n\nQUESTIONS:\n1. The core idea of this work shares strong similarities with MetaFormer [1], which also posits that a combination of token mixer and channel mixer is sufficient for vision tasks. Additionally, several architectural choices appear closely aligned with those proposed in Spike-Driven Transformer v2 [2]. It would be valuable for the authors to discuss the distinctions between their work and these prior approaches.\n\n2. Furthermore, since this paper emphasizes the importance of high-frequency components in Spiking Transformers, could the authors elaborate on whether this frequency sensitivity also exists in convolutional SNN architectures, such as MS-ResNet [3] and its variant GAC-MS-ResNet [4]?\n\n3. Lastly, could the authors provide details on the training time of Max-Former? As training efficiency remains a key bottleneck in SNNs, it would be insightful to know whether the proposed design contributes to faster or more efficient training.\n\n[1] MetaFormer Baselines for Vision. IEEE TPAMI 2024.\n\n[2] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. ICLR 2024.\n\n[3] Advancing spiking neural networks toward deep residual learning. IEEE TNNLS 2024.\n\n[4] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.\n\nLIMITATIONS:\nSee Questions before", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reveals a profound insight about SNNs: spiking neurons act as low-pass filters at the network level, causing critical high-frequency information loss. The authors propose Max-Former, which recovers high-frequency signals through two lightweight operators: max-pooling in patch embeddings and depth-wise convolution replacing early-stage self-attention. This simple solution achieves remarkable results, 82.39% ImageNet accuracy (+7.58% over Spikformer) with 30% less power consumption, plus consistent improvements on CIFAR and neuromorphic datasets. This work challenges the prevailing assumption that binary activation constraints are the main Spiking Transformer limitation, instead identifying frequency domain characteristics as the key bottleneck. It provides theoretical proof and practical solutions that significantly improve performance and energy efficiency while preserving the spike-driven computing paradigm.\n\nSTRENGTHS AND WEAKNESSES:\n***Strength***\n\n1. **Motivation**\n- This paper challenges the common view that binary activations of spiking neurons in SNNs are the main reason for their performance limitation. The perspective is supported by both a clear theoretical foundation and empirical evidence. \n\n2. **Simple Design with Good Performance**\n- The design of Max-Former demonstrates that simply adding high-frequency enhancement operators (such as Max-Pooling and Depth-wise Convolution) can bring significant and consistent performance and energy efficiency improvements on almost all datasets, which can also inspire practical solutions for unlocking the full potential of Spiking Transformers. \n\n3. **Comprehensive Experiments**\n- The authors provide comprehensive experiments in both the main paper and the supplementary material to prove the importance of high-frequency information. Overall, the paper is well-written, clearly presented, and the insights are very interesting. \n\n\n**Weakness**\n\n1. Although low-pass filtering in spiking neurons is a common phenomenon, this paper only discusses how to improve spiking Transformers to achieve performance improvements on CV tasks based on this insight. A more general discussion will help to enhance the impact of this paper.\n\n2. As noted in the limitation section, Max-Former requires manually balancing frequency components, which the high-frequency enhancing strategy needs to be modified according to the task, rather than adaptively adjusting. However, I agree that Max-Former will inspire future architectural design in the SNN field.\n\n3. Technical details: There are typos in lines 236-237 of the main paper. Besides, in Section 3.2.3, the inputs to the DWC module appear to be non-spiking signals. A LIF neuron layer is probably missing here.\n\nQUESTIONS:\nMost of my questions are aligned with the weaknesses.\n\n1. Given that low-pass filtering in spiking neurons is a general phenomenon, would integrating high-frequency enhancement operators also improve other SNN architectures or tasks? \n\n2. The results in Tables 2-3 and Appendix A4(b) show that QKFormer* and Max-Former with hierarchical designs significantly outperform the other models. Does the hierarchical design itself help preserve high-frequency information?\n\n3. In Appendix A1.2(5), how is the DWC operation inserted into SSA? The explanation seems to be missing in the article.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
In this paper, the authors analyze the Spiking Transformer in the frequency domain. Interestingly, they found that LIF tends to pass low-frequency information and block high-frequency information. This phenomenon has been verified using an intuitive example. I am no expert in frequency-domain analysis, but I found the conclusion interesting and valuable for this area. The final resulting model, called MaxFormer, is a straightforward modification of the existing Spiking token-mixer and achieves a significant performance improvement.

STRENGTHS AND WEAKNESSES:
Strength:

+ Strong support of low-pass filter nature in LIF neuron, based on the analysis in Section 3.1.

+ Impressive performance improvement over average-pooling-based baselines. However, I did not verify the code's correctness. Given that the authors have provided the implementation of CIFAR100 models, I will buy the results for now. 


Weakness: 

- Throughout the paper, the authors keep talking about the difference between LIF and ReLU neurons and claim that the ReLU neuron is more than just a low-pass filter. It would be interesting to see the ANN version of MaxFormer (at least on CIFAR-100). The current table only shows some trivial ANN practices, which do not fully convince the readers.

- I hope the authors can fully open-source their implementation in rebuttal/after submission, or clarify the reason that only the CIFAR-100 code is provided.

QUESTIONS:
1. How do you explain the performance improvement from the pre-spike shortcut or vanilla shortcut with the frequency information? 

2. Except for max-pooling, is there another way to increase the high-frequency information by modifying LIF?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHere is a **precise summary** in bullet points:\n\n1. Issue Addressed:\n\n   Spiking Transformers underperform compared to traditional deep networks due to information loss, especially of high-frequency signals, from their binary and sparse activations.\n\n2. Proposed Solution:\n\n   Introduce Max-Former, which enhances high-frequency feature retention via:\n\n     Max-Pooling in patch embedding.\n     Depth-Wise Convolution replacing self-attention.\n\n3. Evaluation:\n\n   CIFAR-100: Improves accuracy from 76.73% (Avg-Pooling) to 79.12% (Max-Pooling).\n   ImageNet: Achieves 82.39% top-1 accuracy, a +7.58% gain over Spikformer, with comparable model size.\n\nSTRENGTHS AND WEAKNESSES:\n1. Lack of Theoretical Justification for Low-Frequency Bias in Spiking Neurons:\n   While the paper claims that spiking neurons preferentially propagate low-frequency information, there is no rigorous theoretical or empirical analysis provided to support this assertion. A formal frequency-domain analysis or signal decomposition study would significantly enhance the credibility of this claim.\n\n2. Insufficient Literature Review on High-Frequency Modeling:\n   The manuscript omits discussion of several **relevant works that address high-frequency representation in vision models**, such as:\n\n   *SVT (Scattering Vision Transformer)[1], which models spectral components using wavelet-inspired mixing,\n   * HiLo Attention [2], which splits frequency bands for efficient attention computation,\n   * SpectFormer and WaveViT, which are designed to efficiently encode high-frequency textures.\n\n   Including these would provide valuable context and distinguish the proposed method more clearly from existing approaches.\n\n3. Lack of Comparison with Contemporary Spectral Mixing Techniques:\n   The proposed Max-Former does not compare or contrast with similar spectral decomposition and frequency-aware channel mixing methods, such as:\n   * Monarch Mixing, which exploits structured linear transforms for spectral control,\n   * EinFFT used in SiMBA-TS[3], which mixes channels via frequency-domain transformations.\n\n   A detailed comparison is essential to understand how Max-Former is uniquely positioned among these recent advances.\n\n4. Limited Generalizability Discussion:\n   It remains unclear whether the proposed Max-Former architecture is generalizable to non-spiking architectures or other modalities such as image captioning or multimodal tasks. The paper should explicitly discuss whether the frequency enhancement mechanisms apply only to spiking transformers or more broadly across architectures.\n\n5. Unsubstantiated Claims About Max-Former’s High-Frequency Behavior:\n   The claim that Max-Former captures high-frequency components more effectively is not backed by sufficient evidence. There is no frequency-domain visualization, filter response plot, or Fourier spectrum analysis to demonstrate that the method indeed enhances high-frequency signals across layers or tasks.\n\n6. Missing Comparison with Strong Baselines:\n   Tables 2 and 3 lack comparative results with strong recent baselines, particularly:   SVT[1], HiLo Attention[2], EinFFT-based SiMBA[3].\n\n   These omissions make it difficult to assess whether Max-Former truly offers a state-of-the-art tradeoff between accuracy, frequency preservation, and efficiency.\n\n7. Potential Overlap with Existing Work:\n   A recent preprint available at [arXiv:2505.18608](https://arxiv.org/pdf/2505.18608) appears to tackle a similar problem space of frequency-aware modeling in spiking networks. The authors should clearly articulate how their contributions differ from or improve upon this work, including differences in methodology, theoretical insight, or empirical scope.\n\n[1] Patro, Badri, and Vijay Agneeswaran. *\"Scattering Vision Transformer: Spectral Mixing Matters.\"* NeurIPS 2024.\n[2] Fast Vision Transformers with HiLo Attention, NeurIPS 2022\n[3] SiMBA-TS: Simplified Channel Mixing and Mamba for Long-term Time Series Forecasting, ICASSP-2025\n\nQUESTIONS:\nRefer to Strengths And Weaknesses session\n\nLIMITATIONS:\nRefer to Strengths And Weaknesses session", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-06"}