Title: TDFormer: Top-Down Attention-Controlled Spiking Transformer
Venue: NeurIPS 2025
Accepted: no
Keywords: Top-down mechanism

Brain-inspired Computing

Spiking Neural Networks

Vision Transformers
Abstract: Traditional spiking neural networks (SNNs) can be viewed as a combination of multiple subnetworks with each running for one time step, where the parameters are shared, and the membrane potential serves as the only information link between them. However, the implicit nature of the membrane potential limits its ability to effectively represent temporal information. As a result, each time step cannot fully leverage information from previous time steps, seriously limiting the model's performance. Inspired by the top-down mechanism in the brain, we introduce TDformer, a novel model with a top-down feedback structure that functions hierarchically and leverages high-order representations from earlier time steps to modulate the processing of low-order information at later stages. The feedback structure plays a role from two perspectives: 1) During forward propagation, our model increases the mutual information across time steps, indicating that richer temporal information is being transmitted and integrated in different time steps. 2) During backward propagation, we theoretically prove that the feedback structure alleviates the problem of vanishing gradients along the time dimension. We find that these mechanisms together significantly and consistently improve the model performance on multiple datasets. In particular, our model achieves state-of-the-art performance on ImageNet with an accuracy of 86.83\%.


Peer reviews:
comments: SUMMARY:
This article introduces a Top-Down Attention-Controlled Spiking Transformer (TDFormer), which aims to improve the performance of SNNs by incorporating top-down attention mechanisms inspired by human brain processes.

STRENGTHS AND WEAKNESSES:
The paper presents a well-written and clearly articulated study, offering a method that effectively integrates principles from neuroscience with SNNs. 

This interdisciplinary approach enhances the biological plausibility of the model and contributes meaningful insights to the development of energy-efficient neural computation.

QUESTIONS:
1. While the proposed method demonstrates promising results, it would be more convincing to evaluate its performance on a more advanced architecture such as Spikformer v2 [1], which may better reveal its generalizability and scalability. Additionally, the method presents a trade-off between power consumption and accuracy—raising the question of whether further optimization could help reduce power usage without significantly compromising performance. Exploring such improvements would enhance the practical applicability of the method, especially for energy-constrained scenarios.


[1] Spikformer v2: Join the high accuracy club on imagenet with an snn ticket

LIMITATIONS:
1. The proposed method has a trade-off between power consumption and accuracy, meaning that achieving higher accuracy may come at the cost of increased energy usage, while reducing power consumption could lead to a slight degradation in performance.

2. The experimental results appear to be highly dependent on the strength of the baseline. When evaluated against more competitive baselines, the proposed method shows only limited improvements, suggesting that its overall effectiveness may be modest.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TDFormer, which integrates top-down attention-controlled modules (TDAC) into Transformer-based SNNs, incorporating the biological system-inspired top-down attention mechanism to enhance the performance and robustness. However, the proposed Processing Module (PM) and Control Module (CM) within the TDAC closely resemble the design of prior work [1], which raises concerns about the novelty and contributions of this paper. Additionally, the two feedforward stages introduced in the paper result in increased FLOPs, memory, and parameter costs, as well as higher inference latency. These factors are not adequately reflected in the experimental comparison tables. While the proposed method yields marginal performance improvement, it significantly undermines the inherent advantages of SNNs, such as low power consumption, low computational cost, and fast inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe structure of the paper is clear and the problem is well-defined. It sets up new modules to introduce top-down attention. The experiments on classification datasets are comprehensive and include tests on model robustness against attacks and noise.\n\nWeaknesses:\n\n1. The two feedforward stages introduced in this paper result in additional FLOPs, memory, and parameter costs, and increase inference latency. \n\n2. The experimental results don't adequately reflect or compare these additional costs. For example, Table 1 shows that integrating TDAC into QKFORMER improves ImageNet accuracy by 0.92%, but the corresponding cost increase isn't fully evaluated. \n\n3. The accuracy improvement is at the cost of over 40% more power consumption, which undermines SNNs' advantages of low latency, low computational cost, and low power consumption.\n\nQUESTIONS:\n1. The proposed method requires two inferences, with the equivalent number of time steps being close to doubling the baseline's time steps, which also nearly doubles the energy consumption. I doubt that this method outperforms simply doubling the time steps of the baseline.\n\n2. The biological relevance of the paper is questionable. Does TDFormer truly simulate the attention mechanisms in the human brain? Although the paper mentions the existence of top-down attention mechanisms in biological systems, is this simulation accurate enough? Are there other mechanisms or model architectures that can more closely mimic the attention mechanisms of the human brain? In traditional autoregressive attention transformer structures, newly generated words are fed back into the input to guide the next generation.\n\n3. The inference and training costs are close to double that of the original structure. This paper does not provide pseudocode for the training process. During training, will spiking attention use BPTT to propose a decline? Additionally, why does using both L1 and L2 in the loss function promote model convergence? The paper does not provide a reasonable description.\n\n4. This paper lacks visual case studies to show what changes occur in the attention maps, output attention features, or membrane potentials after top-down guidance. Without visualization experiments, it is difficult to convince that top-down attention has the advantage.\n\n[1] Shi, Baifeng, Trevor Darrell, and Xin Wang. \"Top-down visual attention from analysis by synthesis.\", CVPR, 2023.\n\nLIMITATIONS:\nSee Questions before", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TDFormer, a novel spiking neural network (SNN) architecture that incorporates a top-down feedback structure inspired by brain mechanisms. The authors identify limitations in traditional SNNs where temporal information is poorly utilized due to reliance solely on membrane potential dynamics. TDFormer addresses this by adding feedback connections between time steps through Control Module (CM) and Processing Module (PM) components. The authors provide theoretical analysis showing the feedback structure increases mutual information across time steps and mitigates vanishing gradients. Experiments demonstrate state-of-the-art performance on ImageNet (86.83% accuracy) and consistent improvements across multiple datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Clear motivation, effective results: The biological inspiration from prefrontal-visual cortex feedback is compelling and well-articulated. The identification of poor temporal information utilization in traditional SNNs through mutual information analysis (Figure 1) provides strong motivation.\n2. Theoretical analysis support: The paper provides rigorous theoretical analysis including: Proof that feedback structure mitigates vanishing gradients (Theorems 4.3) and Upper bound analysis for processing module variance (Proposition 4.1).\n\nWeaknesses:\n1. The experiments focus primarily on image classification. The effectiveness for other domains such as object detection, semantic segmentation, or natural language processing tasks remains unclear.\n2. What are the practical considerations for implementing multi-hierarchical thresholds and feedback connections on actual neuromorphic chips? Are there hardware-specific constraints that would limit deployment?\n\nQUESTIONS:\n1. How sensitive is the method to the choice of time steps T? Is there an optimal range?\n2. Some theoretical assumptions (e.g., independence assumptions in Lemma B.3) may not hold perfectly in practice.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper starts from the perspective that current SNNs fail to effectively utilize temporal information across time steps. Inspired by the top-down pathways in the human brain, it introduces a top-down control mechanism into the Spiking Transformer to enhance the model’s ability to capture and model temporal dependencies. This approach is compatible with various Spiking Transformer architectures, and the authors argue that it can achieve significant performance improvements over the baseline without introducing excessive energy overhead. The authors corroborate this claim with a substantial body of experiments and present a detailed energy-consumption analysis to further demonstrate the superiority of their approach. However, I believe the paper still suffers from several significant shortcomings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The top-down pathway is designed based on brain-inspired principles, which lends the method a strong degree of biological plausibility.\n2. The energy consumption analysis is thorough and demonstrates that the proposed method preserves the inherent low-power advantage of SNNs.\n3. The plug-and-play design of the modules allows for easy integration into various representative Spiking Transformer architectures.\n\n**Weakness**\nSee Questions & Limitaions\n\nQUESTIONS:\n1. The authors mention that they analyze the insufficient temporal information extraction capability in Spiking Transformers; however, the paper seems to lack a thorough theoretical derivation to support this claim.\n2. To the best of my knowledge, this is not the first work that enhances Spiking Transformers by improving temporal modeling—prior methods such as TIM and TE-Spikformer have already explored this direction. However, the paper neither compares its approach with these methods nor demonstrates superior performance over them, which raises concerns about whether the literature review and benchmarking are sufficiently thorough.\n3. In TIM, additional modules and parameters are introduced only in the forward pass(bottom-up pass), yet it achieves better performance on CIFAR10-DVS compared to the method proposed in this paper. Why is that the case?\n4. The reproduction of some baseline methods appears to be problematic—for instance, Spikformer performs 0.8% lower and SDT 3% lower than the original papers on CIFAR10-DVS. Does this indicate that the baseline implementations were not successfully reproduced?\n5. On certain datasets, the performance improvement of TDFormer over the baseline is marginal (and the possibility of reproduction errors has not been ruled out). Since TDFormer introduces additional parameters, it remains unclear whether these minor gains stem from enhanced temporal modeling or simply from the increased parameter count—this question is not addressed through any ablation study in the paper.\n\nLIMITATIONS:\nAs stated in the Question, the paper has several limitations:\n\t1.\tIt lacks rigorous theoretical derivation for the core problem it aims to solve;\n\t2.\tThe related work survey appears insufficient;\n\t3.\tSome baseline reproductions may be problematic—the observed Acc@1 discrepancies seem to go beyond typical experimental variance;\n\t4.\tThe ablation studies are not well-designed—those provided are inadequate to convincingly attribute the performance gains to improved temporal modeling capabilities.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}