DeepTAGE: Deep Temporal-Aligned Gradient Enhancement for Optimizing Spiking Neural Networks
Authors: Wei Liu
Li Yang
Mingxuan Zhao
Shuxun Wang
Jin Gao
Wenjuan Li
Bing Li
Weiming Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Spiking Neural Networks
Deep Temporal-Aligned Gradient Enhancement
Spatio-Temporal Deep Supervision
Optimization Imbalance.
Primary area: applications to neuroscience & cognitive science

Abstract: Spiking Neural Networks (SNNs), with their biologically inspired spatio-temporal dynamics and spike-driven processing, are emerging as a promising low-power alternative to traditional Artificial Neural Networks (ANNs). However, the complex neuronal dynamics and non-differentiable spike communication mechanisms in SNNs present substantial challenges for efficient training. By analyzing the membrane potentials in spiking neurons, we found that their distributions can increasingly deviate from the firing threshold as time progresses, which tends to cause diminished backpropagation gradients and unbalanced optimization. To address these challenges, we propose Deep Temporal-Aligned Gradient Enhancement (DeepTAGE), a novel approach that improves optimization gradients in SNNs from both internal surrogate gradient functions and external supervision methods. Our DeepTAGE dynamically adjusts surrogate gradients in accordance with the membrane potential distribution across different time steps, enhancing their respective gradients in a temporal-aligned manner that promotes balanced training. Moreover, to mitigate issues of gradient vanishing or deviating during backpropagation, DeepTAGE incorporates deep supervision at both spatial (network stages) and temporal (time steps) levels to ensure more effective and robust network optimization. Importantly, our method can be seamlessly integrated into existing SNN architectures without imposing additional inference costs or requiring extra control modules. We validate the efficacy of DeepTAGE through extensive experiments on static benchmarks (CIFAR10, CIFAR100, and ImageNet-1k) and a neuromorphic dataset (DVS-CIFAR10), demonstrating significant performance improvements.

Published as a conference paper at ICLR 2025
DEEPTAGE: DEEP TEMPORAL-ALIGNED GRADIENT ENHANCEMENT FOR OPTIMIZING SPIKING NEURAL NETWORKS
Wei Liu', Li Yang", Mingxuan Zhao', Shuxun Wang', Jin Gao', Wenjuan Li* Bing Lit, Weiming H23.2,3*
Beijing Key Laboratory of Super-Inted Intel-entlll Security of Multi-Modal Information, MAIS, CASIA School of Artificial Intelligence, University of Chinese Academy of Sciences $ School of Information Science and Technology, ShanghaiTech University
mnacmingxuannbuug edu.cn on {liuwei, wangshuxun2022 aiia. ac
yai, jin.gao,
gao, wenjuan. 1i, bii, on.p....lpr.ia.
ABSTRACT
Spiking Neural Networks (SNNs), with their biologically inspired spatio-temporal dynamics and spike-driven processing, are emerging as a promising low-power alternative to traditional Artificial Neural Networks (ANNs). However, the com- plex neuronal dynamics and non-differentiable spike communication mechanisms in SNNs present substantial challenges for efficient training. By analyzing the membrane potentials in spiking neurons, we found that their distributions can in- creasingly deviate from the firing threshold as time progresses, which tends cause diminished backpropagation gradients and unbalanced optimization. To ad- Gradient Enhance-
dress these challenges, we propose Deep Gemporal-Aligne ment (DeepTAGE). a novel approach that improves optimization gradients in SNNs from both internal surrogate gradient functions and external supervision methods. Our DeepTAGER dynamically adjusts surrogate gradients in accordance with the membrane potential distribution across diffcrent time steps, enhancing their respective gradients in temporal-aligned manner that promotes balanced training. Moreover, to mitigate issues of gradient vanishing or deviating during backpropagation, DeepTAGE incorporates deep supervision both spatial (net- work stages) and temporal (time steps) levels to ensure more effective and robust network optimization. Importantly, method can be seamlessly integrated into existing SNN architectures without imposing additional inference costs or requir- ing extra control modules. We validate the efficacy of DeepTAGE through exten- sive experiments on static benchmarks (CIFAR10, CIFAR 10, and ImageNet- 1k) and neuromorphic dataset (DVS-CIFAR1O), demonstrating significant perfor- mance improvements.
INTRODUCTION
Spiking Neural Networks (SNNs) are a biologically inspired computing paradigm designed with dy- namic spatio- temioral connections and binary spike-driven communication mechanisms (Roy al. 2019 Schuman et al. 2022 Li et al. 2023). Unlike traditional Artificial Neural Networks (ANNs), which rely continuous activations, SNNs operate with discrete spikes, offering the potential for more efficient and energy-saving computations, particularly in neuromorphic hardware implementa- tions. The event-based nature SNNs, where computations
triggered only upon the receipt of spike, provides inherent sparsity and lower power consumption, making them attractive for real-time applications in energy constraiid environments (Pei al. 2019 Davies et al. 2018 Merolla al.) 2014 Frenkel et al. 2023).
"Corresponding author: wmhu umnhu@nlprie imhu@@lpriia.ac.cm

---

Published as a conference paper at ICLR 2025
However, despite these advantages, the complex dynamics of SNNs pose significant challenges for effective training. The simultaneous propagation of backpropagation gradients through both spa- tial and temporal dimensions is complicated by the non-differentiable nature of the spike activation function. As a result, training SNNs remains considerably more difficult than training ANNs. To mitigate this issue, previous works have employed surrogate gradient methods to approximate gra- dients during backpropagation by replacing the non-differentiable spike activation function with smoother alternatives. This approach has achieved strong results in shallow networks but faces challenges when scaling to deeper networks due to the vanishing gradient problem. While various methods, including residual connections (He et al., 2016b Fang al.. 2021a| Zheng et al. [02T] Hu et al. 2024). normalization techniques (Kim & Panda 2021), and attention mechanisms (Duan et al. 2022 Yao al. 2023e a b). have been adopted for improved SNN learning, they not ade- quately address the challenges of maintaining effective gradient propagation across both spatial and temporal dimensions in SNNs.
In this work, we focus on optimizing SNNs by enhancing their backpropagation gradients across spatial and temporal dimensions. Our analysis reveals that as the time step increases, the distri- bution of membrane potentials in SNNs tends to shift away from the firing threshold, leading to diminished gradient magnitudes and unbalanced training processes. This shift can further exacer- bate the vanishing gradient problem, particularly during the backpropagation of gradients through multiple network layers and time steps.
To address these challenges, we propose Deep Temporal-Aligned Gradient Enhancement (Deep- TAGE), a novel method that enhances the optimization gradients of SNNs from both the internal surrogate gradient functions and external supervision approaches. DeepTAAGE introduces tem- poral alignment mechanism that dynamically adjusts the surrogate gradient functions based on the membrane potential distribution at each time step, ensuring that neurons receive more effective gradients when their membrane potentials diverge from the firing threshold. Additionally, we incor- porate
Deep Supervision (STDS), which supplements deep supervision at multiple network stages and time steps of SNNs, further enhancing gradient flow and reducing optimization bottlenecks. Our gradient enhancement approach brings several key advantages: (1) improving spik- ing activity across network layers and time steps, leading to enhanced representation capabilities; (2) accelerating the training convergence of SNNs and ultimately achieving better performance; (3) improving SNN optimization without the need change the network architecture or add additional control modules.
We validate the effectiveness of DeepTAAGE through extensive experiments both static and neu- romorphic datasets, including CIFAR10 10 (Krizhevsky et al.) 2010), CIFARIO0 (Krizhevsky et al.) 2010), ImageNet-1k (Deng al., 2009), and DVS-CIFARTO (Li et al.) (017). Across these bench- marks, our method consistently outperforms existing tfate-of-the-ee methods, confirming its effi- cacy in optimizing SNNs for various tasks, Our main contributions are threefold:
Temporal-Alie -liigned Gradient Enhancement. We propose a novel gradient enhancement tech- nique that adapts the surrogate gradient function according to the distribution of membrane potentials at each time step. This ensures more balanced and effective backpropagation throughout the network.
Spatio-Temporal Deep Supervision. We introduce an optimization paradigm that supple- ments deep supervision into multiple network stages and time steps of SNNs, mitigating gradient vanishing issues and enhancing the convergence of the network.
We validate the effectiveness of our methods through extensive experiments and demon- strate significant improvements over existing SNN methods on multiple benchmarks.
2 RELATED WORK
SNN optimization has progressed through two major avenues: ANN-to-SNN conversion and direct training via surrogate gradients. The ANN-to-SNN conversion approach hinges on approximating spiking neuron firing rates through ReLU activations in ANNs. Early work by Rueckauer et al. (017) explored methods to minimize conversion errors by adjusting activation functions and intro- ducing techniques like SpikeNorm and threshold balancing to reduce accuracy loss during conver- sion (Rueckauer al. 2016, Cao et al. 2015 Sengupta et al. 2019). Despite these improvements,
2

---

Published as a conference paper at ICLR 2025
conversion methods tend require large time steps, which increases latency (Bu et al. 2023
Jiang
ct al. ).23).
the other hand, utilizes surrogate gradients handle the differentiability of spike functions, as seen in the spatiotemporal backpropagation (STBP) method introduced by Wu et al. (2018). This approach has achieved strong results in shallow networks but faces challenges when scaling to deeper networks due to vanishing gradients. To address this, sev- eral normalization techniques have been proposed, such as temporal batch normalization (BNTT) (Kim & Panda (02T) and threshold-dependent batch normalization (tdBN) (Zheng et al. 2021
Direct SNN training,
which improve gradient flow and convergence in deep SNNs.
Additionally, residual connections, inspired by ResNet architectures, have been adapted for SNNs. Fang et al. (2021a) proposed the SEW-ResNet framework, which enables more efficient spike prop- agation and helps maintain performance as network depth increases. Recent gradient optimization techniques, like Temporal Efficient Training(TET) (Deng et al., 2022) and Spatial Learning Through Time (SLTT) (Meng al. 2203), further improve performance by refining backpropagation through time, with SLTT demonstrating that focusing on spatial gradients can boost optimization efficiency.
Our work builds these foundations by introducing Deep Temporal-Aligned Gradient Enhance- ment (DeepTAGF)), which adapts surrogate gradients according to the membrane potential distribu- tion at each time step. This novel approach enhances training efficiency without adding computa- tional complexity, improving SNN performance across various benchmarks.
3 METHOD
3.1 MOTIVATION
SNNs characterized by their unique method of processing information through the temporal dynamics of membrane potential updates and spike activations across multiple time steps. These dynamics crucial for understanding and optimizing the computational capabilities of SNNs. In Figure (a), we present histograms that depict the distribution of membrane potentials at each time step within an inner layer of an SNN. These distributions evolve over time: they are sharply peaked and closer to the threshold voltage Ueh at initial time steps, but become broader and drift away from Wtt later steps.
2000
3000
2000
1500
500
0.6
1000
3000
2500
2506
044
2000
2006
1500
1506
1000
1000
500
500
tw.
0%
5
t-4
0.
-10 -35 00 35 1.5 3.0 2.5
10
Figure 1: Visualization of membrane potential dynamics and surrogate gradient function in the spiking ResNet-19, (a) Histograms of membrane potentials at each time step from inner layer of the 1esNet-19, showing sharper distributions close to the threshold voltage (Wrx) at earlier time steps and flatter distributions further from vas later time steps. (b) Curve of the surrogate gradient function derived from the arctangent function, highlighting larger gradients near UeA and smaller gradients as inputs deviate from UtΛ (Uth
When training SNNs, due to the con-differentiable nature of the spike activation function for input membrane potentials, surrogate gradient functions are utilized to approximate the gradients nec- essary for effective backpropagation. Figure (b) illustrates commonly used surrogate gradient
3

---

Published as a conference paper at ICLR 2025
function, which is derived from the derivative of an aretangent function (with Mha set to 1):
sg
1 T )
This function produces larger gradients when input values are proximal to oth, with gradients de- creasing the values diverge from this threshold.
Considering the membrane potential distribution in Figure(T(a), we can find that as time progresses, an increasing proportion of membrane potentials settle into regions where the surrogate gradients are negligible. This tends to result in an unbalanced optimization process for SNNs of multiple time steps and hinders the convergence of the network. Moreover, the vanishing gradient issue may be exacerbated as the network undergoes multiple layers of backpropagation, potentially impairing the network's ' spiking functionality and limiting representational capabilities.
To tackle these challenges, we propose novel approach called "Deep Temporal-Alighe Gradient Enhancement" optimize SNNs. This approach aims refine the gradient computation process and introduce deep supervision at both temporal and spatial levels. By balancing and enhancing the gradients over different time steps and network stages, our approach is expected to not only stabilize the learning process but also to amplify the representational power of SNNs by improving spike generation capability.
3.2
TEMPORAL-ALIGNE GRADIENT ENHANCEMENT
Based on the above analysis, it is natural to expect that the surrogate gradient functions should be varied at different time steps for better optimization of SNNs. In line with this, we propose method termed Cemporal-Aligned Gradient Enhancement (TAGE), which dynamically adapts the surrogate gradient function based the distribution of input membrane potentials to balance gradient com- putation across various time steps.
To this end, first model the degree of deviation of the input membrane potentials 2; from the firing threshold eu at each time step t. We introduce a metric og which quantifies this deviation as the 1.2 norm of the difference between the potentials 2a and threshold Ura:
ot || vtal/2-
A larger dr reflects that more membrane potentials at time step t are distributed away from the threshold Uzh, often resulting in smaller gradicnts due to the properties of the surrogate gradient function described by equation [
As delineated in Section I.1 the distribution membrane potentials progressively shifts away from the threshold Vth in subsequent time steps. This shift tends cause unbalanced optimization and potentially exacerbate the vanishing gradient problem-a significant challenge when training SNNs over multiple time steps. To address this issue, we use a, a vital measure for comparing the distribution membrane potentials across different time steps and adaptively modify the surrogate gradient function ensure more uniform and effective training. The adaptation involves normal- ization factor )(o,), defined as follows:
5(0.) = w. + w),
(3)
where w is a hyper- parameter that determines the impact of the distributional deviation ratio 0z/01 between times steps and 1, where 5(a,) when o; 01. This ratio ensures that the surrogate gradient function remains responsive to the changing distribution of inputs across time steps.
This normalization factor 5(0t) is integrated into the surrogate gradient function as follows:
sg, (2,)
(4)
1 + x² Z, Zz-nieh
b(a.)
This formula essentially scales the membrane potentials 2, centered Up when computing the surrogate gradients. The scaling does not change the size relationship between H, and Vas and allows inputs with flatter distributions and farther away from Usa receive more effective gradients.
4

---

Published as a conference paper at ICLR 2025
Deep Temporal-Aligned Gradient Enhancement
R-T Σ
Cortol v Σ coolpi
Stage 1
Stage 2
Stage K
Time step
Gradient Enhancement
Coodent
P2
cradiats
Conv
Conv
Conv
GAP
t=T
LiFNode
Pi
p
Conv
Conv
Conv
GAP
f 2
LIFhode
p1
1=1
Cony
Conv
Conv
117Mode
GAP
pk
Network stage
The overall framework of our Deep Cemporal-Alignee Gradient Enhancement method. We propose to enhance gradient-basee SNN optimization by adjusting the surrogate gradient func- tions of different time steps according their membrane potential distributions. Our method also incorporates spatio-temporal deep supervision to further improve the gradient flow for SNN opti- mization.
Figure 2:
3.3 SPATIO-TEMPORAL DEEP SUPERVISION
The proposed Temporal-Aligned Gradient Enhancement (TAGE) optimizes SNNs by adapting the surrogate gradient functions the varying membrane potential distributions at different time steps. However, due to the inherent dynamic temporal dependence and the non-differentiable nature of ac- tivations within SNNs, gradients may vanish deviate significantly as they back-propagate through multiple layers and time steps, which impedes the convergence and overall performance of the net- work. To tackle these challenges, we extend our approach to Deep Temporal-Alige Gradient Enhancement (DeepTAGE), which introduce Temporal Deep Supervision (STDS) into the network to enhance the gradient flow on the back-propagation path.
For SNN with K network stages, denote its final classification output for each time step separately by (p' t E [1, To supplement deep supervision into the SNN, as illustrated in Figure we introduce auxiliary classifier networks {9u & [L, 1]} at all network stages prior to the final K-th stage. Each classifier network consists of three convolution layers, followed by global average pooling layer and a fully connected layer produce classification predictions. Let (F), t E [1, & E [ - []} denote the output feature maps of different time steps from previous 1 network stages within the SNN. We feed these feature maps the auxiliary classifier networks to produce extra classification predictions:
P 91 (i
P = g2 (2
Pk-1 = 11 K-K)
p = ===1((F;) (F?), 2 (F
P
-1 K-
(5)
p 91 (F 55
(FF)......
......p 9K OK-IFF
We then apply the cross-entropy loss to all the classification predictions:
K-1 T
T
Σ
LcE(ph.)) + Σ Lce(p' .0)
L
(6)
K-T
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison results with different methods CIFARIO and CIFAR100.
Dataset
Method
Architecture
T
Train Method Accuracy (%
SlipReLU (Jiang et al.
2023
ResNet-18
4
ANN-SNN
94.59
Dspike (L1
2021
ResNet-18
6
SG
94.25
TET Deng
2022
ResNet-19
SG
94.50
SLTT Meng ct al
2023
ResNet-18
6
SG
94.59
CIFARIO
PLIF aan el al.
2021b
7-layer CNN
8
SG
93.50
TEBN Duan
2022
ResNet-19
6
SG
94.71
IM-Loss Guo
2022a
ResNet-19
SG
95.49
LSG (Lian et al.
20233
ResNet-19
SG
95.17
Deep IAGE: (Ours)
ResNet-18
4
SG
95.86
SlipReLU Jiang et al 2023
ResNet-18
128
ANN-SNN
78.55
Dspike (Li et al. 2021)
ResNet-18
4
SG
73.35
TET (Deng at. MPBN Guo
202
ResNet-19
4
SG
74.47
2023
VGG16
SG
74.74
CIFAR100
SLTT [Meng ct al. 20:23
ResNet-18
SG
74.67
STBP-wdBN Zheng et al. 2021
ResNet-19
6
SG
71.12
Sew ResNet l'ang el al. 2021a
ResNet-34
4
SG
67.04
GLIF Yao et 2022
ResNet-19
4
SG
77.05
TEBN Duan
ResNet-19
SG
76.13
Deep' Deep (Ours)
ResNet-19
SG
81.39
By implementing direct supervision multiple endpoints within SNNs, our approach further en- hances gradient flow the backpropagation path along multiple network stages and time steps, thereby facilitating training process and improving performance. Additionally, the auxiliary classification networks introduced can be removed after training without incurring extra computa- tional overhead.
EXPERIMENTS
In this section, we conduct thorough analysis and comparison of our approach against other state- of-the-art SNNs using both static and neuromorphic datasets. Furthermore, we validate the effec- tiveness of the proposed method through extensive ablation studies.
4.1 IMPLEMENTATION DETAILS
Datasets. We evaluate models using both static and neuromorphic datasets: CIFAR-10. CIFAR- 100 and ImageNet- 1 are static datasets, while DVS-CIFARI 10 is neuromorphic dataset. CIFAR-10 consists of 60,000 images size 32 x 32, distributed evenly across 10 classes with 6,000 images per class, CIFAR-100 has 100 classes, with 600 images per class, including 500 training images and 100 testing images. ImageNet-I1 1k much larger dataset containing over 1.2 million training images and 50,000 validation images. For static datasets, images are replicated across multiple time steps as input frames. DVS-CIFARIO an event-stream dataset derived from CIFAR-10 10, which captures pixel-level changes in brightness at high temporal resolutions, resulting in stream of events rather than static frames.
We train our SNN model using the Stochastic Gradient Descent (SGD) optimizer with a momentum 0.9. A cosine learning rate schedule is employed starting from 0.1 and grad- ually decreasing to O. The training process involves batch size of 32 over 320 epochs. Our SNN model employs Leaky Integrate- and-- -ire (LIF) neurons, setting the threshold voltage at 1 and the membrane potential decay constant 2. We utilize an NVIDIA A100 GPU for training and infer- ence on the CIFAR-100 and VVS-CIFAR1O datasets, while four NVIDIA A A100 GPUs are used for training the ImageNet dataset.
Training Setup.

---

Published as a conference paper at ICLR 2025
Table 2: Comparison results with different methods ImageNet.
Dataset
Method
Architecture T Train Method Accuracy (%)
TET Deng al.
2022
ResNet-34 6
SG
64.79
MS-ResNet (Fang
2021 la
RcsNet-18
6
SG
63.10
OTTT Xiao el al. 2022
ResNet-34
SG
63.10
ImageNet
Real Spike (Guo cl al. 2022b
ResNet-18
4
SG
63.68
Sew ResNet (Fang al. 202
ResNet-18
4
SG
63.18
MPBN (Guo et al. 2023
ResNet-18
4
SG
63.14
Deep'AAGE (Ours)
ResNet-18
4
SG
68.52
Table 3: Comparison results with different methods on DVS-CIFARIO.
Dataset
Method
Architecture
Train Method
Accuracy (%)
OTTT
Xiao
2022
VGG-11
10
SG
76.30
SLTT
Meng ct al.
2023
VGG-11
SG
77.30
DVS-CIFARIO STBP-ıdBN
ZZhnng el al.
2021
ResNet-19
10
SG
67.80
MPBN Guo el al.
2023
ResNet-19
10
SG
74.40
TEBN Duan
2022
7-layer CNN
10
SG
75.10
LSG
Lian al.
ResNet-19
SG
77.90
Deep IAGE (Ours)
VGG-11
SG
81.23
4.2 COMPARISONS WITH STATE-OF-THE-AR METHODS
CIFARIO. Table presents the comparative results of our method with the current state-of-the- art methods on the CIFAR1O datasets. Our method achieves an accuracy of 95.86%, surpassing TET's 94.50% with fewer time steps required. Moreover, our method also demonstrates superior performance compared to the leading ANN-SNN conversion method SlipReLU (Jiang et al. 2023)
CIFAR100. Table details our performance comparison with other methods on the CIFAR100 datasets. Utilizing spiking ResNet-19 model trained over four time steps, our method attains an accuracy of 81.39%, exceeding the Spatial Learning Through Time (SLTT) (Meng et al. 2023) method by 6.72%. Notably, even the ANN-SNN conversion method SlipReLU (Jiang et al. 2023). which achieves competitive 78.55% accuracy using 128 time steps, is outperformed by model in limited 4 time steps scenario. Additionally, our method surpasses recent advancements like the GLIF (ao al. (022) and TEBN (Duan al. 2022 methods by significant margins of 4.34% and 5.26% respectively.
TetgeNet. Experimental results for the ImageNet dataset using ResNet- backbone detailed in T Under direct training, our method surpasses the Sew ResNet (Fang et al.) (021a) by a substantial margin 5.34% Furthermore, compared to Real Spike (Guo al. 2022b) with the same backbone and four time steps, our approach achieves a significant accuracy ad advantage, reaching 68.52% versus 63.68%. These results demonstrate that our method remains highly effective even larger-scale datasets and within well-established architectural frameworks.
DVS-CIFARIO. Performance evaluations on the DVS- WWS-CIFARIO dataset, utilizing VGG-11 back- bone and spanning ten time steps, are shown in Table 3 Our method achieves an accuracy of 81.23%, significantly outperforming the LSG (Lian al., 2023) and MPBN (Guo et al.) (023) methods, which report accuracies of 77.90% and 74.40% respectively. These results demonstrate the robustness and efficacy of our approach when applied to neuromorphic data.
4.3 ABLATION STUDY
Gradient Enhancement and Spatio-Temporal Deep Supervision. We con- duct ablation studies both static and neuromorphic datasets evaluate the efficacy of Temporal- Aligned Gradient Enhancement (TAGE) and Spatio-Temporaa Deep Supervision (STDS) our method. In Table 4 reimplement the and SLTT (Meng et al. (223) as basclines to evaluate the effect TAGE across various models. On the CIFAR 100 dataset, applying TAGE to the Sew ResNet base model yields an accuracy improve-
(Fang et al. 202 TET (Deng al. 2022

---

Published as a conference paper at ICLR 2025
Table 4: Ablation studies of TAGE on different methods.
Dataset
Method
Accuracy(%))
Sew ResNet (ang et al.
2021a
77.89
w TAGE
4
78.80
TET Deng al.
2022
4
79,94
CIFARIO0
W/ IAGE
4
80.53
SLT Meng (002
74.46
W/ IAGES
76.36
VGG-11
10
78.20
w/ TAGE
10
79.60
TET (eng 2022
78.43
DVS-CIFARIO
w/ TAGE
79.84
SETT (Meng et al. 2023
10
77.67
w/ IAGE
10
79.75
Table 5: Ablation studies of TAGE and STDS in our method.
Dataset
Model
TAGE
STDS
Accuracy($6)
Sew ResNet
78.80
CIFARI00
Sew ResNet
V
v
81.39
VGG-II
79.60
DVS-CIFARI0
VGG-11
81.23
ment of 0.91 percentage points. Similarly, implementing TAGE on the TET results in accuracy increase of 0.59 percentage points, while its application on the energy-efficient SLTT improves performance by 1.9 percentage points. These improvements confirm the effectiveness of TAGE in optimizing SNNs across various network architectures and designs. For the neuromorphic DVS- CIFARIO dataset, the introduction of TAGE into the VGG-11 baseline model raises classification accuracy from 78.20% to 79.60%, demonstrating TAGE's effectiveness for SNN optimization on neuromorphic datasets with larger time steps. In Table 5 we further evaluate the effect of TAGE and STDS in our method, respectively. The accuracy of Sew ResNet increases from 77.89% to 78.80% and then to 81. 81.39% with the integration of TAGE and then STDS. For the VGG-11 net- work, the incorporation of TAGE and STDS results in accuracy increase of 3.03% in total. These results indicate that our TAGE and STDS methods can effectively improve the gradient- based SNN optimization and hence the overall network performance.
Firing Rate Statisties. To better evaluate the effect of our Deep Cemporal-Alignat Gradient En- hancement (DеepTTAGE), we further investigate its impact the firing rate of SNNs. In Figure 3 utilizing Sew ResNet19 as the backbone architecture, we analyze the firing rates at different layers and time steps both with and without DeepTAGEE (denoted as w/ DeepTAGEE and w/o DeepTAGE, respectively). As shown in Figure Jaa, Deep improves firing rates across nearly all network layers, especially in the shallow layers, where improvements more significant. Moreover, Fig- ure 3(b demonstrates that firing rates at different time steps have also improved and become more balanced.
Feature/Class Activation Maps. In Figure c)) we compare the feature activation maps and class activation maps before and after applying our methods (the feature maps of different channels are tiled on single map for visualization). Compared with the baseline Sew ResNet in the first row, the introduction TAGE and subsequent DeepTAGEE progressively enhance the activations of feature maps, which helps to improve their representational capabilities. Moreover, after using our gradient enhancement methods, the class activation maps are better focused on the key semantic regions of the input image. shown in the last column of Figure ())..
Training Convergence. Figures 4 (a) and (b) display the training loss and test accuracy curves over training epochs, comparing scenarios with and without the DeepTAGE method. The train- ing loss decreases more rapidly when applying DeepTAGE, while the test accurate shows a shaper increase, demonsrating that our method significantly speed up training convergence.
The Hyperparameter w. We also conduct an ablation study on the hyperparameter w, which is Table 6 presents the test accuracies for
used to calculate the normalization factor in equation g
8

---

Published as a conference paper at ICLR 2025
Dourhic
Convi
Stagel
Stage2
CAM
009
006
004
0.2
8
0.0
0.0
Soathea
Time step (b)
DeepTAGE
Figure 3: Analysis firing rates and feature activation maps using Sew ResNet19 the backbone. (a) Firing rates across different layers with and without DeepTAGE (denoted as w/ DeepTAGE and w/o DeepTAGEE. Dee significantly enhances firing rates, especially the shallow layers. (b) Comparison of firing rates across time steps, showing more balanced and improved firing rates with DeEE (c) Visualization feature activation maps and class activation maps before and after applying TAGE and DACE.
DeepTAGE
DeeptAGe
w/o Docs DooSTAGE TAGE
./O TAGE
Test accuracy
Train loss
40
42
35
150
250
300
Epoch
Epoch (b)
(a)
Figure 4: Comparing the loss and accuracy of models with and without DeepTAGE on the CI- FAR 100 dataset using the Sew 19 backbone: (a) demonstrates that the model with Deep- TAGE converges significantly faster across epochs compared to the one without. (b) illustrates that the model equipped with DeepTAGE consistently achieves higher performance than its counterpart without DeepTAGE.
Table 6: The ablation studies of w in equation 3
Dataset
hyper-parameter
T
Accuracy
1/2
4
80.96
w 1/3
81.02
CIFAR100
W = 1/4
81.39
t0 1/5
4
81.32
1/6
81.17 4
w
80.05
t = 1/3
10
81.14
DVS-CIFARI0
w 1/4
10
81.23
1/5
10
81.20
1/6
81.21
CIFARI00 and DVS- CIFARIO with different values of w. The optimal performance is achieved when 1/4, which is also the default setting in method.

---

Published as a conference paper at ICLR 2025
5 CONCLUSION
In this work, we present the Deep Temporal-Aligned Gradient Enhancement (DeepTAGE) method to tackle critical challenges optimizing Spiking Neural Networks (SNNs), specifically address- ing membrane potential distribution imbalances and vanishing gradients. DeepTAGE dynamically adjusts surrogate gradient functions based on the temporal distribution of membrane potentials, enhancing spike firing efficiency and network convergence. Our approach also incorporates deep supervision at multiple network stages and time steps to strengthen gradient flow, effectively allevi- ating gradient vanishing issues and boosting overall performance. Extensive experiments various datasets, including CIFAR1O, 10, CIFAR100, ImageNet- Ik, and DVS-CIFAR10, show that DeepTAAEE outperforms safe-of-theear SNN methods in accuracy without increasing computational costs or adding extra modules. These results confirm the efficacy of DeepTAGE as a scalable and efficient solution for SNN optimization.
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Major Project(2220AAA00055501 the National Natural Science Foundation of China (No. 62202469, 62403462,623300 (2230222223332222 1,6003662,,6206111, 6206601111 62192782, U2441241), Beijing Natural Science Foundation (L223303), the Project of Beijing Science and Technology Committee (No. 7231 100000003933344 The Key Research and Development Program of Xinjiang Uyghur Autonomous Region No. 2023B03024.
REFERENCES
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- conversion for high-accuracy and ultra-low-latency spiking neural networks. arXiv preprint arXiv:2303.04347 2023.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113:54-66, 2015.
Mike Davies, Narayan Srinivasa, Tsung- Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38( 38( 1):82999,, 2018.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202. 2022.
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective
batch normalization in spiking neural networks. Advances in Neural Information Processing Systems, 35:34377-34390, 2022.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056 21069, 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2661-2671, 2021b.
Charlotte Frenkel, David Bol, and Giacomo Indiveri. Bottom-up up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence. Proceedings of the IEEE, 2023.
10

---

Published as a conference paper at ICLR 2025
Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 326-335, 2022a.
Yufei Guo, Liwen Zhang, Yuanpei Chen, Xinyi Tong, Xiaode Liu, YingLei Wang, Xuhui Huang, and 7he Ma. Real spike: Learning real-valued spikes for spiking neural networks. In European Conference Computer Vision, pp. 52-68. Springer, 2022b.
Yufei Guo, Yuhan Zhang, Yuanpei Chen, Weihang Peng, Xiaode Liu, Liwen Zhang, Xuhui Huang, and Zhe Ma. Membrane potential batch normalization for spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19420-19430, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630-645, Cham, 2016a. Springer International Publishing.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016b.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2024. doi: 10.1109TTNNLS 20043333333..
Haiyan Jiang, Srinivas Anumasa, Giulia De Masi, Huan Xiong, and Bin Gu. A unified optimization framework of ann-snn conversion: Towards optimal mapping from activation values to firing rates. 2023.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 15:773954, 2021.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). URL htt:://ww. es. toronto. edu/kriz/cifar. html, 5(4):1, 2010.
Guoqi Li, Lei Deng, Huajing Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass.
Brain inspired computing: A systematic survey and future trends. 2023.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34:23426-23439, 2021.
Shuang Lian, Jiangrong Shen, Qianhui Liu, Ziming Wang, Rui Yan, and Huajin Tang. Learnable surrogate gradient for direct training spiking neural networks. IJCAL, pp. 3002-3010, 2023.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Towards memory-and time-efficient backpropagation for training spiking neural networks. arXiv preprint 1333:23022.14311, 2023.
Paul Merolla, John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (5197):6686673 2014.
Jing Pei, Lei Deng, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572 2019.
Kaushik Roy. Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- spike-based machine intelligence with neuromorphic computing. Nature, 575(7784)60776617, 2019.
11

---

Published as a conference paper at ICLR 2025
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion analog to spiking convolutional neural networks. arviv preprint arXiv.1612.04052, 2016.
Bodo Rueckauer,
Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con- version of continuous-valuee deep networks to efficient event-driven networks for image classifi- cation. Frontiers in neuroscience, 11:294078, 2017.
Catherine D Schuman, Shruti Kulkarni, Maryam Parsa, J Parker Mitchell, Bill Kay, et al. Oppor- tunities for neuromorphic computing algorithms and applications. Nature Computational Science. 2(1):10-19, 2022.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training nigh-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. Advances in Neural Information Processing Systems, 35:20717-2077 2022.
Man Yao. Jiakui Hu, Guangshe Zhao, Yaoyuan Wang. Ziyang Zhang, Bo Xu, and Guoqi Li. In- herent redundancy in spiking neural networks. Proceedings of the TEEE/CVF International Conference Computer Vision, pp. 16924-16934, 2023a.
Man Yao, Hengyu Zhang. Guangshe Zhao, Xiyu Zhang, Dingheng Wang, Gang Cao, and Guoqi Li. Sparser spiking activity can be better: Feature refine-ane refine-and-man ans- spiking neural network for event-based visual recognition. Neural Networks, 166:410-423 2023b.
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(8):9393-94) 9411,,991122000 2023c.
Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng, Glif: A unified gated leaky integrate-and-firg neuron for spiking neural networks. Advances in Neural Information Processing Systems, 35: 11160-2217, 2022.
Hanle Zheng. Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with ftrectly-traines larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11062-11070, 2021.
12


Reviews:
[{"review_id": "JDYfJbslYY", "reviewer": "Reviewer_Z7sN", "summary": "This paper proposes a new approach, Deep Temporal-Aligned Gradient Enhancement (DeepTAGE), which optimizes gradients in SNNs according to the distribution of membrane potentials at each time step. Additionally, it introduces Spatio-Temporal Deep Supervision, incorporating deep supervision across multiple network stages and time steps in SNNs. The effectiveness of DeepTAGE is validated on several popular datasets, achieving SOTA performance.", "questions": "See the 'Weakness'", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a novel approach to optimizing gradients in SNNs by leveraging both internal surrogate gradient functions and external supervision techniques. \n- The experiments are extensive and demonstrate strong results. \n- The writing is clear, and the paper is well-organized, making it easy to follow.", "weaknesses": "- The motivation for the paper could be strengthened. While the authors present the membrane potential distribution from time steps 1 to 4, it would be beneficial to explore how these distributions evolve over longer time steps. This would emphasize the importance of the problem and make the motivation stronger.\n- It would be better if the authors could include the membrane potential distribution after applying TAGE. This comparison would help illustrate the impact of TAGE on stabilizing or shifting the distribution and clarify its effectiveness in optimizing SNN training.\n- Regarding the STDS, my intuition suggests that applying CE loss to each stage may actually limit the ability of shallow layers to effectively learn elementary features. This may also create conflicts in the gradient directions across layers. Could the authors provide additional insights into why this approach might enhance training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzo6kidync", "reviewer": "Reviewer_62ou", "summary": "This paper claims increasing latency pushes the distribution of membrane potential away from the threshold, causing a reduction in the performance of the model.  An approach Temporal-Aligned Gradient Enhancement (TAGE), is proposed which adaptively adjusts the surrogate based on the distribution of the membrane potential.", "questions": "Could you please provide some results showing does the distribution shift still exists, after the model is trained with the modified surrogate? (Expected it should not exist)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to read, writing is good. \n\nThe observed phenomenon, shift in the membrane potential distribution away from the threshold with increasing latency is interesting. The proposed method considers this phenomenon and adjusts the surrogate while training. \n\nThe authors demonstrate the performance of their approach through empirical results on CIFAR10/100 and ImageNet datasets.", "weaknesses": "The phenomenon observed is interesting, however its not clear how and why this distribution shift is affecting the performance of the model. ?\n\nIn traditional methods(w/o DeepTage), the firing rate is low compared to DeepTage. Doesn't this cause the increase in energy consumption of the model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8H5r0DGqlR", "reviewer": "Reviewer_j3Kc", "summary": "This paper proposes the DeepTAGE method that encompasses two sub-methods. TAGE is used to dynamically adjust the surrogate gradient function based on the membrane potential distribution at different time steps, ensuring that each neuron obtains more effective backpropagation gradients. And STDS alleviates the problem of gradient vanishing by introducing auxiliary network after each SNN stages. On various datasets, DeepTAGE has achieved satisfactory results.", "questions": "1.How to interpret the sentence on line 168: \"Our approach is anticipated to not only stabilize the learning process but also to enhance the representational capacity of SNNs by improving their spike generation capability\"?\n\n2.Is DTAG applicable to other types of surrogate gradients?\n\n3.The ImageNet accuracy of ResNet-18, as shown in Table 2 of this paper, is remarkable. DeepTAGE significantly surpasses other methods and is only approximately 1% lower than the reported results of ANN. Could the authors provide the relevant parameters of the model?\n\n4.The accuracies of the spiking ResNet-19 in Table 1 and the Sew ResNet in Table 5 are both 81.39%. Is this a coincidence? What is the base model that Sew ResNet ueses?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Previous related papers often design different surrogate gradients for different layers, ignoring the distribution differences of time steps. But this paper focuses on the adaptability of surrogate gradients at different time steps and provides a good solution. Meanwhile, the results obtained in this paper are very satisfactory, especially the exceptional performance on large-scale datasets. The writing of this paper is clear and easy to understand.", "weaknesses": "1.The DTAG method lacks theoretical analysis of how it influence the SNN performance and firing rate.\n\n2.The auxiliary networks introduced by STDS generates extra training overhead. But the authors have neglected the discussion of the extra training cost.\n\n3.Concurrently, there is a lack of analysis regarding how DTAG can ensure the provision of effective gradients to the SNN.\n\n4.DeepTAGE will cause SNN to have an activation frequency, resulting in higher inference energy overhead. How to balance the energy and performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LkoCn8hlAN", "reviewer": "Reviewer_N3KN", "summary": "This paper introduces Deep Temporal-Aligned Gradient Enhancement (DeepTAGE), a method for optimizing SNNs. The key idea is to enhance the backpropagation of gradients by dynamically adjusting surrogate gradients based on membrane potential distributions at different time steps. DeepTAGE also incorporates deep supervision across both spatial and temporal levels to ensure balanced optimization. The method is validated through experiments on static (CIFAR10, CIFAR100, ImageNet-1k) and neuromorphic datasets (DVS-CIFAR10), demonstrating performance improvements over state-of-the-art methods without adding computational complexity.", "questions": "Please see weekness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper proposes a well-motivated and novel solution to the improve the optimization of SNNs, backed by comprehensive analysis and empirical validation. \n2.The integration of temporal-aligned gradient enhancement and deep supervision is effective, leading to substantial improvements in accuracy without increasing computational costs.", "weaknesses": "1.I noticed you tested the gradient enhancement method using the arctangent function. Can this method be generalized to other surrogate gradient functions?\n2.Could you provide more details about the structure of the auxiliary classifier networks? Like, how many layers do they have? And how does the number of layers impact the overall network performance? \n3.How does deep supervision affect gradient flow in the network? Is there a significant increase in the gradients being backpropagated through the shallow layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)