Spatio-Temporal Approximation: A Training-Free SNN Conversion for Transformers
Authors: Yizhou Jiang
Kunlin Hu
Tianren Zhang
Haichuan Gao
Yuqian Liu
Ying Fang
Feng Chen
Venue: ICLR 2024 poster (2024)
Keywords: Spiking Neural Network
Spike Calibration
Transformer
Primary area: applications to neuroscience & cognitive science
TL;DR: This paper proposes the first approach to convert pretrained Transformers to spiking neural networks without retraining or finetuning.

Abstract: Spiking neural networks (SNNs) are energy-efficient and hold great potential for large-scale inference. Since training SNNs from scratch is costly and has limited performance, converting pretrained artificial neural networks (ANNs) to SNNs is an attractive approach that retains robust performance without additional training data and resources. However, while existing conversion methods work well on convolution networks, emerging Transformer models introduce unique mechanisms like self-attention and test-time normalization, leading to non-causal non-linear interactions unachievable by current SNNs. To address this, we approximate these operations in both temporal and spatial dimensions, thereby providing the first SNN conversion pipeline for Transformers. We propose \textit{Universal Group Operators} to approximate non-linear operations spatially and a \textit{Temporal-Corrective Self-Attention Layer} that approximates spike multiplications at inference through an estimation-correction approach. Our algorithm is implemented on a pretrained ViT-B/32 from CLIP, inheriting its zero-shot classification capabilities, while improving control over conversion losses. To our knowledge, this is the first direct training-free conversion of a pretrained Transformer to a purely event-driven SNN, promising for neuromorphic hardware deployment.

Published as a conference paper at ICLR 2024
SPATIO-TEMPORAL APPROXIMATION: A TRAINING- FREE SNN CONVERSION FOR TRANSFORMERS
Yizhou Jiang'; Kunlin Hu Tianren Zhang', Haichuan Gao", Yugian Liu", Ying FangSt, Feng Chen 1.41
DDpartment of Automation, Tsinghua University, Beijing, China
TTingguu Shenzhen International Graduate School, Tsinghua University, Shenzhen, China *College of Computer and Cyber Security, Fujian Normal University, Fuzhou, China
'LSBDPA Beijing Key Laboratory, Beijing, China
(jiangyz20, huk122, 22 (taeeemmmlllsssninslee edu. cn, ghc2023
wet.maal..ssniiiii tsinghua. edu. en, 1 1iuyugian22emaill tsinghua. edu. en, Ey208fjnu. edu. en, chenfengemail www.aaii.siinnuua.ee...mm
ABSTRACT
Spiking neural networks (SNNs) are energy-efficien and hold great potential for large-scale inference. Since training SNNs from scratch is costly and has limited performance, converting pretrained artificial neural networks (ANNs) to SNNs attractive approach that retains robust performance without additional train- ing data and resources. However, while existing conversion methods work well convolution networks, emerging Transformer models introduce unique mech- anisms like self-attention and test-time normalization, leading to non-causal non- linear interactions unachievable by current SNNs. To address this, approximate these operations in both temporal and spatial dimensions, thereby providing the first SNN conversion pipeline for Transformers. We propose Universal Group Op- erators to approximate non- non-linear operations spatially and Temporal-Correcti Self-Attention Layer that approximates spike multiplications at inference through an estimation-correction approach. Our algorithm is implemented on a pretrained ViT-B/32 from CLIP, inheriting its zero-shot classification capabilities, while proving control over conversion losses. To our knowledge, this is the first di- rect training-fiee free conversion of a pretrained Transformer to a purely event-driven SNN, promising for neuromorphic hardware deployment. Codes are available at https://gttub..com/iiie ViviA.u/STA.
INTRODUCTION
The recent success large Transformer models has increased the need for efficient inference. Spik- ing neural networks (SNNs), the third generation of neural networks, use multi-step sparse spike accumulations instead of dense muttilyy-ccumulltion,,, providing significant advantages energy and speed. This makes SNNs prospective candidate replace ANNs for large-scale deployment.
Due to the non-differentiabiiity of spiking neurons, obtaining large-scale SNNs remains a challenge.
Existing method using surrogate gradients (Neftci et al. 2019 Lee al.) 2020 Zhu al.) (022)
or synaptic plasticity (Bicknell & Häusser 2021 Liu et al. (0022) requires training from scratch on large datasets, incurring high complexity, and still struggle to achieve high performance. Instead, in practice, limited training data and resources create more urgent need to directly convert powerful ANNs into equivalent SNNs in a training-free fashion (Dichl al. 2015). Such ANN-to-SNN con- version replaces ANN activations with temporal spike sequences, nearly preserving all capabilities of the source model. Thus, can directly reduce the inference power consumption open-source ANN models without other modification, even for those pretrained on large private datasets.
Nevertheless, such training-free conversion seem to impossible for mainstream large-scale ANNs based on Transformers (Vaswani et al. 2017 Dosovitskiy et al.) 2020 Radford et al. 2021). Their computational characteristics differs from convolutional networks, leading to two critical conflicts non-causal (Li et al. 2022). First, the matrix products between variable features in self-attention during inference, relying on complete input spike sequences. Such multiplications are incompatible with the additive accumulation time in SNN and thus cannot be directly calculated. Second,
"Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2024
unlike ReLU and BatchNorm in CNNs, operations such as GELU and LayerNorm in Transformers depend on complicated non-lincarities at test-time, so that cannot be accurately represented by the quantized piece- piece-wise linearity of spiking neurons.
Due to such inherent discrepancies, existing spiking networks cannot strictly implement Transformer operations through directly corresponding structure. Fortunately, the spatial population coding and temporal memory properties of SNNs can be further leveraged to enhance the representational capacity on both dimensions. By redefining spiking computations as a gradual approximation pro- cess to ANN floating-point values, we propose our conversion pipeline, termed Spatio-Temport Approximation (STA), consisting of novel spiking modules as universal approximators. Spa- tially, we adopt the strategy of trading space for precision, introducing local neuron populations to simulate precise non-iiirasss through multiple discrete binary spikes. These modules are driven by synthetic data regardless of their actual input at inference for universality. Temporally, to ob- tain stationary spike emissions for rate-coding, we remodel the non-causal multiplications into an sstimation-correctti process. Based on the accumulated input memory, first approximately estimate future reactions, then correct the results with the actual input as time progresses.
c
With our STA pipeline, we convert ViT-B/32 model pretrained on CLIP (Radford et al., 202T) into an SNN. The resulting SNN directly inherits the capabilities like zero-shot classification and trans- ferability from the large multimodal Transformer. It also achieves state-of-the of-the-art accuracy for SNNs on multiple benchmarks after supervised fine-tuning, Additionally, our converted SNN requires no floating-pon point operations, enabling energy-efficient deployment neuromorphic hardware.
In summary, our main contributions are as follows:
We propose Opatio-Tempon Approximation (STA), a training- free pipeline convert ANN Transformers to SNNs via universal approximations in both spatial and temporal domains.
We provide theoretical analysis on the error bounds and convergence rates of both key mod- ules in STA, proving their efficacy approximating ANN computation.
To our knowledge, we are the first to directly convert a pretrained mainstream Transformer (ViT-B/32 from CLIP) into SNN without additional training fine-tuning, while still retaining the generalization performance of the original model.
2 RELATED WORK
2.1
ANN-TO-SNY CONVERSION
Converting ANNs to SNNs is an active area of research for improving performance and training efficiency large-scale tasks (Dichl et al., 2015). whereby ReLU activations in ANN replaced
by "soft-reset" IF neurons (Rueckauer et al. 2017 Han et al. 2020). Its key directions include:
Training- free conversion is directly conducted pretrained ANNs through threshold balancing (Dihll et al.] 2015 Rueckauer et al. 2017), parameter calibration (Li et al. 2021) and functional spike emission (Wang al.) 2022a Li & Zeng 2022) to convert to SNN and calibrate by only few examples without retraining or fine-tuning Thus, they can be applied on high-performing open- source ANN models. However, these methods are mostly limited to CNNs, lacking applicability to Transformers (Li et al. 2022) and suffering from long simulation steps.
Training-dependent conversion tailors the ANN for SNN compatibility before conversion (Bu et al. 2021 Ding et al. 2021] Bu et al. 2022 Jiang et al. 2023 Hao et al. 2023), or fine-tunes the SNN after conversion (Wang et al. 2022b). Despite reducing conversion loss and latency, they entail greater training costs and weaker generalization, while maintaining CNN-like structural constraints.
Our work presents a training-free free approach that extends conversion beyond CNNs to Transformers. As spiking equivalents of Attention Blocks, our proposed modules approximates them spatially and temporally, thus retaining the applicability of large-scale pretrained models to complex scenarios.
2.2 TRANSFORMER AND SPIKE- BASED TRANSFORMER
Transformers have achieved impressive results on numerous tasks like natural language processing (Brown et al.) 2020) Devlin et al., 2018) and computer vision (Dosovitskiy et al.) (020) via the self-attention mechanism that captures global dependencies by aggregating features across spatial
2

---

Published as a conference paper at ICLR 2024
Input
LayerNorm
Attention
LayerNorm
MLP
Output
Non-Linear
1//I(x n)
Matmul Product
Q PKK
Non-Linear
Non-Linear GELU
Product
Scalar
k (x(x-1)
Nan-Linear
Softmax
Product
Scaler
Ee
Figure The modules and operators each Residual Attention Block of ViT.
dimensions. Transformers differ from CNNs in two key aspects: interactions between spatial features, and 2) complex non-linenn oo-/iinerritynorraizaaionn both not achievable by existing SNNs.
Spske-Based Transformers are recently proposed models for direct SNN training. Li et al. (2022)
substitutes the activations with spiking neurons but retains many floating- point operations. Zhou
al. (2022) intruduces purely spiking self-attention module by modifying the Softmax opera- tion. Zhou et al. (2023) presents the first fully event-driven Transformer through tailored residual connections. Additionally, Zhang et al. [2022a] by design specified Transformers for event-based cameras, which do not readily extend to conventional visual data. All these models differ from ANN Transformers structurally and require training from scratch, while our method directly lever- ages conversion to inherit capabilities from pretrained ANN Transformers without training.
3
PRELIMINARIES AND PROBLEM ANALYSIS
3.1 NEURONS FOR ANN & SNN
In ANNs using ReLU activation, for neurons in layer we denote their output as vector æ', and the weight matrix between layer | 1 and as W Ignoring bias, its floating-poin inference process is:
x'
maz (W'''-1 .). I= 1,
(1)
As for SNNs, similar Han 2020). we consider the soft-reset Integrate- and- (IF) neurons. When the 1-th layer receives weighted binary spikes x² 1-1 (t) E {0, 1}, the update rule is:
s (t) H (m' (t) v/h))
m ml((1 -pp p''
o oz
p'(t) = m m'(()))))))))))))))) dynn 2 xr(())
where (t) and p'(1) represent the potentials before and after the trigger of spike (1), v/, the threshold, and (-) is Heaviside step function. The firing rate is measured as the average number of spikes over time T', denoted as s¹. The converted SNN exhibits similarities with ReLU ANN on the activation values for each layer, i.e., x x s due to their comparable linear growth arithmetic.
3.2 OPERATIONS IN TRANSFORMERS
basic attention block in Transformer shown in Fig. | relying on two main types of opera- tions that differ from those in conventional CNNs for conversion. More details the modules in Transformer are provided in the Appendix[A]
1) Non-linear operators. While CNNs primarily ReLU activation for non- non-linearity, Trans- former involves more complex nonlinear functions like GELU (Hendrycks & Gimpel| 2016). square root, exponentiation, etc., which cannot be directly achieved by the piece-wise linear dynamics of IF neurons. This requires us to approximate their computational characteristics in the spatial domain.
2) Variable Scalar Matmul product. The inference in CNNs is conducted through variable fea- tures multiplied by constant weight matrices, while Transformers contain more variable-variable multiplications, such as the query-key products in self-attention. Additionally, LayerNorm in Trans- former computes normalization coefficients dynamically during inference, preventing integration into weight matrices as with BatchNorm in CNNs (Rueckauer et al., (017). Thus, computing these multiplications with spiking neurons is challenging and may require temporal modifications.
4 SPATIAL APPROXIMATION FOR NON-LINEARITY
As Transformer's floating-pont point non-linearity poses challenges for SNN conversion, our goal is developing spiking counterparts to simulate their spatial reactions. The proposed approximators
3

---

Published as a conference paper at ICLR 2024
Non- Non-Linearity
ANN Approximator
SNN with Spiking UGO
Grund Truth GELL AMN Approxim otor SAN UGO
f(x)
(<HLL())
y, fir(x,)
T-step
N neurons
simulation
M samples
x E D
X; E D,I = 1,...M
Figure 3: An approximated UGO for GELU with N 16, 16.
Figure 2: Spatial approximation process with UGO.
should: 1) consist only IF neurons, and 2) be universally applicable all operations, models and data. Due to the insufficient representation capability of each single neuron, we adopt groups of neu- rons to substitute individual operators. These approximators are pre-trained by synthetie floating- point data independent of real examples, and thus universally applicable to all scenarios.
4.1 NEURON GROUPS FOR UNIVERSAL. APPROXIMATION
We first examine common non-linear operators like GELU or square root that are low-dimensional with complicated computations. We note that with the Universal Approximation Theorem (Hornik al. 1989). single-layer ANNs can approximate these continuous functions over definite intervals. Further, ANNs with ReLU activation can be efficiently converted to equivalent SNNs. Therefore, propose the Universal Group Operator (UGO), small groups of spiking neurons for approximation. y defined on domain x E 'D be real continuous unary function. Its spiking universal group operator f comprises two fully connected (FC) layers surrounding a single hidden layer with N neurons, such that > where for any spike imput I with mean E, the output spikes ys satisfy yys y S c.
Definition 1 (Universal Group Operator). Let
The input and output layers have weights w1, W2 E R", and biases b E R" R,,b, b2 E R. respectively.
Construction. Three stages are required to obtain universal group operator, shown in Fig. 2
Data Synthesis. Due LayerNorm in Transformers, the input range of any function f always empirically restricted to a small continuous interval D, c.g., statistically, D = [-10, 10] for GELU. To enable the UGO to approximate f without real training data, we roughly synthesize mixture of uniform/normal distribution D that covers D, and sample M points {2;} from D to cover all possible inputs. The floating-point data pairs (a;, [x;)] serve the synthetic training data.
ANN Construction. We manually select suitable hyperparameter size N define the scale of an ANN fo based on the complexity of f. with typically N E [8, 32] for balanced accuracy and efficiency. It is then trained on the synthetic data using ReLU other tailored activation as in Jiang cet al. (2023) for approximation.
3. SNN Conversion. The pretrained ANN is finally converted to an SNN fur IF neurons over T time-steps using existing methods like Li al. (2021). Its conducts purely event-driven inference via spike accumulation and can directly replace its ANN counterpart with equivalent functionality.
The universal group operators thus allow implementation of all low-dimensional operations in Trans- formers for SNN conversion. As the synthesized data covers all possible inputs during inference, the pretrained UGOs are universally applicable to all test samples at high accuracy. Fig. 3 demonstrates a conversion result for GELU with N = 16, T = 16, and more details are in the Appendix B
Approximation Error Analysis. While bringing high efficiency, the small scale of UGOs also raise concerns about their accuracy and generalizability. To qualitatively analyze how the design impacts performance, we consider errors from three sources: insufficient sampling, limited parameterization and spiking quantization. This yields the following error bound:
Theorem (Error Bound for Spatial Approximation). For an optimal f*, the error e" satisfies
c o
(
N log log M M
+ o
(f1
(f& Iy
||m | |olma
bb Ilaee ||wa|||
(3)
N2
T
Empirical Gap
Parameterization Gap
Quantization Gap
4

---

Published as a conference paper at ICLR 2024
where y is the Lipschitz constant of f on D. Proof in Appendix. C
The terms correspond to the gap between function f. the optimal learner, the optimal fixed-scaled ANN, and its SNN counterpart. This theoretical analysis guides our implementation in two aspects:
1. ANN training: The Quantization Gap reflects that the two weighted layers contribute differently to the error depending distinct norms | |xr max + b1 billac and |/wy|11-- Thus, unlike common L1/ L2 regularizations, it is adopted as a layer-specifie regularization during training.
2. Hyperparameter determination: While larger M and T always improve performance, the optimal scale N depends on the case. Note that |22||2 can be scaled up to N-w2 max. all three gaps correlate differently with N, requiring experimental search for balance on accuracy and conversion loss.
4.2 INTEGRATION FOR HIGH-DIMENSIONA OPERATIONS
f(a*) 1/v0R+
By proposing the universal group operator, we have achieved event-driven unary operations. However, such scheme is infeasible for normalization func- tions like LayerNorm and Softmax, as their higher- dimensional input space cannot be sufficiently cov- ered by the synthesized training data as in UGOs.
(x- (x)²
To address this issue, we achieve them by integrat- ing three types of basic spiking operations. Take LayerNorm as an example, as in Fig 4 (and Soft- D). The ANN implementation is
scalar product
max in Appendix D).
x p
Vo² +
LN(2.) Va² +
+ B, where is small con-
stant, decomposed into the following parts:
Figure 4: Integration for LayerNorm.
1. Weighted addition: Simple, high-dimensional
computations such as zero-centering and variance for binary inputs via fxxe-weiigt linear layers.
2. Universal group operator: The normalization coefficient 1/Vo² + cooputed by a UGO.
3. Multiplication: Scalar Matmul product between two variables, to be achieved in Section
Such modular integration enables constructing high-dimensional spiking operators with UGOs, demonstrating the spatial aspect of our Spatio-Temporal Approximation pipeline. Nevertheless, per- forming variable multiplication in SNNs remains an unresolved issue due to its temporal characteris- tics. This computational requirement arises not just for normalization. but is critical for self attention in Transformers. Therefore, we next focus on the spiking implementation of multiplications.
TEMPORAL APPROXIMATION FOR MULTIPLICATIONS
Unlike conventional networks, the self-attention in Transformer performs multiplications between variable fcature matrices rather than fixed weights. During inference, these matrices are encoded by incomplete temporal sequences, so directly computing their product is non-causal. Naively avoid- ing this can lead to uneven spike outputs and performance degradation. To address this, we pro- pose
Cooreetive Self-Attention Layer (TCSA), employing an estimation-correctio mech- anism. The product is first estimated using the temporally available sequences, and then corrected by the next actual spike input. This distributes each spikes' contribution to the product across all time steps, smoothing the output for enhanced stability of multiplication.
5.1 TEMPORAL SPLIT FOR SPIKE-BASED MULTIPLICATION
To analysis this problem, first consider basic matrix multiplication A B. For simplicity, assume a matrix M with shared scalar threshold v, for each element is split into a spike sequence M, (t) e (0. 1),1 1,...,? T. In conventional architectures, such operations typically oceur between fixed-
5

---

Published as a conference paper at ICLR 2024
Naive Temporal Split (Pt)
Corrective Temporal Split
(Q:3
A
A
/
t=i-1
1-1
Q..-TT
... B
B
...
B
estimation
correction
A
A
19 ((- 11%
t=i
20
P
O0
* Q-
B-1
B.
(2i 1)
AB
B--1
B.
AB
improve stability
Sequence Mean
AB P.T
keep equivalence
AB 4,
Q-
Figure 5: Spike multiplications with naive temporal split and estimation-ccorreciion mechanism.
weight matrix W and binary variable features X, computed
T
UP Σ Wx.(1).
WX = W X,
(4)
T
T WJ (t) are used as weighted spike output at each step, and are accumulated for result.
In contrast, for common inter-variable multiplications in Transformer such as query-key products, the operations are rather different. Note that before the imput at step t, both matrices are incomplete, with only inputs at [1, t available in their temporal split sequences.
Definition (Naive Temporal Split for Causality). Let A, B, A., B, be two variable matrices and their encoded spiking sequences in T steps with thresholds Vo. vb. The temporary product ((t) is the sum of all currently available binary terms in the matrix product at step considering causality:
((t) A Σ
A.
B.O.
Σ A.ii)..i).
(5)
5=1
1-=1
Since Φ(t - 1) is available before step t, the increment s(t) to obtain (() is defined as below:
6(1) - ((1) -) A.(()..() + (. Σ B.(1) Σ A,))),,(,,,
(6)
which uses only Boolean ANDs and additions. Accordingly, let P(t) e 2ath (tt) be the output at t:
T P
P= P= Σ P(O = P()=
Σ Da Ub
Va T2
$ (T) AB,
1=1
which aligns with the objective of ANN-1o-SNN conversion.
5.2 ESTIMATION-CORRETTION FOR FIRING-RATE STABILITY
Although the naive method in Def 2 maintains numerical equivalence in the conversion, its output P(t) contains 2t - 1 terms due to the incomplete sequence temporarily. This implies linearly growing magnitude over time, leading to uneven firing rates along the time dimension. As these spikes propagate, the large inputs in the last few steps make subsequent neurons hoard substantial residual membrane potential, preventing effective spike emission. To mitigate such instability, is necessary to estimate the distribution of future input spikes earlier on, so as react proactively.
Considering the temporal consistency of rate-coding, we propose that by regarding the available sequence at as t-point sampling of the complete T T-step simulation, the overall firing rate can be approximated by that of shorter t-step time interval. The estimation is thus defined as:
Methodology.
Theorem (Temporal Estimation). The unbiased estimations of A and product AB at step t are
Á(I)
Pa Σ A.(1), 1
v(() (())(())))))))))))) Vg
((t),
(8)
12
Such estimation provides two key benefits: 1) Guaranteed evenness: As EV = AB for any t, the estimation is independent of /. with small temporal variation, resulting in sparse spike outputs.

---

Published as a conference paper at ICLR 2024
2) Progressive approximation: Since lime-r (t) imates the exact statistic for the full sequence. Each step's 's output brings the estimate closer to the final result. Thus, we propose:
(T) AB, the estimate gradually approx-
Definition 3 (Temporal Correction). The corrective increment Q(t) as the output sequence is:
Qtt) - (t) (t - 1) - -----))-- Ub
- iit + o(t)
(9)
where all computations are Boolean ANDs and their weighted additions, such that
T Σ -(1)=)10 (T) = AB. Q T
(10)
This mechanism is the core of our Temporal-Correctte Self-Attention Layer as a spiking self- attention module, and also similarly adopted in Section 4.2 for multiplications. practice, spike multiplications are always constantly weighted, e.g., A. WA BB and the weights of additions at each step can be pre-integrated into the linear layers W before inference. Thus, the computations remain hardware friendly. Moreover, our estimation-correcion algorithm allows reusing accumulated (t) values from prior time steps during the update, reducing computations.
Estimation Error Analysis. The performance of our corrective multiplication method relies heavily on accurate estimation. We quantitatively analyzed how our estimate converges to the ground truth over time steps. Considering that all multiplications are obtained from scalar multiplications, for clarity, we assume all elements are independent with a threshold U/Λ =
Theorem (Convergence Rate of Temporal Estimation). Assuming two independent foating-poin elements & b, and their converted Ttep spiking sequence follows stationary independent pro- cess with Ta & Tb spikes emitted. Denote the number of arrived spikes by step t as x, the estimated (t) satisfy: (Proof in Appendix
ab(1 aa) b) (T - 1)2
(T
1 T
E (V) = ab,
ID (V())
α
(11)
It demonstrates the estimation error decreases quadratically with initially, then stabilizes in the final few steps. This mechanism acts as smoothing filter, providing the temporal component of our Approximation pipeline.
IMPLEMENTATION AND EXPERIMENTS
To demonstrate the advantages of our training-free Transformer conversion approach, we apply our pipeline to the Image Encoder of CLIP (Radford et al., 202T). prevalent Language-Image model. This allows converted model to leverage CLIP's powerful generalization abilities such zero- shot classification. In comparison to conventional ResNet architectures, Transformers can better exploit large-scale pretraining achieve superior performance. Furthermore, for fair comparison with existing methods, we tune the pretrained ViT on benchmarks like CIFAR and ImageNet, achieving state-of oftte--f-hhearrr results of SNN with smaller conversion error and faster simulation.
6.1 CONVERSION IMPLEMENTATION
Our work enables all Transformer computations in SNN to be conducted without specified con- version methodology. In practice, we combine prior techniques complete the entire conversion, including MMSE (Li et al., 2021) to determine optimal neuron thresholds, signed neurons (Wang al. 2022a) to handle negative weighted inputs, and burst spikes (Li & Zeng. 2022] mitigate lagging inputs and reduce residual potentials. Implementation details are provided in Appendix
6.2 ZERO-SHOT CLASSIFICATION
Settings and Models. CLIP is a multi-modal ANN trained on image-text pairs with diversified Image Encoder backbones including ResNet and Vision Transformer (VTT). It performs various
7

---

Published as a conference paper at ICLR 2024
Table Comparison with other backbones and baselines on zero-shot classification of CLIP.
Dataset
Model
Method
ANN Acc.
T=32
T=64
T= 28
T=256
Calib. (Li ct al. 2021
64.08
68.13
71.04
71.19
ResNet-50
72.35
SNM
Wang et al. (022aa)
58.69
61.22
70.68
70.88
10
Calib.
[Li et al. 2021
38.21
55.37
67.44
71.21
ResNet-101
79.64
SNM
Wang
al. 2022a)
43.25
52.68
68.42
72.96
ViT-B/32
STA (Ours)
89.74
87.71
88.20
88.29
88.34
Calib.
(Li et al.
2021
24.67
33.41
38.20
39.01
ResNet-50
41.01
CIFAR-100
SNM Wang et al.
(022a)
35,64
34.71
39,95
41.13
ViT-B/32
STA (Ours)
64.26
62.55
62.74
62.98
63.01
Calib. (Li al. 2021
22.50
34.51
41.82
42.03
ResNet-50
45.63
ImageNet-200
SNM Wang al. 2022a)
25.43
38.17
42.25
42.95
ViT-B/32
STA (Ours)
62.25
59.79
61.24
61.53
61.66
Calib. (Li et al. 2021
61.01
63.44
64.39
64.42
ResNet-50
65.05
CIFAR-10.1
SNM Wang al. 2022a)
44.56
58.26
63.53
64.06
ViT-B/32
STA (Ours)
84.15
83.05
83.25
83.58
83.52
Calib. (Li et al.
2021
58.97
61.01
62.50
62.68
ResNet-50
63.90
CIFAR-10.2
SNM Wang al. 2022a)
46.83
54.68
62.94
63.08
ViT-B/32
STA (Ours)
80.35
78.55
79.65
79.77
79.83
tasks based natural language prompts. Since existing methods directly convert Transformers, we use pretrained ResNet-50 backbone for our baselines. Following standard CLIP configuration for zero-shot prediction, we evaluate on CIFAR-I0/100 ImageNet-200 benchmarks, and distribution- shifted CIFAR-1001/10. datasets. Details in Appendix G.1
Classification performance. The results in Table Sablee[sho show that the converted VIT model substantially exceeds ResNet across all datasets and time settings. This confirms that large-scale pretrained Trans- former are superior convolutional networks for zero-shot classification, emphasizing the value of SNN conversion targeted on Transformers over CNNs.
Accuracy loss from conversion. Despite having more parameters than ResNet-50 (87.8M vs 25.6M). our ViT model still experiences much lower accuracy drop after conversion. Two main factors contribute: 1) Self-attention layers have lower precision requirements than convolutions, making them less prone to numerical errors. 2) Transformer architecture provides more robust fea- tures with larger label margins, maintaining predictions even under conversion perturbations.
Limitations of existing works. We make two key observations: 1) like ResNet-101 do not improve SNN conversion performance over ResNet-50, as their ANN ac- curacy still lags behind VIT while depth exacerbates conversion errors. This highlights the need for advanced architectures like Transformers. 2) Many current conversion methods only succeed on models like resnet-20 VGG-16, while being incompatible with deep residual networks. Thus selectively demonstrate those with better ResNet-50 results from CL.I..
Larger convolutional networks
6.3 STANDARD CLASSIFICATION AND ABLATION STUDIES
Standard Classification. We fine-tune our ViT on benchmarks and compared its performance on con- ventional image classification tasks resnet-20 and
100
80
1.61
1.79
ANN
Accuracy
non-spiking UGO
pretrained ResNet-50 baselines from CLIP. Table shows results on CIFAR-100. with other results on CIFAR-10 ImageNet in the Appendix G.2 Com- pared to other conversion methods, our algorithm achieves near peak accuracy with fewer steps (I 32 or 64), while most baselines require over 128 steps for optimal accuracy. The remaining small accuracy gap to ANN VIT is largely due to the unavoidable approximation error from the Universal Group Operators. This demonstrates the faster simulation time advantages of our approach.
Gaps
corrective spltt
UGO
spiking UGD+ spiit
20
Correction
Figure
Ablations on components in CIFAR-100, T=32.
Ablations. We also conduct ablation experiments to analyze the spatial and temporal impact in our pipeline, Fig Our results lead the the following conclusions: 1) UGO nearly eliminates the three
8

---

Published as a conference paper at ICLR 2024
Table Comparison with other backbones and baselines on standard classification of CIFAR-100
Model
Method
ANN Acc.
T=32
T=64
T=128
T=256
RMP Han et al. 2020)
30.60
42.61
62.59
69.86
TSC Han & Roy
2020
35.87
49.70
65.42
70.59
Opt. (Deng & Gu
2020
49.81
69.82
75.75
75.94
resnet-20
76.12
Calib. L1 ct al. 2021)
74.25
75.08
75.58
76.24
SNM Wang ct al.
2022a
74.58
75.89
76.11
76.18
Burst Li & Zeng
2022)
71.14
75.50
75.89
76.03
Opt. (Deng & Gu
2020)
64,48
71.71
76.67
79.52
ResNet-50
Calib. [Li et al. 2021)
81.13
75.61
77.29
78.13
80.02
(CLIP)
SNM
Wang et al.
2022a)
68.24
75.30
77.91
80.75
ViT-B/32
STA (Ours)
87.35
84.15
85.25
85.69
85.98
Gaps in 3 thereby retaining nonlinear computation capabilities after spatial approximation. 2)
The cooiaattioncorrcctiin mechanism for temporal multiplication prevents large residual potential accumulation caused by output lag, thus significantly improving performance over the naive method.
6.4 ENERGY ESTIMATION
The energy efficiency of SNN stems from two aspects: 1) Sparsity and event-driven computation, where only small fraction of synapses active during inference. 2) Low- Low-power synaptic opera- tions like Boolean logic and weighted additions instead of expensive floating-poin operations. The consumption of ANN inference is characterized by floating-poin operations (FLOPs) with energy cost AC while SNNs rely on synaptic operations (SOPs) with EAC- Therefore, the ratio of inference energy for SNN versus ANN for module is estimated in Rathi & Roy (2020)
ESNN SOPs EAG
with FM F 4.6J, FAC R 0.9.J
(12)
EANN FLOPs
EMAC'
Using empirical firing rate denoted 1), analyze both components in pipeline:
Universal Group Operator. unary non-linear operator like GELU requires FLOPs R 70 pri- marily due to exponents in tanh, while UGO with N neurons requires SOPs = 2NTT For high accuracy implementation with N = 32, T = 32, r) R 9.1 UGOs reduce computational costs by 41% compared to GELU. This saving is further amplified in high high-dimension operations.
Spike Multiplications. We illustrate this with the N X N query-key matrix products, where FLOPs 3. While naively implementing matrix multiplication requires O(T2) spike prod- ucts, our proposed TCSA layer reduces complexity to O(T) with accumulated @(t). Specifically, SOPs 4 With 7) E [3%, 13%] at T 32 across all 12 blocks, the attention modules achieve 33% savings on average, up to 75% for the sparsest cases.
Admittedly, due to the unique computational demands of Transformer, its energy savings from SNN conversion are not superior than convolutional spiking networks. However, our work still demon- strates potential for low power usage: training UGOs with sparsity constraints optimizing multi- plication estimations could further reduce the 1j in our Spatial-Tempon
Approximation pipeline. In addition, the latest hardware (Pei al. 2019) allows utilizing both seating-poimi and event-driven computation synergistically, thereby further improving energy performance.
7 CONCLUSION AND DISCUSSION
For the first time, this paper establishes a bridge between mainstream pretrained Transformers and SNNs. By designing novel spiking operators and layers, we approximate Tranformers in both spa- tial and temporal dimensions in a purely event-driven fashion, breaking with convention. Since all Transformer- based models share similar computation modules, our proposed pipeline is broadly ap- plicable to various language and vision models, including the Text Encoder in CLIP, or even Large Language Models, as our subsequent work. These pretrained large models are often transferable without additional training fine-tuning, and our training-free conversion pipeline avoids perfor- mance degradation, promoting practical SNN usage on various downstream applications. While the converted ViT has slightly higher computations than conventional spiking CNNs, it provides stronger performance and robustness with fewer simulation steps. This enables potential energy- efficient deployment of open-source large models in the future with neuromorphic hardware.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China under STI 2030-Majjj Major Projects 20212D02200300, and in part by the National Natural Science Foundation of China under Grant 62176133, and in part by the Siinhuu-Guugqian research pro- gram under Grant 2019GQG0006 and in part by the Natural Science Foundation of Fujian Province, China, under Grant 2022J01656.
REFERENCES
Peter L Bartleut, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, (0(11)2285-23..1
2019. 2019.
Brendan A Bicknell and Michael Häusser. A synaptic learning rule for exploiting nonlinear dendritic computation. Neuron, 109(24)::001-4017,, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- conversion for high-aceuracy and ultra-low-latent spiking neural networks. In International Conference on Learning Representations, 2021.
Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low- latency spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11-20, 2022.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. International Conference on Learning Representations, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-clasiifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference neural networks (IJCNN) pp. 1-8. icee, 2015.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint arXiv:2105. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision, pp. 188-404. Springer, 2020.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high- agghracc and low-latency spiking neural network. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13558-13566 2020.
Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion error through residual membrane potential. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11-21, 2023.
Dan Hendrycks and Kevin Gimpel. 1706.0660..04111 2016.
Gaussian ertor linear units (gelus). arXiv preprint
Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural nefworks, 2(5):359-366, 1989.
10

---

Published as a conference paper at ICLR 2024
Haiyan Jiang, Srinivas Anumasa, Giulia De Masi, Huan Xiong, and Bin Gu. A unified optimization framework of ann-snn conversion: Towards optimal mapping from activation values to firing rates. In International conference machine learning. PMLR, 2023.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Fron- fiers neuroscience, pp. 119, 2020.
Yang Li and Yi Zeng. Efficient and accurate conversion of spiking neural network with burst spikes. arXiv preprini 1rXi::2204.1371,, 2022.
Yudong Li, Yunlin Lei, and Xu Yang. Spikeformer: A novel architecture for training high- performance low-latency spiking neural network. arXiv preprint arXiv:2211. 2022.
Yuhang Li. Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International conference on machine learning. pp. 6316 6325. PMLR, 2021.
Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. aattogically-plassbbee backpropagation through arbitrary timespans via local neuromodulators. Advances in Neural Information Processing Systems, 15:17528-17422 2022.
Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 536)):56555506, 2021.
Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? closer look distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robustness in Deep Learning, volume 5, pp. 15, 2020.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrggate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 13, 2019.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):06-111 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Nitin Rathi and Kaushik Roy, Diet-snn: Direct input encoding with leakage and threshold optimiza- tion in deep spiking neural networks. arXiv preprint arXiv:2008. 200.::0000006888, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar- 10? arXiv preprint arXiv: 180618000055 00451, 2018. https::/erxiv www.vvorg/abss...
18C6. 00451
Bodo Rueckauer, Iulia-Alexandrr Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con- version of continuous-vale valued deep networks efficient event-driven networks for image classifi- cation. Frontiers in neuroscience, 1 11:622, 2017.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19511))195811900 2008
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2024
Yuchen Wang, Malu Zhang. Yi Chen, and Hong Qu. Signed neuron with memory: Towards simple, accurate and high-efficient ann-snn conversion. In International Joint Conference on Artificial Intelligence, 2022a.
Ziming Wang, Shuang Lian, Yuhao Zhang, Xiaoxin Cui, Rui Yan, and Huajin Tang. Towards loss- less ann-snn conversion under ultra-low latency with dual-phase optimization. arXiv preprint arXiv:2205.07473. 2022b.
Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 8801-8810, 2022a.
Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In European Conference Computer Vision, pp. 34-52
Springer, 2022b.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-basee -ased spiking neural net- work. arXiv preprint arXiv:2304. /1954, 2023.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint 2rX1v:2209.15425. 2022.
Yaoyu Zhu, Zhaofei Yu, Wei Fang. Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Train- ing spiking neural networks with event-driven backpropagation. Advances in Neural Information Processing Systems, 35:30528-30541, 2022.
12

---

Published as a conference paper at ICLR 2024
Appendix
A MODULES IN TRANSFORMERS
This section provides detailed overview of all the modules, operators, and formulas in the Residual Attention Block of the Transformer. Taking VIT-B/32 as an example, it consists one convolutional layer. 12 sequentially connected Residual Attention Blocks, and final linear layer. Each Attention Block contains the following modules:
1. Multi-Head Attention is the module of each block, allowing the model to jointly attend to information from different representation subspaces. In self-attention, the same input vector is first projected h times into queries Q, keys K, and values V using different learned linear projections. Attention is then performed in parallel for each projection:
H Head; = (Q;
for pori=1....
(13)
The outputs of the heads are concatenated and projected once more by W to get the final values:
MultiHead( Q, K, V) Concat(Head,... Head,, W
(14)
The uniqueness of Attention is that it performs a large number matrix multiplications between feature matrices, such X Q.K," which is different from the multiplications with constant weight matrices like W0
In addition, the Softmax function is applied to the result of the query-key multiplication for the attention weights in each heads. Specifically, Softmax normalizes the attention weights to output probability distribution:
Softmax(I:) Σ, E
(15)
which requires nonlinear operations like exponentiation and inverse.
Layer Normalization (LayerNorm). is used twice in each block before and after the attention module to stabilize and accelerate training. It normalizes the activations of each layer by subtracting the mean and dividing by the standard deviation:
LN(x,) = 24
+ +3,
(16)
V
where y and 2² are the mean and variance calculated over all hidden units in the same layer.
Notably, unlike BatchNorm which tracks global statisties during training across entire channels, LayerNorm normalizes each element independently. This normalizing occurs at both training and inference, requiring dynamic statistics, which hinders weights absorption into matrices compared to BatchNorm for SNN conversion. Therefore, it also requires more complex nonlinear operations like square root and inverse at inference.
3. Gaussian Error Linear Unit (GELU) is used as the non-linear activation function in the MLP in each self-attention block. It applies the cumulative distribution function of the Gaussian distribution to each input element 24, and is approximated as:
QuickGELU<r)) = x aigmoid(1.702:n)
(17)
which allows gradients to flow efficiently through the activation during backpropagation.
Input
LayerNorm
Attention
LayerNorm
MLP
Output
Non-Linear
= 1//E(x>> R)
Metmul Product
Q
Non-Uinear
Non Lon-Linear
GELD
ko (x
Product
Non-Linear
Softmax
Product
New
Figure 7: The modules and operators in each Residual Attention Block of ViT.
13

---

Published as a conference paper at ICLR 2024
B SETTINGS OF UNIVERSAL GROUP OPERATOR
The Universal Group Operator achieves four main operations: 1) Exponentiation Softmax, 2) GELU in MLP, 3) Inverse in Softmax, 4) Inverse of Square Root LayerNorm. The settings required for training a UGO are organized as follows:
1. Data Synthesis. The synthesized distribution D and the sample number M.
2. ANN Construction. The hidden-layer size N, the loss function, optimizer and scheduler.
3. SNN Conversion. The selected threshold Ven-
These settings are summarized in Table where the Loss Penalty refers to the Quantization Gap in [ To empirically demonstrate 'D practice, the inputs of each module under real sampling conditions are provided in Fig
The fitting results of Universal Group Operators implemented in algorighm are shown in Fig 9
Table 3: Hyperparameters and settings for UGO training.
Exp
GELU
Inverse
LayerNorm
D
U(-35,3) 50%
U(-25,25) 50%
U(3,38) 75%
U(0.01,1) 100%
U(-12,2) 50%
U(-1,1) 50%
U(2,75) 25%
M
164384 samples 128 batch * 1000 epoch
N
32
32
Loss
Huber
Huber+Penalty Huber+Penalty
MSE
Optim
LR SGD-0.01
SGD-0.01
SGD-0.01
Adam-0.01
MultiStepLR
StepLR
MultiStepLR
CosineAmeealiggL.
Scheduler
milestones=(500,800
step=100
milestones=
0.5
= 0.5
0.1
Vib
Determined by
Li al. (022))
Exp
GELU
Inverse
LayerNorm
Probablity
0.02
0.2
0.02
0.05
0.01
0.00
0.00
0.00
0.0
-20
-10
-10
0.0
0.5
1.0
2
Figure 8: Empirical input distribution D sampled from 10 CIFAR-10 input images.
Exp
Inverse
Sertinv
10.0
10
fx
0.8
7.5
8
ANN
SNN
4
0.2
2
0.0
-20
-15
-10
-5
0
-10
-5
10
0.25
0.60
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 9: Fitting Results for the Universal Group Operators.
14

---

Published as a conference paper at ICLR 2024
C PROOF FOR THEOREM
Theorem (Error Bound for UGO), For an optimal f*, its approximation enror é" satisfies
log N log M
C ly
|
b -bll |wall1
E" s'
max
(18)
M
N2
T
Empirical Gop
Porameterisation Gap
Quantization Gop
where y is the Lipschitz constant of f on D.
Proof. We decompose the error into three gaps:
The Empirical Gap between f and the optimal learning machine fm. due limited M and model complexity regarding to N.
The Parameterization Gap between fm. and single-layer ANN fn with N neurons due to the limited parameters determined by N.
The Quantization Gap between fn and the UGO due to spiking discretization regarding to T.
Empirical Gap. We first quote lemma from Bartlett et al. 2019
Lemma For deep neural networks with arbitrary piecewise linear activation function where W is the number of weights and L is the number of layers, its VC-dimension is bounded by (lWWlog(WL))) and O( L Lo).
For fm. we have L = and W = 2N, thus dvc O(N log(N)) According to the classical conclusion Vapnik (199), the empirical gap between fm and with samples is:
dvc log dve -
N N N log M @
Cemp
(19)
M
M
Parameterization Gap. Considering that the parameter quantity 2N of the ANN is much smaller than the sampling quantity M in practice, the ANN fa cannot empirically all data, leading to gap between fn and fm We modify the conclusion from Lu et al. (221)
Lemma 2. For deep ReLU networks with width and depth approximating f E C((O, 1))d with Lipschitz constant L the optimal approximation error is (Lg. N-2/d, 1,2/6.11-2/0)
Accordingly, the gap in implementation for f is
1yl Wlaa O (C, (CG.NN-) 2 (cr Iylmaxx
Cparm
(20)
N2
Quantization Gap. When converting the ANN fn to an SNN, we set the threshold as the maximum output of neurons to avoid truncation emrors:
V max(wi b1) < [w [2| max + b al
(21)
The quantization error the IF neuron outputs
T so the error of the result is:
Ven ||
|| | Ialmax
+ b alae ||
Cquant
(22)
T
can also be generalized to:
[22
Equantt
T
|mll + bi 1100
(23)
Combining Eq
19
20 22
the result is proved.
15

---

Published as a conference paper at ICLR 2024
f(ot) 1//0²++
fun (ee))
ge
data clamping.
e
scalar product
w)²
scalar product
(xp( efi
x-P
data damping
Voa +
data offset
Figure 10: Integration for Layernorm.
Figure 11: Integration for Softmax.
D DETAILS OF HIGH-DIMENSIONAL OPERATIONS
We have roughly introduced the decomposition process of high-dimensional operations represented by LayerNorm in 4.2 this section, the implementation details of LayerNorm and Softmax will be explained.
D.I LAYERNORM
In order adapt LN(x) = = Vo"te
æ-p + + to SNN computations, we unroll it into the following steps:
1. Calculate the decentralized inputs 1 1
" Ei= o; Wde x, (0, 1}.
2. Calculate self-scalar product of ni i.e. h. (Wds (Wde.)), which can be imple- mented with TCSA in SNN.
3. Calculate the variance x² = h = W (Wdcx)) (Wdd
4. Approximate the inverse of the standard deviation via (spiking) UGO, i.e. Vo
f(a²) = et.y. y/ E {0,
5. Calculate scalar product of the inverse and the decentralized inputs with factor 7, 3 to get #-p LN(x) = vor4e + 3 x= yuu (Wdcz) o y + 8, implemented with TCSA in SNN.
D.2 SOFTMAX
Like LayerNorm, Softmax can also be roughly decomposed into three suboperations: weighted summation, UGO approximation, and split multiplication. Specifically, shown Fig Softmax is unrolled into the following steps:
1. Translate the inputs to [- 1] by subtracting an offset (Wen):))ma (Ves can be neuron- wise and (Ut1 mar is usually in dimension 0) to ensure that no overflow occurs during the exponential operation. This translation has no effect on the result, as it will be cancelled out by the numerator and denominator in the subsequent division. Similar to LayerNorm, 2; E {0, 1}.
2. Clamp the translated inputs to suitable range that the exponential UGO can handle, avoid- ing UGO output exceptions due to too small or too large input.
3. Approximate the exponential function value via (spiking) UGO, i.e. et ~ fexp(a;) =
Uth y/s
exp exp E 10,
4. Calculate Ei= e Σm 1hh U/
ExP exp
5. Clamp Σn e to suitable range that the inverse UGO can handle.
6. Approximate the inverse via (spiking) UGO. i.e. S c. Po finv (E e²t) = with
inv inv
y inv E 1).
7. Calculate scalar product of the inputs and the inverse get Softmax(x)) ann iivv x.
En e*r
16

---

Published as a conference paper at ICLR 2024
E PROOF FOR THEOREM 3
Theorem (Convergence Rate of Temporal Estimation). Assuming two independent floating-point elements & b, their converted T-step spiking sequence follows with Ta & Tb spikes emitted. Denote the number of arrived spikes by step as H, the estimated (t)
a stationary independent process
satisfy:
E (v())) = ab, D = ab(1 a b)
(r-1)²
(24)
1T
Proof. Considering single scalar a, let (t) denote the number spikes from sequences a, that have arrived by time Given (T) Ta, the probability of emitting x spikes in the first t steps is:
((N(() 2² (T) Ta)
=P(N(T) = Ta)N(() = ((N(() = x)
( Ta)
(N)(t) = x) =P(N(T - -Ta- ((N = Ta)
(
(Tax) - x
(25)
Ta
For the expectation of and corresponding estimation a(t):
E(x) Σ
Ta
T-t
Σ
Ta
T-f (t11
x--1
T
T'-1
Ta
ta
(26)
T
T
Ta
Ta
E(a(t)) ((x)
a.
(27)
The second order is similarly derived as:
Ta-ml (z)
E(x²))
Ta
=
Σ
1)
Ta-r
Ta
Ta
Ta
(t-2)
Ta-a) (-22
=t(t
+ ta
T
Ta
=ta(t - 1) (Ta - 1)
(T [ + ta.
(28)
For the variance:
(1 a)(T - f)
((x) = E(x²)(())))) - E2 (z) (2 = ta
(29)
T -
a(1 - a) D(r)
T
D (ā(())
(30)
t2 T-1
As the input elements a & b are independent in neural networks, the statistics of their product is:
E (+)t)) = E ab = ab
(31)
ID (+()) ab(1 a) b) (T - 1
2
-
(32)
8
17

---

Published as a conference paper at ICLR 2024
F IMPLEMENTATION OF UGO-APPROXMMATED ANN CONVERSION TO SNN
Due to the large network size of ViT, traditional methods like MaxNorm (eeekarmRueckauer et al.) 2017) not sufficient preserve ANN's performance. Therefore use some advanced, training-free free techniques for conversion.
F.1 THRESHOLD BALANCING
When a UGO-approximated ANN is obtained, the activation functions have all been replaced with ReLU, which is convenient for us directly convert it to SNN, A small number of training samples are needed to obtain the maximum activation value and its quantitative estimate to determine the threshold potential of the neurons (Li et al. 2021p. Our optimization objective is: (1
T/A clamp QT (f)
min (QT (z', T, v/n) ReLU (z')), Vin
0,,T
(33)
where = W' As Eq 33 has closed-form solution, we find the optimal threshold by grid search, enumerating nnn 100 in our experiments) values within [0.5z] 2 mman
F.2 SIGNED NEURONS WITH MEMORY POTENTIAL
For the IF neuronal structure that only release positive spikes, if the input from neuron's negative- synapse arrives late and the positive-weighhe input has already been converted into spike, a portion of the significant negative potential information will not be transmitted. There- fore, we introduce Signed Neurons with Memory potentials(SNMM which allow negative spikes to be (lleased/Wanns et al. 2022a) ensure that negative-weighted information is lost. In this perspective, Eq can be refined as:
m'(t) p' 1)+ (t),
71()
(1-
li
ml+((() 2 Uth
tho
(()) -Uth (() < -ith and pla(t) > 0
(34)
otherwise
pt()) = mt()) - (t), (t) = r'(t - 1) s'())
where (1) and (1) denotes the memory potential before and after spikes' triggering.
F.3 BURST SPIKES WITH P-SCALE THRESHOLD
In order to minimize the effeet of lagging inputs generated during SNN inference, spikes mechanism, which allows neurons to clear off residual potentials in the form of ['[ = 2 in experiments) fighfrreueecc spikes between regular emissions & Zeng 2022). The threshold of the residual potential is set to pVtn- Considering its small scale relation to Vth. and without disrupting the quantization relationship established by Eq 33 we p 0.5.
the burst
F.4 ALGORITIM
18

---

Published as a conference paper at ICLR 2024
Algorithm STA Conversion Pipeline
Input Pretrained ANN; Non-linearities (fi) with synthetic distributions {Di}: Simulation length
T
for each nonlinear function fa in Transformer do
Initialize UGO model f; with N; hidden neurons
Sample M points { from D; Optimize ft using labels (1,(()))==......
end for
Replace non-linearites } in pretrained ANN with (f..)
Replace multiplications in pretrained ANN with TCSA (cf.Eq
ReLU layer in the ANN do Collect the input æ' and the output x' Find the optimal threshold v/ for SNN by grid search (cf.Eq 33 end for
1.
Output Converted SNN
Algorithm STA Inference
Input Converted SNN; Simulation length T; Burst length I; Burst scale p for /. 1,2,.... do
for each forward operation g in the SNN do if is a multiplication layer then Output temporary scaled product Σ/=1 k())x end if if is a non-spiking linear layer then Output (x(t)=
using TCSA (cf.Eq
end if
if is a spiking linear layer constructed by IF neurons then Calculate Au (t(() + b Release positive & negative spikes with threshold Vaa, and update potentials v (cf.Eq 34 i do Release spikes with threshold pvrk, and update potentials end for
end if end for
end for
19

---

Published as a conference paper at ICLR 2024
G SUPPLEMENTARY FOR EXPERIMENTS
G.1 DATASETS
CIFAR-10 is a dataset developed by the Canadian Institute for Advanced Research (CIFAR), widely used as a benchmark dataset for developing and evaluating image classification models due to its manageable size and variety of classes. It consists of 60,000 color images sampled from Tinylmages Dataset[Torralos et al. 2008), divided into 50,000 training images and 10,000 testing images. There are 10 different classes in CIFAR-IO, 10, including common objects like airplanes, cars, birds, cats, etc.
CIFAR-10.1 & CIFAR-10.2 are new testing sets for CIFAR-10, 10, each incorporating 2,000 images from Tiny Images Dataset. There are small distribution shifts between them and the original data set, which may be attributed to different generation conditions (such as illumination, angle, etc.) or adversarial attacks. Therefore, they are created to assess the robustness and generalization of models trained on CIFAR-10(Recht al. 2018 Lu al. 2020).
In our experiments, since there is no training set for the above two datasets, we use the training samples of CIFAR- 10 to determine the threshold potentials for SNN. The high accuracy in the results demonstrates that the converted SNN not only retains the generalization ability of the pretrained model but also has the robustness this distribution shift.
CIFAR-100 also a subset of Tinylmages Dataset and serves as a more challenging version of CIFAR-10, consisting of 100 fine-grained classes, categorized into 20 superclasses. Like CIFAR- 10, it includes 50,000 training images and 10,000 testing images.
ImageNet is one of the largest public image databases, containing about 14 million images labeled into 1,000 categories(the full dataset is over 20,000). Unlike Tinylmages Dataset, it consists of high- resolution images from the Internet. ImageNet provides wide range of help for the realization of tasks such as image classification, target detection and semantic segmentation in large-scale scenar- ios. ImageNet-200 is a well-chosen subset of ImageNet containing 200 categories that can help train and evaluate models more efficiently.
G.2 ADDITIONAL RESULTS
We provide additional results on standard classification tasks using fine-tuned ViT-B/32 from CLIP, well other models like resnet-20 and ResNet-34 trained directly these dataset. The pretrained ResNet-50 does not surpass the performance of direct training, while ViT-B/32 performs well on generalization.
To determine the optimal scale N and training method for the Universal Group Operator (UGO), we conducted ablation experiments on the CIFAR-100 dataset with T 32. By modifying the UGO parameters, we compared different settings' impact on overall accuracy. The results in Table that as we increased N, accuracy shows decelerated growth with efficiency continually declined. Considering both factors, we selected balanced approach that achieves good accuracy without excessive computational cost. The ablation experiments guided our selection of an appropriate UGO scale and training method.
To verify the effectiveness of the techniques mentioned we did several sets of ablation exper- iments on cifar-10 10 with T = 32. Thols that the SNM structure significantly improves the performance of the converted SNN. The neuron-wise search of potential threshold and the introduc- tion of the burst spikes mechanism also play an important role in model conversion, especially when combined with SNM. This may be due to the unique dynamic characteristics caused by estimation- correction mechanism.
20

---

Published as a conference paper at ICLR 2024
Table 4: Comparison with other backbones and baselines on standard classification of CIFAR-10
Model
Method
ANN Acc.
T=32
T=64
T=128
T=256
RMP
Han et al.
2020)
38.04 38.04
59.73
90.10
90.47
TSC
Han & Roy
2020
57.64
71.22
91.30
92.30
Opt.
(Dngg & Gu
2020
87.30
92.50
94.32
95.28
resnet-20
95.68
Calib.
[1 al.
2021)
94.77
95.02
95.17
95.44
SNM
Wang et al.
2022a
94.13
95.43
95.75
95.69
Burst Li & Zeng
2022)
94.92
95.51
95.40
95.61
Opt. (Deng Gu
2020)
71.37
81.52
85,43
88.10
ResNet-50
Calib. (Li et al. 2021)
95.71
87.64
91.85
92.79
94.60
(CLIP)
SNM (Wang et 2022a
90.30
91.42
92.44
94.31
ViT-B/32
STA (Ours)
96.16
95.49
95.74
95.68
95.82
Table 5: Comparison with other backbones and baselines standard classification of ImageNet
Model
Method
ANN Acc.
T=32
T=64
T=128
T=256
RMP (Han et al. 2020)
70.64
55.65
Opt. (Deng & Gu 2020)
70.95
33.01
59.52
67.54
70.06
ResNet-34
Calib. (Li al. 2021)
75.66
64.54
71.12
73.45
74.61
SNM (Wang et al. 2022a
73.30
55.28
62.72
65.53
69.31
ViT-B/32
STA (Ours)
83.60
78.72
82.33
82.56
82.79
Table 6: Ablations for settings of Universal Group Operators classification on CIFAR-100, T=32.
N and Penalty
Exp
GELU
Inverse
LayerNorm
T=32, N in Table
Baseline
Accuracy=84.15
10.46
-5.24
-6.43
+0.00
+ Penalty
-1.62
-4.92
-9.76
-6.14
16
-4.62
-2.43
-0.14
+0.02
16 Penalty
-4.51
-1.41
+0.00
-1.79
32
+0.00
-0.40
+0.08
-0.24
32 Penalty
-0.08
+0.00
+0.56
-1.58
64
+0.22
-0.03
+0.52
-0.05
Table 7: Ablations for conversion techniques CIFAR-10, T=32
Technique Settings
SNN Acc.
MMSE
Burst Spikes
Use SNM
layer-wise
1==
11.35
layer-wise
T
15.57
neuron-wise
T=
X
13.48
neuron-wise
r= 0
V
54.32
neuron-wise
T= 2, 0.5
X
19.84
neuron-wise
r T 0.5
95.26
21

---

Published as a conference paper at ICLR 2024
Table 8: Neuron numbers and weights in each UGO unit.
GELU
Softmax-Exp
Softmax-Inv
LayerNorm
Non-linear Neurons of Original Units Non-linear Neurons of UGO Units
3072
768
768
3072+(3072x)32
768+(768 x
16
768+8
Weights of Original Units Weights of UGO Units
0
0
0
0
32x2
32x2
16x2
8x2
H PARAMETER STATISTICS OF UGO UNITS
In the ANN-SNN conversion for ResNet, the number of neurons is exactly the same as the number of RELU activation layers the original backbone. However, the case of ViT-B/32, replacing many "non-linear operators" with "neurons" significantly increases the required number of activations. Also, defining the number of "neurons" used in the Attention Block is challenging, because its non- linearity is based on multiplications and Softmax instead of activation functions. We provide Table to clarify the required number of neurons for ViT-B/32.
It shows the number of neurons and parameters for each module and its corresponding Universal Group Operator (UGO). ViT-B model consists of 12 blocks, each containing GELU activations, 2 LayerNorm operations, and I Softmax operation. Since all UGOs share the same parameters, the increase in weights is minimal. However, some modules like GELU need to be applied to each input feature, resulting in significant computational load (see numbers listed in the brackets). But since we have replaced the non-linear operation with a linear one, the actual complexity does not increase significantly. The non-linear modules originally do not involve synaptic operations, thus the number of weights for these modules zero.
22


Reviews:
[{"review_id": "ScutC9ckc8", "reviewer": "Reviewer_jaAV", "summary": "This study introduces a training-free method to convert ANN transformers into SNNs, preserving the weights of the original pretrained model to ensure its inference capability remains intact. The resulting SNN Transformer model outperforms its convolutional network counterparts.", "questions": "1. Would employing model compression strategies, such as pruning, enhance the efficiency and reduce the size of the Universal Group Operators?\n\n2. The conversion implementation integrates multiple existing ANN-SNN conversion algorithms, including SNM and Burst. Using the same combined conversion for ResNet baselines could enable a more fair comparison.\n\n3. What hurdles might one encounter when adapting this technique to language Transformers? Would the method necessitate modifications?\n\n4. What challenges might arise when adapting this method to larger-scale transformer models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.This paper overcomes the differences in computational paradigms between ANN and SNN Transformers, and can accurately approximate ANNs with converted models.\n\n2.The proposed training-free conversion strategy could enable the direct deployment of large-scale pretrained ANN models to low-power neuromorphic hardware.", "weaknesses": "1.The proposed Universal Group Operators use extensive spiking neurons to model fine-grained ANN operations. This high model complexity reduces power efficiency and incurs large memory usage. \n\n2.The current implementation only handles image Transformers. Longer sequences in language models may introduce more unaddressed issues such as threshold variations similar to that in spiking RNN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NVqhLPpRhN", "reviewer": "Reviewer_S5Cb", "summary": "This paper proposed Universal Group Operator (UGO) and Spatio-Temporal Approximation (STA) to fit the functions of LayerNorm, GELU layers and optimize the conversion error about self-attention modules.", "questions": "See Weakness Section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The theoretical analysis about Temporal Estimation & Correction in Eq.5-Eq.11 is convincing.", "weaknesses": "1. From Tab.1-2, it seems that the author's approximate fitting methods for nonlinear operations such as LayerNorm require relatively long time-steps ($\\geq 32$) to be effectively implemented, which will result in more significant time latency and energy consumption. In addition, even under 256 time-steps, the author's approximate fitting and error correction methods cannot completely eliminate the conversion error (there is a ~1% accuracy loss).\n\n2. Regarding the fitting calculation of LayerNorm and Softmax layers, as well as the self-attention layer error correction calculation in Eq.9, it seems that the calculation steps and costs involved are still relatively large. I think this may hinder the algorithm's practical application.\n\n3. I noticed that a previous work [1] achieved similar ANN-SNN Conversion performance to this paper when using BatchNorm layers directly (without involving nonlinear operations) and without error correction for attention modules. So I think the value of this approximate fitting and error correction method still needs to be further evaluated.\n\n[1] Ziqing Wang, Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, Renjing Xu. Masked Spiking Transformer. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hyb2Z7hAt1", "reviewer": "Reviewer_wK6t", "summary": "Summary:\nThe paper proposes a training-free method to convert transformer to SNN platforms. It proposes universal group operators to approximate nonlinear activations and temporal-corrective self-attention layer to approximate spike multiplications. Compared to prior work, it is the first to support pretrained transformers with SNNs without training.", "questions": "Questions are listed in the weaknesses part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strength:\n\n1.\tThe idea is novel. Use multiple spiking neurons to estimate the nonlinearity functions. The temporal-corrective self-attention can achieve unbiased multiplication between two variable matrices. \n\n2.\tThe convergence and error bound have solid theoretical guarantees and experimental validation.\n\n3.\tCompared to prior work with training/calibration, this work can quickly convert ViT to SNN hardware with high fidelity with a small timesteps T.\n\n4.\tIt also shows the efficiency benefits compared to ANNs, which justifies the advantages of SNN-based transformer.", "weaknesses": "Weakness:\n\n1.\tThe latency/runtime benefit of SNN-based transformer with different timesteps T needs to be compared to ANN accelerators.\n\n2.\tThe proposed method can have a high fidelity with a small timestep, but there is still 1% gap compared to ANNs, even with a large T. \nCompared to the training/calibration-based method, the training-free one shows a higher accuracy gap. (Table2, SNM, Calib on resnet20 can fully recover the accuracy). Can the authors comment on that? \n\n3.\tIs there any randomness in the spikes-based multiplications given the current data encoding? If there is, the output of the computing result is not deterministic. How robust is it to randomness? To have a deterministic output, the effective resolution will be reduced, thus harming accuracy. Can the authors comment on that?\n\n4.\tHow does the spiking-based multiplication differentiate from the standard multiplication mechanism in stochastic computing?\n\n5.\tThere are other acceleration methods to speed up and reduce energy consumption by a large factor without sacrificing accuracy, e.g., model compression and better architecture design. Moving to a new hardware platform with 30-40% energy reduction seems not very convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9zD0vjbow", "reviewer": "Reviewer_gaB3", "summary": "This research presents a novel method for spiking neural networks from pretrained Transformer models (or models with multi-head attentions). The suggested Universal Group Operators and Temporal-Corrective Self-Attention Layer allow a pretrained Transformer to be converted to a completely event-driven SNN without the need for training, which holds promise for neuromorphic computing. Effectively positive experimental outcomes were obtained.", "questions": "From Figure 3, I can observe severe, uneven quantization. Can you explain how this quantization affects the accuracy of the output?\n\nHow do you prepare the data for pretraining non-linear activations? For GeLU, do you record the actual responses of the ANN and train it on these activation values? For other nonlinearities (inverse, exp, layer norm), how do you pretrain?\n\nAlso, why do these nonlinearities need so many kinds of losses here (Table 3)? Can you explain why Huber loss fits exp, gelu, and inverse? And why does MSE fit the layer norm?\n\nBesides, do you evaluate the actual increment of neurons by using UGO? I want you to give me a table reporting the difference in neuron number and weight according to the models listed in Table 1.\n\nAre you going to release the weight of the pretrained nonlinearity? Do you test the sensitivity of changing $N$ and $T$?\n\nI think there is a typo in equation (22). $V_{th} \\Vert w_2\\Vert_1$ should be $(V_{th} \\Vert w_2\\Vert_1)/T$.\n\nWhy do you mention setting $V_{th}$ using the strategy proposed by Li et al., who proposed to use dynamic thresholds, and demonstrating the quantization gap using the maximum activation as a threshold?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The ability to convert pretrained Transformer models into spiking neural networks without the need for training is more appealing than training Spiking Transformers directly.\n\n\nFor neuromorphic computing and widespread deployment, the pure implementation on spiking neurons is promising.\n\n\n\nThe universal nature of the conversion method used here makes use of linear models' capacity for global approximation. The inverse function is converted correctly.", "weaknesses": "In comparison to conventional artificial neural networks, there may be a slight accuracy gap due to the approximation error from Universal Group Operators.\n\nThe proposed method has only been tested on the ViT-B/32 model from CLIP; it is unknown if it can be applied to other models.\n\nThe converted models perform computations at a marginally higher rate than traditional spiking CNNs (which have more synapses and neurons).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)