Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of adding spiking neurons to transformer architecture. 
strength_and_weaknesses: Strength:

+ Clear paper structure, neat paper presentation. 

+ Compared to spiking convolutional architectures the accuracy are higher. 

Weakness: 

- Seems like the method to alleviate the full precision multiplication is to add more LIF neurons. 

- Can authors at least provide transfer learning results since high transferability is the core argument for artificial transformers? Also, more LIF may provide higher training latency in practice. Can the authors compare training GPUs between SpikeTransformer and SpikeCNN?

clarity,_quality,_novelty_and_reproducibility: Clarity; 8/10
Quality: 8/10
Novelty: 7/10
Reproducibility: Yes
summary_of_the_review: Based on the quality of this paper, I intend to give acceptance. If the concerns can be adequately addressed, I will increase my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes the spiking version of the transformer by designing spike-form of Query, Key and Value of self-attention. The experiments are performed on both static and neuromorphic datasets for the classification tasks.
strength_and_weaknesses: ++ The figures are nice

--The novelty of this paper is limited. The major contribution is that the authors apply the transformer framework to SNNs. The performance of the proposed spiking transformer is lower than the transformer of ANNs. This research cannot give full play to SNN strengths, like temporal processing capability, and will cause researchers to go astray. 

--The authors over-claimed their contributions. The authors claim that they are the first to implement Transformer in SNNs. Actually, there exist many spiking transformer works, e.g., [1-3].

-- The authors use floating point operations to add spikes, which defeats the purpose of SNNs.

--The authors are not familiar with the research progress of SNNs. There are many of the latest works, which are ignored by the authors. E.g. [4-7].
 
--The comparison is unfair. As the network structure of the spiking transformer differs from SEW-ResNet-152, there is no doubt the performance will be better. I do not think it is a fair comparison. Please show the performance of the corresponding ANN. 

--Ignoring all that, the results are not convincing. The performance of the proposed method is not SOTA. The authors only chose favourable comparisons. For example, TET achieves 83.17% accuracy on CIFAR10-DVS dataset with 10 steps, while the proposed method gets 80.9% with 16 time-steps. 

--The authors did not compare the training cost of the methods. Please add it in Table 2. 

--The ANN2SNN method achieves better performance with the increase of time-steps. How about spiking transformer? If you train the network with time-steps 4 on ImageNet, please show the performance of the network when the time-steps are 2,3,10,20,50.
 
--Please compare SOPs and Power in Table 3 and 4.

[1] Zhang, Jiqing, et al. "Spiking Transformers for Event-Based Single Object Tracking." CVPR. 2022.

[2] Zhang, Jiyuan, et al. "Spike Transformer: Monocular Depth Estimation for Spiking Camera."ECCV. 2022

[3] Mueller, Etienne, et al. "Spiking Transformer Networks: A Rate Coded Approach for Processing Sequential Data." 2021 7th International Conference on Systems and Informatics (ICSAI). IEEE, 2021.

[4] Xiao, Mingqing, et al. "Training feedback spiking neural networks by implicit differentiation on the equilibrium state." Advances in Neural Information Processing Systems 34 (2021): 14516-14528.

[5] Meng, Qingyan, et al. "Training much deeper spiking neural networks with a small number of time-steps." Neural Networks153 (2022): 254-268.

[6] Bu, Tong, et al. "Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks." International Conference on Learning Representations. 2022.

[7] Wang, Yuchen, et al. "Signed Neuron with Memory: Towards Simple, Accurate and High-Efficient ANN-SNN Conversion." International Joint Conference on Artificial Intelligence. 2022.
clarity,_quality,_novelty_and_reproducibility: The novelty of this paper is limited, and the experimental evaluation is flawed and fails to adequately support the main claims.
summary_of_the_review: Overall, the significance of this work is limited, and the authors over-claimed their contributions.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors introduce Spikformer a SNN version of Transformer, which includes a novel implementation of self-attention with spiking representations. The authors show that their proposed architecture outperforms SOTA SNNs on both traditional image tasks as well as neuromorphic tasks.  
strength_and_weaknesses: Strengths 
The paper's motivation is a relatively simple idea (implement a SNN version of Transformers), but its implementation is non-trivial given the constraints of SNNs. The authors' solution is a simple and efficient one, which sets solid ground for future iterations. The paper strengths are the novel architecture contributions and the SOTA results when compared with other SNNs.

Weaknesses
Even though it is not the authors' motivation, it can be argued that this paper does not offer an alternative architecture to Transformers. Transformers likely outperform Spikformers in many ways and most importantly in prediction quality. On the other hand, we are also not learning about new biological features that are important for brain computing. This paper is about advancing the field of SNN research, which has not yet produced an algorithm that can broadly compete with ANNs. Yes, we know for a fact that NNs in the brain are SNNs, but we still do not know why, and this paper is obviously not helping us clarify that.
clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is very clearly written. The different aspects of the proposed architecture are clearly outlined both in the text and the figures of the paper. The paper clearly highlights its novel contributions and makes references to key relevant related work. 

Quality
The quality of the paper is sound. The claims in the paper are backed by a sufficient number of relevant experiments comparing Spikformers with similar SNN architectures. The theoretical parts of paper are solid and I cannot find any obvious errors in them.  

Novelty
The authors introduce a novel version of self-attention for SNNs. This is the first attempt to implement self-attention in SNNs that I am aware of. 

Reproducibility
The authors provide enough information to enable others to reproduce their work. I have not personally attempted to reproduce the key results of the paper, but I believe this will be easily possible once the source code is released. 
summary_of_the_review: Overall a solid contribution to the field of SNNs research and one that should deserve a spot in the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors provide a feasible implementation of SNN-oriented self-attention mechanism and Vision Transformer, obtaining superior results on both static and neuromorphic benchmarks.
strength_and_weaknesses: Strength:
1.	The first time to implement self-attention and transformer in large-scale model and dataset.
2.	The novel module pays attention to spike-based operations and avoids multiplications to obtain high energy efficiency.
3.	The model achieves new state-of-the-art results on various datasets.

Weakness:
1.	The paper provides much about the experimental results, while the contents for the method and motivation seem insufficient.
2.	Not much insights have been given as to why spiking self-attention should be designed in this way.
3.	Other concerns detailed in Summary.
clarity,_quality,_novelty_and_reproducibility: The motivation is straightforward and the paper is overall clearly written. However, this article is slightly less innovative and less rigorous in spite of the superior results.
summary_of_the_review: 1. In Eqn. (16), additional SN-BN-Linear is applied to SSA’. What is the role of this extra part? 
2. Table 1. ①Why is the ratio of (SOPs of A_SSA)/(FLOPs of A_ReLU) much smaller than those on other datasets? ② It is stated in the paper that the VSA with float-point-form Q, K and softmax is redundant for spike-form X, V, which cannot get more information from X, V than SSA. Does it mean that a continuous attention map cannot work better than the spike-formed attention map? Can the authors provide more explanations for this and why the opposite conclusion is drawn from Table 5? ③ It seems that sparsity has been taken into account for both SOPs and FLOPs (in ReLU), but this is not explicitly stated and seems confusing. 
3. The authors use the data 77fJ/SOP for their energy estimation. However, it only stands for the Energy per Synaptic Event reported in ROLLS, while there are other operations like producing a spike which consumes 3.7pJ. The authors should provide a more rigorous comparison in terms of energy consumption.
4. I noticed that the model used skip connections, and I am curious about how the two branches of spikes are merged. If the two spike trains are added together directly, the data will not be limited to the pure spike form and then does the first linear layer in the subsequent block still avoid multiplications? 
5. What is the scaling factor s in Eqn. (15)? 
6. In Table 3, it would be better to compare with ANN-Transformers instead of ANN-ResNet19.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper describes an approximation of transformer networks that can be implemented by spiking neural networks. The main and only contribution of the paper is to simplify the self-attention module of the transformer by removing the softmax function to enable a spiking network approximation. Hence the technical contribution appears quite limited and the paper's main contribution is in the fact that it pushes the SOTA for spiking models. But as expected, while the resulting spiking transformer outperforms spiking CNNs on imagenet and other image datasets used to evaluate spiking neural networks, all spiking networks fair far below non-spiking SOTA models. Hence, this paper will only generate interest from people interested in spiking neural networks (which is a rapidly growing but remains somewhat of a niche for ML).
justification_for_why_not_higher_score: The significance of this work is limited to pushing the SOTA in spiking neural networks.
justification_for_why_not_lower_score: There is general support for this paper to be accepted except for one reviewer. My read is that the authors addressed relatively well the criticisms but I was not able to get any of the reviewers to discuss...
Decision: Accept: poster