Spiking PointNet: Spiking Neural Networks for Point Clouds
Authors: Dayong Ren
Zhe Ma
Yuanpei Chen
Weihang Peng
Xiaode Liu
Yuhan Zhang
Yufei Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: Spiking Neural Networks
Point Clouds

Abstract: Recently, Spiking Neural Networks (SNNs), enjoying extreme energy efficiency, have drawn much research attention on 2D visual recognition and shown gradually increasing application potential. However, it still remains underexplored whether SNNs can be generalized to 3D recognition. To this end, we present Spiking PointNet in the paper, the first spiking neural model for efficient deep learning on point clouds. We discover that the two huge obstacles limiting the application of SNNs in point clouds are: the intrinsic optimization obstacle of SNNs that impedes the training of a big spiking model with large time steps, and the expensive memory and computation cost of PointNet that makes training a big spiking point model unrealistic. To solve the problems simultaneously, we present a trained-less but learning-more paradigm for Spiking PointNet with theoretical justifications and in-depth experimental analysis. In specific, our Spiking PointNet is trained with only a single time step but can obtain better performance with multiple time steps inference, compared to the one trained directly with multiple time steps. We conduct various experiments on ModelNet10, ModelNet40 to demonstrate the effectiveness of Sipiking PointNet. Notably, our Spiking PointNet even can outperform its ANN counterpart, which is rare in the SNN field thus providing a potential research direction for the following work. Moreover, Spiking PointNet shows impressive speedup and storage saving in the training phase. Our code is open-sourced at https://github.com/DayongRen/Spiking-PointNet.

Spiking PointNet: Spiking Neural Networks for Point Clouds
Dayong Ren, Zhe Ma, Yuanpei Chen, Weihang Peng, Xiaode Liu, Yuhan Zhang, Yufei Guo Intelligent Science & Technology Academy of CASIC, China Scientific Research Laboratory of Aerospace Intelligent Systems and Technology, China eewpp.uu.edu........ edu. cn
rdyedu@gmail.. com, [iguu_bbi@1200 com, yfguoêpku.
Abstract
Recently, Spiking Neural Networks (SNNs), enjoying extreme energy efficiency, have drawn much research attention on 2D visual recognition and shown gradually increasing application potential. However, it still remains underexplored whether SNNs can be generalized 3D recognition. To this end. we present Spiking PointNet in the paper, the first spiking neural model for efficient deep learning on point clouds. We discover that the two huge obstacles limiting the application of SNNs in point clouds are: the intrinsic optimization obstacle of SNNs that impedes the training of big spiking model with large time steps, and the expensive memory and computation cost of PointNet that makes training big spiking point model unrealistic. solve the problems simultaneously, we present trained-less but learning-more paradigm for Spiking PointNet with theoretical justifications and in-depth experimental analysis. specific, our Spiking PointNet trained with only single time step but obtain better performance with multiple time steps inference, compared to the one trained directly with multiple time steps. We conduct various experiments on ModelNet10. ModelNet40 to demonstrate the effectiveness of Spiking PointNet. Notably, our Spiking PointNet even can outperform its ANN counterpart, which is rare in the SNN field thus providing potential research direction for the following work. Moreover, Spiking PointNet shows impressive speedup and storage saving in the training phase. Our code is open-sourced at "Spiking-PointNel
Introduction
The advent of deep learning technologies, notably PointNet [38], has considerably amplified our capa- bilities to comprehend and manipulate intricate 3D data from real-world settings. With autonomous driving and augmented reality, which often require real-time interaction and fast response, becoming increasingly prevalent, the reliance on efficient point cloud processing techniques has been escalated. and usually needs powerful devices.
However, computation for the point cloud is
53 seen as more energy efficient than Artificial Neural Networks (ANNs) due to their event-driven computation mechanism and the energy-saving multipiication-addiooo transformation ad vanttaee, have received extensive attention recently in many fields. For example, in [36], SNNs were used to handle sequential learning and show better performance and less energy cost on sequential learning compared to ANNs with similar scales. In [31], SNNs were leveraged to study the Human Activity Recognition (HAR) task. The results show that the SNN can reduce up 94% energy consumption while being comparable to homogeneous ANN counterparts in accuracy. There are also some works that apply SNNs autonomous driving. LaneSNNs 45) presented SNN-based approach to detect
Spiking Neural Networks (SNNs) 40: 4
12 39
35
2
55 22.
57 56
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Training Phase
LIF
LIF
Results
100
Perturbation
Perturbation
Perturbation
Results
LIF
LFF
LIF
00000100
Inference Phase
Figure The overall of the trained-less but learning-move more framework. The Spiking PointNet is trained with only one single time step in the training phase, while is used with multiple time steps in the inference phase. To improve the performance of the SNN, we also add some membrane potential perturbation in the training.
the lanes with event-based camera input with very low power consumption of about W. For the more challenging point cloud task, a question is naturally raised: Could SNNs be transferred to the 3D domain and retain the energy-efficient advantage?
To this end, we present Spiking PointNet, the first spiking neural network approach to deep learning on point clouds. To better apply the SNNs in the point cloud field, we focus solving two huge obstacles staying on this road. The first is optimizing difficulty. Though the binary spike information transmission paradigm makes SNNs much energy efficient, also introduces the training challenge since the gradients for firing procession of the spiking neuron are not well-defined, and they are all almost zero infinite sometimes. The rero-but-all gradient makes it impossible to train SNNs via gradient- based optimization methods like ANNs. To handle this problem, various Surrogate Gradient (SG) methods have been proposed (35 49 39 30 14. This kind of method tries to find an alternative function to replace the firing function w when doing pack-propagation of the spiking neurons. Thus, the SNN can be also trained with the current gradient-based optimization framework. However, it not easy to find a suitable surrogate function, especially for these SNNs with large time steps. With the increasing of time steps, the explode or vanish problem and the gradient error problem will be severe. We will provide detailed analysis in Sec. 3.3
The second problem is that training networks for point clouds need more expensive memory and computation than images since point cloud data requires more dimensions to describe itself. To overcome this limitation in point clouds, researchers have proposed various model simplification strategies. These strategies include but are not limited to, sparse convolution [7]. optimization during the data processing phase [27]. and optimization the local feature extraction stage [34] 33)
However, for applying the SNN to point clouds, the memory and computation will be enlarged greatly still with the increasing time steps, and the above methods cannot handle this problem well. Thus, there is no existing way to train SNNs with large time steps on common deep- deep-learning devices.
To solve the above problems simultaneously,
present trained-less but learning -ooee more paradigm for Spiking PointNet. Specifically, we propose a new framework for Spiking PointNet, that we train the SNN using suitable SG method with only single time step and infer it with multiple time steps to obtain better performance. We will prove theoretically and experimentally that this framework can result in a better SNN than training with multiple time steps directly in Sec. 3.4 To improve the framework further, we also embed a membrane potential perturbation method in the framework based on the observation that the residual membrane potential of SNN coming from the previous time step cannot transmit the temporal information for static point cloud datasets but a perturbation increase the generalization. The overall workflow of the framework visualized Fig.
The contributions of our paper are as follows:
We prove that is not easy to train a well-performed SNN with large time steps directly for point clouds with theoretical justifications and in-depth experimental analysis and propose

---

-orni the Spiking PointNet with a trained-less but learning- more framework, a first simple yet effective SNN framework for point clouds.
Furthermore, we also propose membrane potential perturbation method for the framework to increase the SNN generalization.
We evaluate our methods on various datasets and the experimental results show the effective- ness of our method. Rather, our Spiking PointNet even can outperform its ANN counterpart, which is very rare in the SNN field.
2 Related Work
2.1 Spiking Neural Networks
Generally, there are three kinds of methods to train SNNs [I6]: (1) spik--tmmingddepeddenl plasticity
(STDP) [1] approaches, (2) ANN to SNN conversion approaches [25]
24: 32;
8
10
3 22 29]
and (3) directly training approaches 35 49 39
13]. STDP a kind of biology-inspired method (23 9 that updates the weights with the unsupervised learning algorithm
30
46 47 18 21
called Hebbian learning (3 However, it is limited to small-scale datasets yet. The ANN-to- SNN conversion 8 [29] converts well-trained ANN checkpoint to the SNN counterpart. Since training an ANN is much faster than training an SNN, this kind of method provides a fast way to obtain an SNN without using gradient descent for SNNs at all. However, it does not have its own learned feature. In specific, all the converted SNN does is to mimic the ANN. Moreover, this type of method requires many time steps to obtain high-accuracy SNN. The direct training method tries to find an alternative function to replace the firing function of the spiking neurons when doing back-propagation. This kind of method can narrow the time steps greatly, even less than 5 (20 [5] 14], hence has received much attention recently. However, it is not easy to find a suitable surrogate function for these SNNs with large time steps. In this work, we focus on solving the problem.
2.2 Deep Learning on Point Clouds
Training networks for point clouds need expensive memory and computation. To address the challenges posed by expensive computation and memory requirements, researchers have proposed a series of model simplification strategies to overcome the limitations of current point cloud models in practical applications [7] 41 27
33 42]. For instance, Lee el al. (28) introduced PillarAcc, algorithm-hardwarc hardware co-design that significantly enhances the performance and energy efficiency of 3D object detection. However, its reliance on complex sparse convolution and dynamic pillar pruning may introduce additional complexity in the design and implementation process. Choy et al. 77) proposed MinkowskiConv, which provides a comprehensive solution for handling sparse spatio-temporal data, greatly enhancing its ability to capture complex temporal patterns in the data. Nevertheless, the inherent computational complexity and memory demands of 4D convolutions present new challenges. Hu el al. (27) introduced RandLA-Net to conserve computational resources in point cloud analysis by leveraging random sampling and an efficient local feature aggregation module. However, a limitation of RandLA-Net is that random sampling may lead to the loss of critical information and cannot be seamlessly applied to existing networks without a decline in performance. In comparison, the SNN version of PointNet offers an effective solution by significantly improving algorithm execution efficiency without altering the overall network structure, reducing dependence on rereo-eeffrmacce devices in the inference. This enables -pnrrol- networks to more effectively address the computational resource consumption challenges of practical point cloud networks without the need to redesign network structures. However, for applying the SNN to point clouds, the memory and computation will be enlarged greatly still with the increasing of time steps in the training time. And, there is no existing way to train SNNs with large time steps on common deep-learning devices.
34;
an innovative algorithm
3 Preliminary and Methodology
In the paper, we mainly apply the SNN for the PointNet [38] the first deep learning model that processes raw point clouds directly, and modify it to the Spiking PointNet. Here, we first introduce the PointNet and widely used SNN neuron model, Leaky Integrate-and- (LIFF model in detail. Then we will elucidate the difficulty of optimizing the Spiking PointNet with large time steps. Next,
3

---

a trained-less but learning-mor more framework to solve the above problem will be presented. Finally, we further improve with membrane potential perturbation method.
3.1 PointNet
represents repreennts novel application of deep learning to process point cloud data [38]. effectively addresses two primary challenges: permutation invariance, the unordered nature of point cloud data, and rotational invariance, the freedom to rotate the point cloud in 3D space without altering the represented object. Specifically, to tackle these challenges, PointNet employs symmetric function in conjunction with a spatial transformer network. It processes each point through a shared fully connected network, followed by max pooling operation. This approach inherently ensures permutation invariance as it remains indifferent to the order of input points. Formally, given point cloud data ( ...,,... Fn ) each point z; is transformed via shared Multi-Layer Perceptron (MIP) denoted by h. followed by a max pooling operation to enforce symmetry, yielding a global feature descriptor. Therefore, PointNet approximates general function defined on point set by applying a symmetric function on transformed elements in the set:
I (( Fal) Re g (h
(xx)),
where f 2 R _K and g: R x R + R is symmetric function.
For rotational invariance, PointNet introduces a spatial transformer network specialized neural network proficient at predicting the required spatial transformation matrix for the point cloud, thereby enabling PointNet to manage rotating point cloud data.
The principal divergence between PointNet and conventional point cloud processing methodologies resides in the implementation of deep neural networks. This represents significant leap from the traditional approach of manually designed features to Artificial Neural Networks (ANNs). The proposed model, Spiking PointNet, advances this progression by transitioning from ANNs to Spiking Neural Networks (SNNs). SNNs, which emulate the neural mechanisms of the brain more closely, promise to enhance the efficiency and precision of point cloud processing outcomes.
3.2 Explicitly Iterative LIF Model
SNNs the spiking neuron, which is inspired by the brain's 's natural mechanisms, to transmit information. A spiking neuron will receive input spike trains from the previous layer neuron models along times update its membrane potential, In the paper, we adopt the widely used leaky integrate and fire (LIF) neuron model, which can be described as follows:
du di
(u Urea) + R I(I),
< Vih-
(2)
In the above equation, I represents the input current, Va is the threshold, and R and Tm are the resistance and time constant, respectively. A spike will be generated when reaches Via. and is subsequently reset to the resting potential u = 4eest typically set to zero 30 (9].
To use the mature machine learning framework (e.g., TensorFlow, Pytorch) to train the SNNs, an explicitly iterative LIF spiking model was proposed in [49] given by
ult + 1] = A (u/(t) - &t)) + Σ (1y8(((] + b,.
stft + 1] H (u, + 1] Vin)
Here, I(()= - I(()= E, (1) + bi where the subscript denotes the i-th current neuron, wy is the weight from j-th neuron in the previous layer connected the current neuron i, and by is bias. II (x) signifies the Heaviside step function, s is the spike train of neuron i at discrete time step 6, and À < 1 a leaky term for _. typically is 0.20 0.25 in 30 39 20
Tm The main difference between ANNs and SNNs is the nonlinear computational neuron. Replacing the ReLU neuron from PointNet with LIF spiking neuron will transform the PointNet to Spiking PointNet.

---

[++1
S-+1
3SL-1
0s!-1
L+1
1+1
t
255
1-1
85543
W
W
W
0L
aW
aw
2w!
s
96+1
G7.
UL
St
i+1
05c
Time
Time
Figure 2: Chain rule graph for gradients w.r.t. weights of SNNs
1.0
k-
5
0.8
0.6
0.4
0.2
0.0
00.7
-0.1
12
0.7
1.7
2.2
x
Figure The surrogate function (left) under different values of the coefficient, & and its corresponding gradient (right). The blue curves represent the firing function (left) and its true gradient (right).
3.3 Optimizing Difficulty for SNNs with Large Time Steps
A notorious problem in SNN training is the non con-differentiabiliy of the firing function, see Bq.Eq. (3) To discuss this problem concretely, we denote the loss function as L and calculate the gradients w.r.t. weights using the chain rule following [51 shown in Fig. 2
and given by
OL aw
aL
Os
H+1
[1]
Ou' +1 [4.
Σ
II
Oult+ fi +++
Ou
[ + 1]
0s
Os
0u
Ou
(t]
OW
On
Ou
aW
(4)
where W represents the weights from layer / to / 1, T the total time steps, and L is the loss. The terms Dulld for firing function is non-differentiable. Its gradient is O almost everywhere except for the threshold. Therefore, the actual updates for weights would either be 0 infinity when recalling the gradient descent. To handle this problem, many surrogate gradient methods are proposed (49 58 19]. In this kind of method, when performing the forward pass, the firing function remains exactly the same, while, when for the back ward pass, the firing function will become surrogate function, and the surrogate gradient is computed based on it. A typically surrogate function may refer to the tanh-like given by
function [14 30
1
(x)) x tanh (K (a Voa)) +
(5)
where le a constant. The (xx and its gradient can be seen in Fig. 3 The surrogate gradient can be adjusted by changing k. Other widely used surrogate functions also enjoy the same characteristic, such as rectangular or sigmoid surrogate functions proposed in (49
It can be seen that, when k is set as large value, more accurate gradient in the backward pass can be obtained, i.e., the gradient will be sharp at a narrow range while gradual in the residual part. However, the gradient explode vanish problem will become more severe in this case since the final
5

---

100
300
(a)
(b)
(c)
13-
13-
2200000 dla 1
(d)
(c)
(f)
Figure 4: The gradient distributions of the first layer for Spiking PointNet on ModelNet40 with different and time steps. (a), (b), and (c) show the distributions for the Spiking PointNet using 1 single time steps with k = 0.5, 5, 20, respectively. (e), (d), and (f) show the distributions for the Spiking PointNet using 4 time steps with k 0.5,5, 20, respectively.
weight gradient is calculated by multiplying many surrogate gradients through layers and time steps according to Eq. (4) which tends be either very big or small. While, when le is set as a small value, more inaccurate gradient in the backward pass will be obtained [14] Hence the gradient error will be accumulated through layers and time steps, thus hurting the performance of the SNN too [48]. Consequently, it is very difficult to train well-performed SNN with large time steps directly, limited by the fact that there is no suitable surrogate gradient for this kind SNN.
3.4 The Trained-less But Learning-moo more Framework
As aforementioned,, except for the optimizing difficulty, there is no existing suitable way to train SNNs with large time steps common deep-learning devices for point clouds, since training network on point clouds is much energy and memory hungry. To handle these two problems simultancously, propose trained-less but learning- more framework.
To better describe the paradigm, we first show the gradient distributions of the first layer for Spiking
PointNet on the ModelNet40 in the Fig. 4 Here, we have several baselines: (1) The Spiking PointNet 0.5, 5, 20, respectively: (2) the Spiking PointNet using time steps along with 1: 0.5, 5, 20, respectively. It can be seen that, when k = 5, the gradient distribution for Spiking PointNet with single time step is relatively suitable. While k 20, the explode or vanish problem is very significant, and when k = 0.5, the distribution is relatively flat, which means it is different from the actual gradient greatly and the gradient error is huge. Hence, a small & a large & is not good idea for SNNs. The results in Tab. I also show that small & or large will reduce the SNN accuracy.
using single time step along with
Nevertheless, we can still find a relatively suitable surrogate function for the SNN with few time steps. However, the explode or vanish problem and the gradient error problem will be more severe with the time step increasing for SNNs. It can be seen that, although the k = 5 is good choice for the Spiking PointNet with single time step, the explode or vanish problem will become very severe for the Spiking PointNet with 4 time steps. Meanwhile, with the time step increasing, the gradient error problem becomes severe too. Note that, when k = 0.5, the gradient distribution for Spiking PointNet with 4 time steps becomes flatter, which means a huger gradient error.

---

Table 1: The accuracy for Spiking PointNet with different time steps and & on ModelNet40.
k
Time step
0.5
5
20
80.34%
86.98%
83.46%
4
76.73%
86.70%
75.36%
Consequently, it is not casy to train Spiking PointNet with large time steps. The Tab. also shows that the Spiking PointNet with time steps even performs worse than the one with only one single time steps. To this end, propose trained-less but Beariingmmoe framework. In specific, we train our Spiking PointNet with only a single time step but use it with multiple time steps in the inference time. By training SNNs with only be mitigated greatly. Thus we can choose a relatively large k. and meanwhile, the gradient error will be reduced at the same time. In the paper, we choose k as 5. The Tab. 2 shows the results of our trained-less but learning-more framework for Spiking PointNet on ModelNet10 and ModeINet40. can be seen that training the Spiking PointNet with suitable surrogate function will outperform the one with 4 time steps, and if we infer the trained model with multiple time steps, the accuracy will increase some still. Thus we name the paradigm the trained-less but learning- more framework.
single time step, the gradient explode vanish problem will
Table The ablation study for the trained-less but learning-mov more framework.
Training: T
Training:
T
Dataset
Inferring: 4T
Inferring: 1 T
Inferring: T
Inferring: 3 T
Inferring: 4 T
ModelNet10
91.05%
91.99%
92.43%
92.53%
92.32%
ModelNe140
86.70%
86.98%
87.26%
87.2 1%
87.13%
Training: T denotes training the Spiking PointNet with time steps. Inforring: n T denotes Inferring the Spiking PointNet with n time steps.
3.5 Membrane Potential Perturbation Method
An interesting phenomenon in our trained-less but learning-more framework that though the Spiking PointNet is trained with only single time step, in the inference, with the increasing of time steps, the accuracy will increase less or more the same time. Some work [37] 31 proves that the SNNs can extract spatio-temporal features for sequential data with multiple time steps. However, the point cloud the static data, thus there is no temporal feature extract. We guess that the reason for the accuracy increase of Spiking PointNet with multiple time steps is that it becomes an ensemble. The residual membrane potential along time steps in the spiking neuron can be seen as the perturbation. The perturbation will provide different initializations for the Spiking PointNet along time steps. Thus the Spiking PointNet at every time step can seen different model. And averaging their outputs can improve the uncertainty estimation and thus may lead to enhancement in SNN accuracy.
To verify our guess, in this section, conducted series of ablation experiments ModelNet40.. We trained the Spiking PointNet with 4 time steps and evaluated its accuracy at every time step and all time steps respectively. The results are shown in Tab. 3 It can be seen that, the collective results outperform those obtained from individual steps, implying that the performance improvement associated with larger time steps might be more related an ensemble learning effect, rather than direct result of the increased time steps. In specific, the Spiking PointNet at each time step can be seen as an independent model casting vote towards the final prediction. This ensemble learning strategy increases the robustness of the model and subsequently improves the prediction accuracy. Our study suggests that a rethinking and optimization of time steps SNNs is warranted. The inherent ensemble learning effect, which is underappreciated in the conventional SNN design, could be viable strategy to enhance the performance of SNNs, while also managing computational resources. Our insights provide valuable implications for future design and optimization strategies in the field of SNNs.
7

---

Table 3: The verification test for the effect of the time step the static dataset.
1-th time step
2-th time step
3-th time step
4-th time step
Averaging all
Accuracy
83.70%
84.65%
85.70%
85.29%
86.70%
Under the perspective that the residual membrane potential of SNN, coming from the previous time step cannot transmit the temporal information for static point cloud datasets but a perturbation to increase the generalization, we further propose membrane potential perturbation method for the framework. In specific, we add some membrane potential perturbation randomly to initial the spiking neurons of the Spiking PointNet at each epoch the training phase, thus the generalization of the model trained with only 1 single time step will be improved like those trained with multiple time steps. The results for the trained-less- based Spiking PointNet with membrane potential perturbation are shown in Tab. 4] It can be seen that with the perturbation method, the Spiking PointNet further gets another performance lift, amounting to 93.3 1.31% and 88.61 final accuracy for ModelNet and ModelNet40 respectively.
Table 4: The ablation study for the membrane potential perturbation.
Training: 1 T
Dataset
Method
Inferring: I
Inferring: T
Inferring: 3 T
Inferring: 4 T
without MPP
91.99%
92.43%
92.53%
92.32%
ModelNet10
with MPP
91.66%
92.98%
92.98%
93.31 %
without MPP
86.98%
87.26%
87.21%
87.13%
ModelNet40
with MPP
87.72%
88.46%
88.25%
88.61% %
MPP denotes membrane potential perturbation.
Experiments
In this section, conduct extensive experiments on ModelNet10 and ModelNet40 [50] to demon- strate the superior performance of our method. ModelNet10 and ModelNet40 are two widely recognized public datasets used for 3D object classification, curated and maintained by a research team at Princeton University. ModelNet10 is a compact dataset comprising 4,899 3D models that span 10 distinct categories such tables, chairs, bathtubs, and guitars. This dataset is a subset of ModeINet40, offering fewer categories but with more pronounced differences between each category. This characteristic makes ModelNet10 an excellent starting point for evaluating the performance of 3D classification algorithms. ModelNet40 is more comprehensive dataset, containing approximately 12,311 3D models across 40 different categories, including tables, chairs, airplanes, guitars, and more. With an expanded array of categories and samples, ModelNet40 serves as robust benchmark for gauging the performance of 3D classification algorithms in more complex and challenging tasks. We leverage the PointNet architecture for point cloud classification tasks. For all our SNN models, we set Vt, as 0.5, The initial perturbations, ö. range from 0 to 0.5.
4.1 Ablation Studies
We first conducted thorough ablation experiments of our method against the vanilla SNN for PointNet on the ModeINet10/40 datasets. The Tab. 3 displays the performances of various methods under different training and testing time steps. On the ModelNet10 dataset, our Spiking PointNet with membrane potential perturbation (MPP) reaches an accuracy of 93.31% with a testing time step of 4, which outperforms both the one without MPP (92.32%) and the ANN-based approach (92.98%). Even with a testing time step of 1, our Spiking PointNet with MPP still achieves an accuracy of 91. 91.66% surpassing the performance of vanilla Spiking PointNet trained with 4 time steps (89.62%).
This validates the effectiveness of our method. Further, on the Mede dataset, our Spiking PointNet with MPP attains accuracy of 88.61% % with testing time step of 4, also outperforming

---

Table 5: Comparison between our method and the vanilla SNN on ModelNer10/40 datasets.
Testing time steps
Datasets
Methods
Training time steps
2
3
ANN Vanilla SNN
92.98%
4
89, 89.62%
90.83%
91.05%
91.05%
ModeINet10
Ours without MPP Ours with MPP
1
91.99%
92.43%
92.53%
92.32%
1
91.66%
92.98%
92.98%
93.31% %
ANN Vanilla SNN
89.20%
4
85.59%
86.58%
86.34%
86. 70%
ModelNet40
Ours without MPP Ours with MPP
1
86.98%
87.26%
87.21 %
87.13%
87.72%
88.46%
88.25%
88.61% %
Table 6: Energy estimation of ANN (PointNet) and SNNs (Spiking PointNet) of computation.
Method
Time step
Acc.
#Add.
#Mult.
Energy
PointNet
92.98%
0.03M
13.94M
1.4 10 pJ
91.66%
0.45M
0.45M
9.2 10
Spiking PointNet
4
93.31%
1.8M
1.8M
3.7 x 10"pl
the without MPP (87.13%) and the vanilla Spiking PointNet (86. (0%). Similarly, even with testing time step of I, our Spiking PointNet with MPP achieves an accuracy of 87.72%, still superior the performance of the vanilla one trained with 4 time steps (85.59%).
4.2 Energy Efficiency
In this section, conducted comprehensive investigation into the hardware efficiency of proposed framework, with a focus on quantifying energy consumption in computational tasks on ModelNet10. For an ANN model, the dot product operation, or Multiply-Accumulae (MAC) operation, involves both addition and multiplication operations. However, the SNN leverages the multiplication-additin transformation advantage, eliminating the need for multiplication operations in all layers except the first layer. Remarkably, in the absence of spikes, hardware can employ sparse computation completely avoid addition operations. To estimate energy consumption, we adopted the methodology using 45mm CMOS technology following [26] 33]. The MAC operation in ANN consumes 4. 4.6pJ of energy, while the accumulation operation SNN requires only 0.9pJ. Notably, in line with our trained-less but learning- more paradigm, we achieved spike firing rate of 18.7% with Based on our findings, we computed the energy cost and presented the results Tab. 6 Our network exhibits remarkable energy efficiency, necessitating only 9.2 10'pJ of energy per forward pass, which equates to a 15.2-fold reduction in comparison to conventional ANNs. Moreover, when we conduct inference in four time steps, the performance reaches 93.31 %, while the energy required merely about 3.8 times less than that of its ANN counterpart.
5 Conclusion
In this paper, we have presented Spiking PointNet, the first spiking neural network (SNN) specifically designed for efficient deep learning on point clouds. This work was motivated by the tremendous potential of SNNs in energy efficiency and the rising demand for efficient point cloud processing techniques, especially in fields such as autonomous driving and augmented reality. We identified two main challenges hindering the application of SNNs in point cloud tasks: the intrinsic optimization difficulty of SNNs, and the high computational and memory cost of point cloud processing, especially for large time steps. To address these obstacles, we proposed novel trained-less but learning-mon more paradigm. This paradigm allows for the training of Spiking PointNet with only a single time step, but is capable of achieving superior performance through multiple time step inference. Theoretical justifications and experimental analysis provided in the paper support our method's effectiveness. Additionally, we introduced membrane potential perturbation method, which significantly en-

---

hanced the generalization ability of the Spiking PointNet without increasing computational and storage requirements. Our extensive experiments on multiple datasets, including ModelNet10 and ModelNet40, demonstrated the robustness and superiority of Spiking PointNet. Notably, in certain scenarios, Spiking PointNer was even able to outperform its Artificial Neural Network counterparts, an uncommon achievement in the SNN field.
Acknowledgment
This work is supported by grants from the National Natural Science Foundation of China under contracts 12204412 and No. No.12202413.
References
[1] Bi, G.q. Poo. M.m.: Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. Journal of neuroscience 18(24), 10464-10472 (1998)
[2] Bohte, S.M.: Error- backpropagation in networks of fractionally predictive spiking neurons. In: International Conference Artificial Neural Networks, pp. 60-68. Springer (2011)
[3] Bu, T., Ding, J., Yu, Z., Huang, T.: Optimized potential initialization for low-latency spiking neural networks (2022)
[4] Carnevale, N.T., Hines, M.L.: The NEURON book. Cambridge University Press (2006)
[5] Chen, Y., Zhang. S., Ren, S., Qu, H.: Gradual surrogate gradient learning in deep spiking neural networks. In: ICASSP 2022 2022 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8927-8931 (2022).
974477777977744
[6] Cheng, X., Hao, Y. Xu, Xu, B.: Lisnn: Improving spiking neural networks with lateral interactions for robust object recognition. In: UJCL pp. 1519-1525 (2020)
Choy, C., Gwak, J.. Savarese, S.: 4d spatio-temporal convnets: Minkowski convolutional neural networks.
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3075-3084 (2019)
Deng, S., Gu, S.: Optimal conversion conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103..m .00466 (2021)
[9] Dichl, P.U., Cook, M.: Unsupervised learning of digit recognition using piik--immng-deeeedeen plasticity. Frontiers in computational neuroscience 9, 99 (2015)
[10] Ding, J.. Yu, Z., Tian, Y., Huang. T.: Optimal ann-sm snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint 2005v 1655 (2021)
[11] Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang. T., Tian, Y:: Incorporating learnable membrane time constant enhance learning of spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1661-2671 (2021)
[12] Chosh-Dastidda S., Adeli, H.: Spiking neural networks. International journal of neural systems 19(04), 295-308 (2009)
[13] Guo, Y.. Chen, Y.: Neuroclip: Neuromorphic data understanding by clip and snn. arXiv preprint arXiv:2306. 12073 (2023)
[14] Guo, Y., Chen, Y., Zhang, L., Liu, X., Wang. Y., Huang, X., Ma, Z.: IM-loss: Information maximization loss for spiking neural networks. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022), httpp:://operreeiee fot//forum?id=J444 _4m22b
[15] Guo, Y.. Chen, Y., Zhang, L.. Wang, Y., Liu, X., Tong, X., Ou, Y., Huang, X., Ma, Z.: Reducing information loss for spiking neural networks. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision ECCV 2022. pp. 36 36-52. Springer Nature Switzerland, Cham (2022)
[16] Guo, Y., Huang, X., Ma, Z.: Direct learningbbased deep spiking neural networks: a review. Frontiers in Neuroscience 17, 1209795 (2023)
[17] Guo, Y., Liu, X., Chen, Y., Zhang, L., Peng. W., Zhang, Y., Huang, X., Ma, Z.: Rmp-loss: Regularizing membrane potential distribution for spiking neural networks. arXiv preprint #tXiv:2308.06787 (2023)
10

---

[18] Guo. Y., Peng, W. Chen, Y., Zhang, L.., Liu, X., Huang. X., Ma, Z.: Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recognition p. 109639 (2023)
[19] Guo, Y., Tong, X., Chen, Y., Zhang. I.., Liu, X., Ma, 7.,, Huang, X.: Reedis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 326-335 (June 2022)
[20] Guo, Y., Zhang. I.,, Chen, Y., Tong, X., Liu, X., Wang, Y., Huang. X., Ma, Z.: Real spike: Learning real- valued spikes for spiking neural networks. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII. pp. 52-68. 68. Springer (2022)
[21] Guo, Y., Zhang. Y., Chen, Y., Peng, W., Liu, X., Zhang, L., Huang. X., Ma. Z.: Membrane potential batch normalization for spiking neural networks. arXiv preprint arXiv:2308. 03359 (2023)
[22] Han, B., Srinivasan, G., Roy, K.; Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In: Procecdings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13558-13567 (2020)
[23] Hao. Y.. Huang. X., Dong, M.. Xu, B.: A biologically plausible supervised learning method for spiking neural networks using the symmetric stdp rule. Neural Networks 121, 387-395 (2020)
[24] Hao, Z., Bu, T., Ding, J., Huang, T., Yu, Z: Reducing ann-snn conversion error through residual membrane potential. arXiv preprint wtXXiv:2302.0009 (2023)
[25] Hao, Z., Ding, J., Bu, T., Huang, T., Yu, Z.: Bridging the gap between amns and snns by calibrating offset spikes. arviv preprint arXiv:2302. 10685 (2023)
[26] Horowitz, M.: computing's energy problem (and what we can about it). In: 2014 IEEE International Staid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, IEEE (2014)
[27] Hu. Q. Yang. B., Xie. L.. Rosa. S., Guo. Y., Wang. Z., Trigoni, N., Markham, A.: Randla-net: Efficient semantic segmentation large-scale point clouds. In: Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 11 1118-11117 (2020)
[28] Lee, M., Kim, H., Park, S., Yoon, M., Lee, J., Choi, J., Kang, M., Choi, J.: Pillarace: Sparse pointpillars accelerator for real-time point cloud 3d object detection on edge devices (2023)
[29] Li, Y., Deng, S., Dong, X., Gong. R., Gu, S.: A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In: International Conference on Machine Learning. pp. 6316-6325. PMLR (2021)
[30] Li, Y., Guo, Y., Zhang. S., Deng, S., Hai, Y., Gu, S.: Differentiable spike: Rethinking mrdieettdescent for training spiking neural networks. Advances in Neural Information Processing Systems 34, 23426-2333 -3439 (2021)
[31] Li, Y., Yin, R., Park, H., Kim, Y., Panda, P.: Wearable-based human activity recognition with spatio- temporal spiking neural networks. arXiv preprint arXiv:2212.02233 (2022)
[32] Liu, F., Zhao, W., Chen, Y.. Wang. Z., Jiang. L.: Spikeconverter: An efficient conversion framework zipping the gap between artificial neural networks and spiking neural networks. pp. 1692-1701. AAAI Press (2022)
[33] Lu, T., Liu, C., Chen, Y., Wu, G., Wang, L.: App-net: Auxiliary-point-based push and pull operations for efficient point cloud classification. arXiv preprint arXiv:2205.0088: 000447 (2022)
[34] Ma, X., Qin, C., You, H., Ran, H., Fu, Y.: Rethinking network design and local geometry in point cloud: A simple residual mlp framework. In: International Conference on Learning Representations
[35] Neftei, E.O., Mostafa, H., Zenke, F: Surrogate gradient learning in spiking neural networks: Bringing the power of gradicnt-basse based optimization to spiking neural networks. IEEE Signal Processing Magazine 36(6), 51-63 (2019)
[36] Ponghiran, W., Roy. K.: Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In: Proceedings of the AAAT Conference on Artificial Intelligence. vol. 36, pp- 8001-8008 (2022)
[37] Ponghiran, W., Roy, K.: Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 8001-8008 (2022)
11

---

[38] Qi. C.R., Su, H., Mo. K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation (2017)
[39] Rathi, N., Roy, K.: Diet-snn: Direct input encoding with leakage and threshold optimization deep spiking neural networks. arXiv preprint arXiv:2008.0366 (2020)
[40] Rathi, N., Srinivasan, G., Panda, P., Roy, K.; Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation, arXiv preprint arXiv:2005. 018000000 (2020)
[41] Ren, D., Li, J., Wu, 7., Guo, J., Wei, M., Guo, Y.: Mffnet: multimodal feature fusion network for point cloud semantic segmentation. The Visual Computer pp. 1-13 (2023)
[42] Ren, D., Wu, Z., Li. J., Yu, P., Guo, J., Wei, M.. Guo, Y:: Point attention network for point cloud semantic segmentation. Science China Information Sciences 65(9), 192104 (2022)
[43] Richardson, D.B.: Hebb, d. o, the organization of behavior. neuropsychological theory (1965)
[44] Shen, J., Xu, Q. Liu, J.K., Wang, Y., Pan, G., Tang, H.: Esl-snns: An evolutionary structure learning strategy for spiking neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 86 93 (2023)
[45] Viale, A., Marchisio, A., Martina, M., Masera, G., Shafique, M.: Lanesnns: Spiking neural networks for lane detection on the loihi neuromorphic processor. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 79 86. IEEE (2022)
[46] Wu, J., Xu, C., Zhou, D.,, Li, H., Tan, K.C.: Progressive tandem learning for pattern recognition with deep spiking neural networks (2020)
[47] Wu, J., Chua, Y., Zhang, M.. Li, G., Li, Tan, K.C.: A landem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions Neural Networks and Learning Systems (2021)
[48] Wu, J. Xu, C., Han, X.., Zhou, D., Zhang, M., Li, H., Tan, K.C.: Progressive tandem learning for pattem recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(11), 7824-7840 (2021)
[49] Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., Shi, L.: Direct training for spiking neural networks: Faster, larger, better. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 1311-1318 (2019)
[50] Wu, Z., Song, S., Khosla, A., Yu, F. Zhang, L.,, Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recogmition. pp. 1912-1920 (2015)
[51] Xiao, M., Meng, Q., Zhang, Z., Hc, D., Lin, Z.: Online training through time for spiking neural networks. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022),
[52] Xu, Q., Li, Y., Shen, J., Liu, J.K., Tang, H., Pan, G.: Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. arXiv preprint arXiv:2304.00 23662627706 (2023)
[53] Xu, Q., Li, Y., Shen, J., Zhang, P., Liu, J.K., Tang, H., Pan, G.: Hierarchical apiking-based model for efficient image classification with enhanced feature extraction and encoding. TEEE Transactions Neural Networks and Learning Systems (2022)
[54] Xu. Q.. Shen, J., Ran, X.. Tang, H., Pan, G., Liu. J.K.: Robust transcoding sensory information with neural spikes. IERE Transactions Neural Networks and Learning Systems 33(5), 1935-1946 (2021)
[55] Zenke, F., Ganguli. S.: Superspike: Supervised learning multilayer spiking neural networks. Neural computation 30(6), 1514-1544 1541 (2018)
[56] Zhang, M., Qu, H., Belatreche, A., Chen, Y.. Yi, Z.: A highly effective and robust membrane potential- driven supervised learning method for spiking neurons. IEEE transactions on neural networks and learning systems 30(1). 123 137 (2018)
[57] Zhang, M., Wang. J.. Wu, J., Belatreche, A., Amompaisannon, B., Zhang, Z., Miriyala, V.P.K., Qu, H., Chua, Y.. Carlson, T.E., et al.: Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE transactions on neural networks and learning systems 33(5), 1947- 1958 (2021)
[58] Zheng, H., Wu, Y. Deng, L., Hu, Y., Li, G.: Going deeper with directly trained larger spiking neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 11066-110700 (2021)
12


Reviews:
[{"review_id": "ddDcuYvdwI", "reviewer": "Reviewer_f5wg", "summary": "In this paper, the authors introduce the spiking neural network into 3D data processing, the point cloud data. In specific, the authors analyze the training difficulty when using SNNs for the point cloud recognition and find that number of timesteps should not be increased in the training phase. Their framework, the trained-less but learning more, obtains better performance than aligning the number of timesteps in training and inference. ", "questions": "Is it possible to convert an ANN PointNet to SNN?", "limitations": "See above for improvements", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ Well-studied SNNs on the point cloud dataset for the first time. This paper brings the advantage of energy efficiency of neuromorphic computing to 3D vision topic. \n\n+ Interesting, counter-intuitive findings but a surprisingly well-performance method. \n", "weaknesses": "- The authors claimed theoretical analysis in Section 3.4. However, I found it mostly empirical analysis. I wonder where is the theory?\n\n- What will happen if the timesteps are increased more than 4 during inference? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our paper and your recognition of our interesting findings, efforts for point cloud,  and the well-performance method. The response to your questions is given piece by piece as follows.\n\n---\n\n**Q1**: The authors claimed theoretical analysis in Section 3.4. However, I found it mostly empirical analysis. I wonder where is the theory.\n\n**A1**: Thanks for this advice. We present the proof to show that larger time steps will make the training of SNNs more difficult as follows.\n\nProof.\n\nThe gradients to weights have been given in our paper, as follows:\n$$\n\\frac{\\partial L}{\\partial \\mathbf{W}^l}=\\sum_{t=1}^T \\frac{\\partial L}{\\partial \\mathbf{s}^{l+1}[t]} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[t]}{\\partial \\mathbf{u}^{l+1}[t]}}\\left(\\frac{\\partial \\mathbf{u}^{l+1}[t]}{\\partial \\mathbf{W}^l}+\\sum_{\\tau<t} \\prod_{i=t-1}^\\tau\\left(\\textcolor{green}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{u}^{l+1}[i]}}+\\textcolor{blue}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{s}^{l+1}[i]}} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[i]}{\\partial \\mathbf{u}^{l+1}[i]}}\\right) \\frac{\\partial \\mathbf{u}^{l+1}[\\tau]}{\\partial \\mathbf{W}^l}\\right)\n$$\nSince the loss can be computed by:\n\n$$\nL = \\frac{1}{T}\\sum_{t=1}^{T} L_{CE}(y,o[t]).\n$$\nthe gradients to weights for large T and small T can be further updated as:\n$$\n\\frac{\\partial L^{large}}{\\partial \\mathbf{W}^l}=\\sum_{t=1}^{T^{large}} \\frac{\\partial L_{CE} / T}{\\partial \\mathbf{s}^{l+1}[t]} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[t]}{\\partial \\mathbf{u}^{l+1}[t]}}\\frac{\\partial \\mathbf{u}^{l+1}[t]}{\\partial \\mathbf{W}^l}+\\sum_{t=1}^T \\frac{\\partial L_{CE} / T}{\\partial \\mathbf{s}^{l+1}[t]} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[t]}{\\partial \\mathbf{u}^{l+1}[t]}}\\left(\\sum_{\\tau<t} \\prod_{i=t-1}^\\tau\\left(\\textcolor{green}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{u}^{l+1}[i]}}+\\textcolor{blue}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{s}^{l+1}[i]}} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[i]}{\\partial \\mathbf{u}^{l+1}[i]}}\\right) \\frac{\\partial \\mathbf{u}^{l+1}[\\tau]}{\\partial \\mathbf{W}^l}\\right)\n$$\n\nand \n\n$$\n\\frac{\\partial L^{small}}{\\partial \\mathbf{W}^l}=\\sum_{t=1}^{T^{small}} \\frac{\\partial L_{CE} / T}{\\partial \\mathbf{s}^{l+1}[t]} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[t]}{\\partial \\mathbf{u}^{l+1}[t]}}\\frac{\\partial \\mathbf{u}^{l+1}[t]}{\\partial \\mathbf{W}^l}+\\sum_{t=1}^T \\frac{\\partial L_{CE} / T}{\\partial \\mathbf{s}^{l+1}[t]} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[t]}{\\partial \\mathbf{u}^{l+1}[t]}}\\left(\\sum_{\\tau<t} \\prod_{i=t-1}^\\tau\\left(\\textcolor{green}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{u}^{l+1}[i]}}+\\textcolor{blue}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{s}^{l+1}[i]}} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[i]}{\\partial \\mathbf{u}^{l+1}[i]}}\\right) \\frac{\\partial \\mathbf{u}^{l+1}[\\tau]}{\\partial \\mathbf{W}^l}\\right)\n$$\n\nTo simplify the proof, $\\frac{\\partial L_{CE} / T}{\\partial \\mathbf{s}^{l+1}[t]}$, $\\frac{\\partial \\mathbf{s}^{l+1}[t]}{\\partial \\mathbf{u}^{l+1}[t]}$, etc, can be viewed as identical. Obviously, the first stem for the above two equations are same, while the second stem for the large T is larger.  It can also be easily understood when combined with Figure 2 in our paper. That is to say $\\sum_{\\tau<t} \\prod_{i=t-1}^\\tau\\left(\\textcolor{green}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{u}^{l+1}[i]}}+\\textcolor{blue}{\\frac{\\partial \\mathbf{u}^{l+1}[i+1]}{\\partial \\mathbf{s}^{l+1}[i]}} \\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[i]}{\\partial \\mathbf{u}^{l+1}[i]}}\\right) \\frac{\\partial \\mathbf{u}^{l+1}[\\tau]}{\\partial \\mathbf{W}^l}$, denoted as **G**, will contribute more to the large T. \n\nAs we have mentioned in our paper, the $\\textcolor{red}{\\frac{\\partial \\mathbf{s}^{l+1}[i]}{\\partial \\mathbf{u}^{l+1}[i]}}$ can be controlled by a $k$. When $k$ is large, a more accurate gradient in the backward pass can be obtained, while the gradient will be sharp at a narrow range while very small in the residual part. Thus the gradient explode or vanish problem will become severe in this case. Due to the multiple connected multiplication characteristic of **G**, the gradient explode or vanish problem will be more severe. While $k$ is small,  an inaccurate gradient in the backward pass will be obtained. Hence more gradient error will be accumulated through time steps based on the multiple connected accumulation characteristic of **G**, thus hurting the performance of the SNN too.\n\nConsequently, it is very difficult to train a well-performed SNN with large timesteps directly.\n\n---\n\n**Q2**: What will happen if the timesteps are increased more than 4 during inference?\n\n**A2**: Thanks for the question. With the timesteps increasing, the accuracy will converge to a relatively high value, better than the single time step. We provide more results on the ModelNet10 and ModelNet40 as below.\n\n| Datasets\\Timestep | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ModelNet10 | 91.66% | 92.98% | 92.98% | 93.31% | 92.94% | 93.37% | 93.28% | 93.21% |\n| ModelNet40 | 87.72% | 88.46% | 88.25% | 88.61% | 88.41% | 88.73% | 88.20% | 88.59% |\n\n**Q3**: Is it possible to convert an ANN PointNet to SNN?\n\n**A3**: Yes. The PointNet is also composed of convolution layers, linear layers, BN layers, and Pooling layers as ResNet, which have been proven could be converted to SNN in these ANN-SNN works.\n\n---\n\nIf you have further questions, please do not hesitate to reply to us. Thanks again."}, {"review_id": "7DEAJMvWD5", "reviewer": "Reviewer_CQRm", "summary": "The authors replaced the ReLU activation function of the original PointNet with the LIF neurons and named it spiking PointNet. The work offers trained-less but learning-more tricks and corresponding analysis. The effectiveness of spiking PointNet is evaluated on the ModelNet10/40 datasets, specifically in point cloud classification.", "questions": "Please see weakness section", "limitations": "No limitation is discussed.\n\nIf the work only replaces ReLU with LIF, I think the authors should show whether the replacement and training tricks would work with other models, such as VGG, and ResNet. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In general, the submission is well-written and easy to follow.  ", "weaknesses": "-1. If I understand correctly, the authors only replaced ReLu with LIF. If so, whether the first or not does not make sense as we can make first spiking VGG-16, first spiking ResNet, etc. \n\n-2. It is unclear to me what's the output of a LIF neuron. Is it a potential value or spiking? I guess the authors used potential value. \n\n-3. How does the leaky part of LIF impact the spiking PointNet? The authors claimed inference multiple times offers better performance. I am wondering, if we input a point cloud multiple times with different time intervals, what are the performance differences?\n\n-4. What's the processing speed with different T? \n\n-5. The validations are only with ModelNet. I think validations with more datasets are necessary, such as ShapeNet dataset. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our paper and your recognition of our effective method and well-written submission. The response to your questions is given piece by piece as follows.\n\n---\n\n**Q1**: If I understand correctly, the authors only replaced ReLu with LIF. If so, whether the first or not does not make sense as we can make first spiking VGG-16, first spiking ResNet, etc.\n\n**A1**: Sorry for this confusion. Since it still remains underexplored whether SNNs can be generalized to 3D recognition, we try to apply the SNN to deep learning on point clouds. We then found it challenging to train a big spiking model with large time steps for point cloud due to that the intrinsic optimization of the SNN with large time steps, and the expensive memory and computation cost of deep learning for point cloud that makes training a big spiking point model unrealistic. To solve the problems simultaneously, we propose a trained-less but learning-more paradigm and a membrane potential perturbation method. We will emphasize our attempt, findings, train of thought, and the solution more in the final version to highlight our contributions while weaken the statement that the first spiking neural network approach to deep learning on point clouds. Thanks for your suggestion.\n\nIn addition, we guess you may think the trained-less but learning-more paradigm is a general framework that could be applied to other networks and domains, thus it can not be seen the first SNN for point cloud. I partly agree with this. However, the trained-less but learning-more paradigm is a counter-intuitive finding. The more timesteps, the higher accuracy seems have been a consensus in the prior works. Hence, someone may have already tried to apply the SNN to point clouds in the past. However, because deep learning for point clouds is memory-intensive and the challenges posed by large timesteps, they were unable to train the SNN effectively, especially when constrained by limited hardware. We also face the same problem, but with theoretical justifications and in-depth experimental analysis, we find that training the SNN with only a single timestep is better and can solve the problem of applying SNN to point cloud. On top of that, our method can also be used for other domains and networks, this shows the novelty of our train of thought to try for point cloud first and then the generalization of our method. \n\n---\n\n**Q2**: The output of a LIF neuron.\n\n**A2**: We apologize for the confusion. The output of a LIF layer is a binary spike map. It could be found in Eq.3 in our paper and line 40-70 in spike_layer_without_MPR.py in our provided code. We will clearly highlight the output in the final version as your suggestion. Thanks.\n\n---\n\n**Q3**: How does the leaky part of LIF impact the spiking PointNet? If we input a point cloud multiple times with different time intervals, what are the performance differences?\n\n**A3**: Thanks for the question. For the leaky part, we add the ablation study with different leaky values on ModelNet10 as below. It can be seen that the leaky should not be too small or large. When it $\\in [0.15,0.3]$, the performance is relatively better. We will add the findings as your suggestion in the next version. Thanks.\n\n| Leaky part \\ Timestep | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- |\n| 0.05 | 90.59% | 90.72% | 91.21% | 91.36% |\n| 0.1 | 90.61% | 90.99% | 90.94% | 91.59% |\n| 0.15 | 91.05% | 92.23% | 92.73% | 92.96% |\n| 0.2 | 91.74% | 92.53% | 93.07% | 93.26% |\n| 0.25 | 91.66% | 92.98% | 92.98% | 93.31% |\n| 0.3 | 91.22% | 92.75% | 93.21% | 93.28% |\n| 0.35 | 90.82% | 91.15% | 91.75% | 92.38% |\n| 0.5 | 89.67% | 90.02% | 91.39% | 91.43% |\n\nAs for the different time intervals, we add the ablation study on ModelNet10 further show the results of every time step and different time intervals as below. It can be seen with multiple times with different time intervals, it is all better than the single one time step, and with more timesteps is some slightly better than the less one in more times. We will add the findings as your suggestion in the next version. Thanks.\n\n|  Timestep | 1th | 2th | 3th | 4th | 5th | 6th | 7th | 8th |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Accuracy | 91.66% | 91.21% | 91.65% | 90.85% | 91.43% | 91.77% | 91.18% | 90.35% |\n| Time interval | 1-2 | 2-3 | 3-4 | 4-5 | 5-6 | 6-7 | 7-8 |  |\n| Accuracy | 92.98% | 92.17% | 91.96% | 92.01% | 92.96% | 93.08% | 91.75% |  |\n| Time interval | 1-4 | 2-5 | 3-6 | 4-7 | 5-8 |  |  |  |\n| Accuracy | 93.31% | 93.24% | 93.42% | 93.27% | 93.19% |  |  |  |\n\n---\n\n**Q4**: What's the processing speed with different T?\n\n**A4**: Thanks for the question. We add the time for 1 epoch inference with batchsize as 48 on ModelNet10 based on a single 3090 as follows. More details, please see our general response. We will add this in the next version. Thanks.\n\n| Timestep | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- |\n| Time(s) | 0.3934 | 0.4758 | 0.5930 | 0.6624 |\n\n---\n\n**Q5**: Validations with ShapeNet dataset.\n\n**A5**: Thanks for the question. We add the experiments on ShapeNet as follows. It also clearly shows our method’s effectiveness. We will add this in the next version. Thanks.\n\n| Methods\\Timestep | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- |\n| PointNet | 83.63% | - | - | - |\n| Spiking PointNet | 81.52% | 82.88% | 82.65% | 82.95% |\n\n---\n\n**Q6**: Whether the replacement and training tricks would work with other models, such as VGG, and ResNet.\n\n**A6**: Thanks for your suggestion, we also add the experiments on CIFAR-10 using ResNet20 and VGG16 as follows.  It shows our method’s generalization. We will add this in the next version. Thanks.\n\n| Methods\\Timestep | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- |\n| ResNet20  | 91.86% | 92.64% | 93.10% | 93.09% |\n| VGG16 | 94.09% | 94.94% | 94.95% | 94.82% |\n\n---\n\nIf you have further questions, please do not hesitate to reply to us. Thanks again."}, {"review_id": "HzwtRgV2SH", "reviewer": "Reviewer_zgZ2", "summary": "To investigate whether SNN can be suitable for 3D data, this paper proposes a Spiking PointNet for point cloud classification tasks. The authors mainly focus on two key issues: (i) Training SNNs with large time steps; (ii) Reducing the extensive memory and computation cost for the proposed spiking point model. The results on ModelNet10 and ModelNet40 datasets show the effectiveness of the proposed Spiking PointNet.", "questions": "i) Could you please explain why the proposed SNN method has better performance than the corresponding ANN? I am very curious about this conclusion. I would like more theoretical explanations, especially informational theory, rather than experimental explanations.\n\nii) Could the proposed Spiking PointNet be deployed on a neuromorphic chip? Please explain from the perspective from the designed structure of Spiking PointNet.\n\niii) Why did the authors add the MPP module to Spiking PointNet? Perhaps, the MPP is to improve the performance. However, this module is independent and generic for any SNN model. Thus, it is difficult to form a whole with the topic of the proposed Spiking PointNet.\n", "limitations": "No. Some suggestions may be improved for Spiking PointNet: (i) Processing point clouds from an asynchronous way or event-driven SNN; (ii) Leveraging temporal cues from point cloud streams; (iii) Attempting to processing event-based data (i.e., 3D event cloud) from event cameras.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "i) The topic of SNN for 3D point cloud classification is novel rather than 2D visual recognition.\n\nii) The writing is straightforward, clear, and easy to understand.\n", "weaknesses": "i) The two key issues (i.e., training large times steps, and reducing the extensive memory and computation cost) do not seem to be well solved. Table 1 and Table 2 demonstrate that the max time step is 4. Table 6 shows that the energy of Spiking PointNet is only 3.8 times less than of its ANN counterpart. In other words, the Spiking PointNet may bring a huge increase in inference speed than ANN-based architecture, but the reduction in power consumption is almost weak.\n\nii) The simulation calculation of Spiking PointNet power consumption in Section 4.2 does not seem to be so convincing. The proposed Spiking PointNet are not really deployed on neuromorphic chips (e.g., Loihi2), thus the simple power consumption simulation does not make sense. As far as I know, most existing SNNs cannot be deployed to neuromorphic chips. Please explain whether the proposed Spiking PointNet can be friendly to deploying to neuromorphic chips with lower consumption.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for recognizing our contributions in point cloud research, our effective method, and clear writing. We will address your concerns and questions in the following response.\n\n**Q1**: The two key issues (i.e., training large timesteps, and reducing the extensive memory and computation cost) do not seem to be well solved. Table 1 and Table 2 demonstrate that the max time step is 4. Table 6 shows that the energy of Spiking PointNet is only 3.8 times less than that of its ANN counterpart. In other words, the Spiking PointNet may bring a huge increase in inference speed than ANN-based architecture, but the reduction in power consumption is almost weak.\n\n**A1**: Sorry for this confusion. The advantage of SNN is to reduce the power consumption, not to increase the inference speed, even will reduce the speed. This is the difference of SNN and ANN, not our weakness. However, compared to other SNNs, we truly have reduced the time steps and extensive memory. Our contribution is to improve SNNs compared with other SNN works, not to improve the disadvantage of the SNN to compare with the ANN. We will further emphasize this in the next version. Sorry for this confusion again. More details, please see the general response.\n\n---\n\n**Q2**: The proposed Spiking PointNet are not really deployed on neuromorphic chips (e.g., Loihi2), thus the simple power consumption simulation does not make sense. As far as I know, most existing SNNs cannot be deployed to neuromorphic chips. Please explain whether the proposed Spiking PointNet can be friendly to deploying to neuromorphic chips with lower consumption.\n\n**A2**: Thanks for this suggestion. The spiking PointNet is only composed of convolution layers, linear layers, BN layers (which can be folded to the convolution layers), LIF layers, and average pooling layers, which are the commonly used modules that have be proven could be deployed to the Loihi in [1][2], and so on. The only obstacle is that limited by the development of the neuromorphic hardware, it is not easy to deploy a large model on them. However, this is also the advantage of the whole SNN domain. On top of that, we provide a relatively fair comparison as in other SNN works.\n\n[1] LaneSNNs: Spiking Neural Networks for Lane Detection on the Loihi Neuromorphic Processor, IROS 2022.\n\n[2] An Efficient Spiking Neural Network for Recognizing Gestures with a DVS Camera on the Loihi Neuromorphic Processor, IJCAI 2020.\n\n---\n\n**Q3**: Could you please explain why the proposed SNN method has better performance than the corresponding ANN? I am very curious about this conclusion. I would like more theoretical explanations, especially informational theory, rather than experimental explanations.\n\n**A3**: Thanks for the question. The ANN can be divided into two parts as in many works like [1,2], the first part can be viewed as a feature encoder, $f_e$, and the second part can be viewed as a classifier, $f_c$. The output feature is a $C \\times 1$ vector, computed by $f_e(x)$. It is shown crucial that output feature representation is rich and powerful for training a highly accurate model for the classification in many recent works like [1]. For the PointNet, the representation output feature of SNN has a chance to outperform the ANN. Take the information entropy concept to analyze it, given a set, $M$, its representation capability, $\\mathcal{R}(M)$ can be measured by the information entropy of $M$, as follows\n$$\n\\mathcal{R}(M) = \\max \\mathcal{H}(M) = \\max (-\\sum_{m \\in M } p_M(m){ log}p_M(m)),\n$$\nwhere $p_M(m)$ is the probability of a sample, $m$ from $ M$. When $p_M(m_1)=p_M(m_2)=\\cdots = p_M(m_N)$, $\\mathcal{H}(M)$ reaches its maximum, $log(N), $where $N$ is the total number of the samples from $M$. Since a single channel needs 32 bits to represent, consisting of $2^{32}$ samples, the $\\mathcal{R}(M) = log(2^{32\\times C}) = 32\\times C$ for the output feature of ANN, where C is the channel number. While for one timestep of the SNN, the output feature comes from the average pooling of 1024 points. One point output a binary spike, thus the average pooling needs $log(1024) = 10$ bits to represent.  Considering the multiple timesteps, it will increase to $10 \\times T$ bits. the $\\mathcal{R}(M) = log(2^{10 \\times T \\times C}) = 10 \\times T \\times C$ for SNN, where T is timesteps. With the increasing of T, the representation output feature of SNN has a chance to outperform the ANN. \n\nBut for the image task, like CIFAR, the average pooling only needs $log(4) = 2$ bits to represent, with the increasing of T, its training difficulty will increase greatly too. Hence it is not easy to outperform ANN in image tasks.\n\n[1] A Simple Framework for Contrastive Learning of Visual Representations. \n\n[2] Learning Transferable Visual Models From Natural Language Supervision.\n\n---\n\n**Q4**: Could the proposed Spiking PointNet be deployed on a neuromorphic chip?\n\n**A4**: Thanks for the question. Please see our response to **Q2**.\n\n---\n\n**Q5**: Why did the authors add the MPP module to Spiking PointNet?  it is difficult to form a whole with the topic of the proposed Spiking PointNet.\n\n**A5**: Thanks for the question. The residual membrane potential along time steps in the spiking neuron for point cloud can be seen as the perturbation which can improve the uncertainty estimation and thus may lead to an enhancement in SNN accuracy, see line 224-228 in our paper. However, to solve the expensive memory of deep learning for point cloud, we propose the trained-less but learning-more paradigm, which will lose the residual membrane potential perturbation with only a single time step. Then to retain the residual membrane potential perturbation advantage for the multiple timesteps again, we propose and embed the MMP method in the proposed trained with one timestep paradigm.\n\n---\n\nIf you have further questions, please do not hesitate to reply to us. Thanks again."}, {"review_id": "RT6sQZ42V3", "reviewer": "Reviewer_o3KR", "summary": "This paper presents the Spiking PointNet, a novel solution for processing point cloud data. The authors propose a \"train-less, learn-more\" strategy, which leverages the Surrogate Gradient (SG) method to train the SNN over a single time step, while deploying multiple time steps during the inference phase to optimize performance. The authors provide both theoretical analysis and empirical evidence to demonstrate that their unique strategy surpasses the conventional method of training the SNN over multiple time steps. To further enhance the performance, they introduce a membrane potential perturbation method into their framework. ", "questions": "Please see weaknesses.", "limitations": "I find no potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The \"train-less, learn-more\" strategy is interesting.\n2.The result is really good on various datasets, even outperforms its ANN counterpart in certain scenarios, an achievement rarely seen in SNNs.\n3.The paper is characterized by its clear and concise writing style, complemented by easily comprehensible illustrations.", "weaknesses": "1.Section 3.4 seems unclear. If the gradient distributions, as depicted in Figure 4, do not match up as expected, would the membrane potential perturbation method introduced in Section 3.5 still be effective? Please clarify this point.\n2.Can this method be applied to point segment task?\n3.In table 6, the 10^8 is misspelling. Please correct it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time in reviewing our paper and your acknowledgment of our work on point cloud, our novel \"train-less, learn-more\" approach, and our clear writing complemented by understandable illustrations. Please find our point-by-point responses to your queries as follows.\n\n**Q1**: Section 3.4 seems unclear. If the gradient distributions, as depicted in Figure 4, do not match up as expected, would the membrane potential perturbation method introduced in Section 3.5 still be effective? Please clarify this point.\n\n**A1**: Our Membrane Potential Perturbation (MPP) is independent of gradient distributions. It's designed to counterbalance the loss of residual membrane potential when adopting a trained-less but learning-more strategy with a single timestep. Simply put, MPP ensures the advantages of membrane potential variations persist across multiple timesteps in our \"single timestep training\" setup. This holds true regardless of gradient specifics, as MPP always functions effectively. We've substantiated this with ModelNet10 dataset experiments for k=0.5, 5, and 20, demonstrating MPP's robustness under different conditions.\n\n| $k$\\Timestep | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- |\n| 0.5 | 87.34% | 88.18% | 89.46% | 90.07% |\n| 5 | 91.66% | 92.98% | 92.98% | 93.31% |\n| 20 | 88.93% | 89.70% | 90.36% | 91.05% |\n\n---\n\n**Q2**: Can this method be applied to point segment task?\n\n**A2**: Indeed, our method is applicable to point cloud segmentation tasks. As evidenced in the table below, we provide the results of part-segmentation experiments conducted on the ShapeNet dataset. These results include comparisons between PointNet and our Spiking PointNet across various testing time steps.\n\n| Datasets | Methods\\Timestep | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- | --- |\n| ShapeNet | PointNet | 83.63% | - | - | - |\n| ShapeNet | Spiking PointNet | 81.62% | 82.88% | 82.65% | 82.95% |\n\n---\n\n**Q3**: Misspelling.\n\n**A3**: Errors of symbols and misspellings are possibly caused by typos. We will thoroughly review and correct these in the camera-ready version.\n\n---\n\nIf you have further questions, please do not hesitate to reply to us. Thanks again."}]

Author rebuttal:
We thank all reviewers’ time and effort in reviewing our paper.  While we will respond to the comments piece by piece, we would like to first provide a general response to the common concern about the inference speed of spiking PointNet.

The advantage of SNNs is to reduce the power consumption, not to increase the inference speed and even will reduce the speed. This is the difference of SNN and ANN, not our weakness. However, compared to other SNNs, we have significantly reduced the time steps and extensive memory usage. Our contribution is to improve SNNs compared with other SNN works, not to improve the disadvantage of the SNN to compare with the ANN. We will further emphasize this in the next version. Sorry for this confusion again.

In addition, the integration of storage and computation paradigm and the asynchronous parallel processing on neuromorphic hardware, like Darwin[1], Tianjic[2], and other memristor-enabled neuromorphic computing systems[3], makes SNNs much different from ANNs. In these hardware, once the previous time step map has been processed by one layer and presented to the next layer to deal with, the layer would process the next time step data right after that while the next layer processes the presented map. Due to this kind of asynchronous parallel processing paradigm, the inference speed is almost unaffected with the time step increasing.

Even on the GPU, the reduction in speed is not linear with the increase in timesteps. We add the time for 1 epoch inference with batch size as 48 on ModelNet10 based on a single 3090 as follows. The inference time for 4 timesteps is only 1.68 times to 1 timesteps.

| Timestep | 1 | 2 | 3 | 4 |
| --- | --- | --- | --- | --- |
| Time(s) | 0.3934 | 0.4758 | 0.5930 | 0.6624 |

[1] Darwin: A neuromorphic hardware co-processor based on spiking neural networks.

[2] Towards artificial general intelligence with hybrid tianjic chip architecture. Nature.

[3] Fully hardware-implemented memristor convolutional neural network. Nature.

Meta-review: The reviewers mentioned several positive aspects of the paper, e.g.:
- Interesting, counter-intuitive findings but a surprisingly well-performance method.
- The paper is well-written and easy to follow.
- The "train-less, learn-more" strategy is interesting.
- The result is really good on various datasets, even outperforms its ANN counterpart in certain scenarios, an achievement rarely seen in SNNs.

However, they also raised multiple doubts about it, such as:
- The presented theoretical analysis in the rebuttal should be included in the appendix.
- Some parts of the paper (e.g., replacing LIF with ReLUs, outputs of a LIF neuron) are unclear.
- The two key issues (i.e., training large time steps, and reducing the extensive memory and computation cost) do not seem to be well solved.

The paper is borderline (7-6-4-4), however, the reviewers did a great job in the rebuttal to address all issues raised by the reviewers. Some reviewers still have som reservations, but overall the paper is well-written and some confusing parts could be easily addressed in the appendix. As a result, I suggest accepting the paper.
Decision: Accept (poster)