Autonomous Driving with Spiking Neural Networks
Authors: Rui-Jie Zhu
Ziqing Wang
Leilani H. Gilpin
Jason Eshraghian
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking Neural Networks
Neuromorphic Computing
Brain-inspired Computing
Primary area: neuroscience_and_cognitive_science
TL;DR: We present Spiking Autonomous Driving (SAD), the first unified Spiking Neural Network (SNN) to address the energy challenges faced by autonomous driving systems through its event-driven and energy-efficient nature.

Abstract: Autonomous driving demands an integrated approach that encompasses perception, prediction, and planning, all while operating under strict energy constraints to enhance scalability and environmental sustainability. We present Spiking Autonomous Driving (SAD), the first unified Spiking Neural Network (SNN) to address the energy challenges faced by autonomous driving systems through its event-driven and energy-efficient nature. SAD is trained end-to-end and consists of three main modules: perception, which processes inputs from multi-view cameras to construct a spatiotemporal bird's eye view; prediction, which utilizes a novel dual-pathway with spiking neurons to forecast future states; and planning, which generates safe trajectories considering predicted occupancy, traffic rules, and ride comfort. Evaluated on the nuScenes dataset, SAD achieves competitive performance in perception, prediction, and planning tasks, while drawing upon the energy efficiency of SNNs. This work highlights the potential of neuromorphic computing to be applied to energy-efficient autonomous driving, a critical step toward sustainable and safety-critical automotive technology. Our code is available at [https://github.com/ridgerchu/SAD](https://github.com/ridgerchu/SAD).

Autonomous Driving with Spiking Neural Networks
Rui-Jie Zhu', ZZiqing Wang"* Leilani Gilpin' Jason K. Eshraghian '
University of California, Santa Cruz, USA
Northwestern University, USA
Abstract
Autonomous driving demands an integrated approach that encompasses percep- tion, prediction, and planning, all while operating under strict energy constraints to enhance scalability and environmental sustainability. We present Spiking Au- tonomous Driving (SAD), the first unified Spiking Neural Network (SNN) address the energy challenges faced by autonomous driving systems through its event-driven and energy-efficiee nature. SAD is trained end-to-end and consists of three main modules: perception, which processes inputs from multi-view cam- eras construct spatiotemporal bird's eye view; prediction, which utilizes a novel dual-pathway with spiking neurons to forecast future states; and planning, which generates safe trajectories considering predicted occupancy, traffic rules, and ride comfort. Evaluated on the nuScenes dataset, SAD achieves competitive performance in perception, prediction, and planning tasks, while drawing upon the energy efficiency of SNNs. This work highlights the potential of neuromorphic computing to be applied to energy-efficient autonomous driving, critical step toward sustainable and safety-critical automotive technology. Our code is available at htttps://github wwwiii..ooomriiii com/ridgerchu/Si.
Introduction
Autonomous driving, often considered the 'holy grail' of computer vision, integrates complex processes such perception, prediction, and planning achieve higher levels of vehicle automation as classified by the SAE J3016 standard [ Many new vehicles now feature Level 2 autonomy, with transitions Level marking notable advancements. However, these systems must adhere to energy constraints of 50-60 W/h (2) and face increasing environmental concerns. Sudhakar et al. highlight the need for hardware efficiency double every 1.1 years to maintain 2050 emissions from autonomous vehicles below those of 2018 data center levels 3
Spiking Neural Networks (SNNs) offer promising solution for energy-efficient intelligence by using sparse, event-driven, single-bit spiking activations for neternneuron communication, mimicking biological neurons
5 6 7]. Such workloads can be accelerated for low latency and low energy, when processed on neuromorphic hardware that utilizes asynchronous, fine-grain processing to efficiently handle spiking signals and parallel operations (8). Much like in the brain, spikes are thought to encode information over time, and have shown improvements in the energy efficiency of sequence- based computer vision tasks by several orders of magnitude in a variety of workloads 9 10 11]
the past several years, SNNs have rapidly improved in performance across various tasks, including image classification 13 14
15
16 17] object detection [18 19 20 semantic segmentation 21],
low-level image reconstruction of these works focused on computer vision. These advancements have brought SNN performance
22
23 24 26), and language modeling (27 28 (2), with most
'Equal contribution
Corresponding author, jsn @ uesc.edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

View
Spiking Driving (BAD) Model
Duppet
Porcaption
--B
Prediction
Speed m0
Planning
Figure How SAD enables autonomous driving from vision to planning: The system processes inputs from six cameras across multiple frames. The perception module encodes feature information related to the present input frame (T = n). the prediction module predicts feature information of the next frame using sequential information (T' = + 1), and the model output generates steering and acceleration plan. This process creates bird's eye view (BEV) and trajectory plan for navigation.
closer to that of Artificial Neural Networks (ANNs) in fundamental computer vision tasks. Despite this progress, SNNs have not yet proven effective in complex real-world computer vision applications that involve multiple subtasks.
We introduce the first SNN designed for end-to-end autonomous driving, integrating perception, prediction, and planning into a single model. Achieving this milestone for SNNs involved spa- tiotemporal fusion of visual embeddings for enhanced perception, probabilistic future modeling for accurate prediction, and and a high performance temporal mixing spiking recurrent unit that effectively incorporates safety and comfort considerations into high-level planning decisions. By leveraging the event-driven and energy-efficien properties of SNNs, our model processes visual inputs, forecasts future states, and calculates the final trajectory for autonomous vehicles. Previously, GRUs or 3D convolutions were used in spatiotemporal visual tasks, though these operators have been entirely replaced with spiking neurons for time-mixing. This work marks a significant advancement in neuromorphic computing. demonstrating the potential of SNNs to handle the complex requirements of low-power autonomous driving. Our experiments show that this SNN-based system performs competitively with traditional deep learning approaches, while offering improved energy efficiency and reduced latency.
2 Related Works
Spiking Neural Networks in Autonomous Systems are particularly effective for low-power, edge intelligence applications, leveraging deep learning and neuroscience principles to boost operational efficiency 66] 4 30
31 32). Many neuromorphic autonomous systems use SNNs as a Proportional Sndivative-Intenne (PID) controller to adapt to changing conditions, such as different payloads in unmanned acrial vehicles [3
34 35 36), to prevent drift in non-ncutral buoyancy blimps [37]. Much of this work successfully deployed SNNs as PID controllers in real-world systems on neuro- morphic hardware, highlighting the potential for low-power autonomous control. Moving from the sky to the ground, SNN-based PID controllers have been used for tne-keeping tasks in simulated environments with reference trajectories provided by the lane [8] 39 40, as well as with LiDAR for collision avoidance in simulated environments [1]. These tasks all show successful use of SNNs in adaptive control, though the objective of PID controller to maintain a desired setpoint which often well-defined and simpler goal than end-to-end autonomous driving in the face of complex and noisy environments. We push the frontier of what SNNs are capable of in this paper.
Driving directly maps sensory inputs to vehicle control outputs using a single, fully differentiable model. Existing approaches can be broadly classified into two categories: imitation learning and reinforcement learning paradigms 42]. Imitation learning methods, such
End-to-end Autonomous
as behavior cloning [3
50 52]. learn driving policy by mimicking expert demonstrations. On the other hand, reinforcement learning
44
45
46, 47 and inverse optimal control [48]
techniques [53 (4|55]
(55 (56] enable the driving agent to learn through interaction with the environment by optimizing a reward function. Recent advancements, such as multi-modal sensor fusion [57
58 59, attention mechanisms
[57 62 58), and policy distillation
4
65 66
have
significantly improved the performance of end-to-end driving systems.

---

Perception: Encoder
Prediction
Perception: Decader
Planning
R Dlaclir
Padeettan
Ilde
Moode Ty
GOd
Salking Reaille
Sulking Baeplab Head
Diiaa Arts
Pr Sig Mouros
Scilionck
Duand
Figure 2: Overview of SAD. The multi-view features from the perception encoder, including a spiking ResNet with inverted bottlnneck and spiking DeepLab head, are fed into prediction module using spiking neurons. The perception decoder then generates lane divider, pedestrian, vehicle and drivable area predictions. Finally, the planning module models the scene and generates future predictions to inform rule-based command decisions for turning, stopping, and goal-directed navigation.
3
Method
This section presents the Spiking Autonomous Driving (SAD) method, an end-to-end framework that integrates perception, prediction, and planning using SNNs (Fig. 2 SAD's biologically-inspire. architecture enables efficient spatiotemporal processing for autonomous driving, with the spiking neuron layer its core. This layer incorporates spatiotemporal information and enables spike-driven computing, making it well-suited for the dynamic nature of autonomous driving tasks.
The perception module is the stage of the SAD framework. It constructs bird's eye view (BEV) representation from multi-view camera inputs, providing a human-interpretahle the environment. This representation serves the foundation for the subsequent prediction and planning modules. The prediction module uses the BEV to forecast future states using a 'dual pathway", which allows data flow through two separate paths, providing a pair of alternative data embeddings. One pathway focuses on encoding information from the past, while the other pathway specializes in predicting future information. Subsequently, the embeddings from these two pathways are fused together, integrating the past and future information to facilitate temporal mixing. This enables the anticipation of dynamic changes in the environment, which crucial for safe and efficient autonomous driving. Leveraging the perception and prediction outcomes, the planning module generates safe trajectories by considering predicted occupancy of space around the vehicle, traffic rules, and ride comfort. To optimize the entire pipeline, SAD is trained end-to-end using a composite loss that combines objectives from perception, prediction, and planning. The following subsections describe each module in detail.
understanding of
3.1 Spiking Neuron Layer
All modules consist of spiking neurons rather than artificial neurons, and so a formal definition of spiking neurons is provided below. Spiking neuron layers integrate spatio-temporal information into the hidden state of each neuron (membrane potential) which are converted into binary spikes emitted to the next layer. Spiking neurons be represented as recurrent neurons with binarized activations and diagonal recurrent weight matrix such that the hidden state of a neuron is isolated from all other neurons (see Ref. [5] for derivation). We adopt the standard Leaky Integrate-and-Fe (LIF)
model, whose dynamics are described by the following equations:
Ult] H it + (
(1)
S[r] = θ (U ueh) [ V.eset Sit + (BU(T) (1 S(()).
(2)
where X is the input the neuron time-step and is typically generated by convolutional or dense operators. Uit] denotes the membrane potential of the neuron, and integrates X [t] with the temporal input component H[t - 1]. O(-) is the Heaviside step function, which is for x 2 and
otherwise. If Utt] exceeds the firing threshold Men. the spiking neuron emits a spike S(t] = 1 as its activation, and the temporal output H[)) is reset to Vreset- Otherwise, no spike is emitted (S[1) 0) and ((4 decays to H with decay factor B < 1. For brevity, we refer to Eq. as S.( where the input is a tensor membrane potential values fed into multiple spiking neurons, and the output S is an deeticclly-sshppee tensor of spikes.
3

---

Repeat timestop
Encoder
Decoder
A
Lane Uivider
Pediotrran en
Align
Vehiciss
Cemera Date
Input
Spiking
Depth
Output
Fused
BLV
Input
Spiking MS Raethal
Drivetre Aron
Estimation
Figure 3: The perception module. The encoder takes multi-camera input data, passes it through spiking ResNet with inverted bottleneck to generate feature representations. each of which has its own depth estimation. These are fused and passed to the decoder, which generates predictions for lane dividers, pedestrians, vehicles and drivable areas.
3.2 Perception: Distinet Temporal Strategies for Encoder and Decoder
Fig. 3 iillustrates the overall architecture the perception module. The perception stage constructs spatiotemporal BEV representation from multi-view camera inputs over t time-steps through spatial and temporal fusion of features extracted from the cameras. It consists of an encoder, which processes each camera input to generate features and depth estimations, and a decoder, which generates BEV segmentation and instructs the planning module. A future prediction module depicted between the encoder and decoder in Fig. It is not used in the first stage where the perception module is trained alone, but it included the second stage once the prediction module included.
The temporal dimension processing in the Encoder/Decoder architecture a crucial design consider- ation, both SNNs and autonomous driving data inherently possess temporal structure. There two approaches to handle this:
Sequential Alignment (SA): sequential input data is passed to the SNN step-by-step by aligning the time-nayying dimension of the input data with the model
Sequence Repetition (SR):
sequential input data aligned with the batch dimension for better parallelism during training, and individual frames are repeated T times over the model sequence, so as create virtual timesteps. SR commonly used for pre-training Sequence-based models on static image datasets.
Given these encoding options, test all four combinations of these options applied to both the encoder and decoder of the perception block. Based on our experiments (detailed in Sec. (43.I). the best performing approach is using SR for the encoder and SA for the decoder. The encoder is also pre-trained on ImageNet-I IK which requires the use of repeated images to create virtual timesteps. Further details regarding the pre-traine pre-training of the encoder can be found Appendix [.I] Conversely, the decoder is trained from scratch, which naturally assumes a temporal -miiigg role, making the alignment of sequential data with the model sequence a more effective approach.
The training process involves first training the encoder-decoder, followed by the prediction module. This approach integrates spatial and temporal information for comprehensive BEV representation in autonomous vehicle perception and planning.
Encoder: Spiking Token Mixer with Sequence Repetition The encoder module can be thought of as spiking token mixer (STM). The STM consists of 12-layers of spiking CNN pre-trained on ImageNet-IK [67] to generate vision patch embeddings, which is effectively deeper version of the 'spiking patch embedding' from Ref. 13 [4]. Across these 12 layers, the number of channels in each layer designed to first increase and then decrease so as act as an inverted bottleneck. While SPS layers are usually terminated by self-attention, we replaced this with dense layers instead, which both reduces computational resources and leads to improved performance. doing so, we achieved 72.1 1% ImageNet top-1 classification accuracy with only 12M parameters. In contrast, previous spiking vision transformers that employs self-attention reached 70.2 20.2% with the same number of parameters [1].

---

The encoder extracts feature embeddings and depth estimations while squeezing the image into a smaller latent space. The overall workflow of the encoder can be summarized as follows:
X = (T)
I E IBNxCuxTxIxHHN X (0, 10,
X [0, (x.1xxxx
F = Headreature (X),
F (0,1,
D Headdepth (X);
X 10, 1X DE (0.1) (0.1)CXXLXHXN YE (0, CIC.TT
Y F D.
The STM encoder described above is used to extract feature embeddings and depth estimates from each camera frame E Nx.x x HxW where N 6 is the number of cameras, C.n 3 refers to the number of input channels (RGB), and H x W refers to the video resolution. Note that the use of sequence repetition means that is the number of times the same frame repeated over the sequence, while L is the number of frames in a continuous camera recording. As such, the dimensions N X are stacked so as to speed feature embeddings F {0. NJNxLxXxxx+ H X and depth estimates D E {0, (0,11.xxxxCa xfW where Cy and C. represent the number of channels for feature embeddings and depth estimates, respectively. These outputs are then fused to form a single tensor Y E {0, 1 ( effectively combining the Cj and C2 dimensions into single dimension. The fused tensor Y is subsequently passed to the decoder for further processing. For more comprehensive understanding of the encoder architecture and its internal workings, please refer to Appendix A
processing. The encoder generates two outputs:
Decoder: Sequential Alignment with Streaming Feature Maps The recurrent decoder aligns feature maps sequentially, introducing a new instance of data at each time-step, contrasting with the encoder's repeated inputs. Using SA rather than SR for the decoder improves performance for two reason: the decoder does not need to be pre-trained on static, repeated data, and 2) the decoder acts as temporal mixer. In this architecture, time- mixing is achieved using LIF neurons and allows them to take on the role of self-attention without the same computational burden. The LIF neurons are composed as shared backbone as a set of layers used to extract features from data before being passed to various specialized heads, each dedicated to a specific task. The high-level dataflow summarized as follows:
X SharedBackbone(TD) I RC X E (0, Cm.1)xmaXTXL
Y4 = (x),
YA R.CRRIXXXHH & E (seg, ped, map, inst)
where In is the input tensor to the decoder with dimensions (Cin. L, H. W), X is the output of the shared backbone with dimensions (Cmed, T, L, H, W),Y; is the output of the k-th head with dimensions (Cout, L, H, W). and k indexes into the heads of the different tasks: vehicle segmentation (seg), pedestrian (ped), HD map (map), and future instance (inst). The shared backbone is implemented using the first three layers of AS-ResNet18 (68 followed by three upsampling layers with factor of and skip connections. More details about MS-ResNet can be found in Appendix BB The resulting features have 64 channels and are then passed to different heads according to the task requirements. Each head consists of spiking convolutional layer.
3.3 Prediction: Fusing Parallel Spike Streams
Predicting future agent behavior is crucial for an autonomous vehicle to be reactive and make informed decisions in real-time. In our approach, we accumulate historical BEV features and predict the next few timesteps using purely LIF neurons. However, the stochastic nature of interactions among agents, traffic elements and road conditions, makes it challenging to accurately predict future trajectories. To address this, we model future uncertainty with a conditional Gaussian distribution.
To accomplish this, two layers of LIF neurons are used. The first parallel layer takes the present and prior output BEV feature maps from the encoder of the perception model (1) as inputs. The first BEV 21 is also used as the initial membrane potential for this LIF layer. The second parallel layer accounts for the uncertainty distribution of future BEV predictions. The uncertainty distribution is generated by passing the present feature v; through 4 spiking MS- MS-ResNet blocks, average pooling, and another 2D spiking convolution with a kernel size of (1, 1) to transform the channel depth to double the output of the first parallel layer. Another averaging pooling operator compresses the
5

---

feature map down to vector. The vector is split in two sub-vectors representing mean μ E R and variance a IR and these values populate a diagonal Gaussian distribution of the latent feature map. Using the j and standard deviation Ø, we can construct the Gaussian distribution at timestep I, denoted as ?/-- This distribution, represented by the parameters jr and 0, can then be concatenated with the input at the current timestep Zt--
Simultaneously, all prior spiking outputs are concatenated the present input 2t, denoted below zo.t- The predicted BEV feature for the next timestep is calculated below:
Lo+1 LIF[BN((Com2(((cnceeenttti ni:)) + (I((BN(Con22(((o)))))
(4)
where 6 represents the predicted BEV features for the next timestep, LIF(-) denotes the LIF neuron layer, concatenate(.) represents the concatenation operation, and @ denotes element- wise addition of the outputs from the two LIF layers, and the inner Conv2d(-) and (N(() lay- ers are used to ensure consistency in the output dimensions of the first and second LIF layers. The mixture prediction serves the basis for the subsequent prediction steps. By recur- sively applying this dual- puthhay prediction method, we obtain the predicted future states (d.41....,f... The overall datapath is illus- trated in Fig. 4 Following the dual-pathway way prediction, all the features are fed into a Spiking ResNet using SA for additional temporal mixing. The historical features (II,...,Iy) xx) and the pre- dicted future features (#1+1..... L(+n) are then passed to the perception decoder, which consists of multiple output heads to generate various in- terpretable intermediate representations.
Pe4s
Pr-z
Peau
X
LIF
Neuron b
56+1
S243
Rusion Hay
Rusion
Faeton
Neuron
S++1
d
LIF
LIF
LIF
LIF
3.4 Planning: Temporal Spiking Temporal for Trajectory Refinement
X1
The primary objective of the SAD system is to plan trajectory that is both safe and comfort- able, aiming towards a designated target. This is accomplished through a planning pipeline that integrates predicted occupancy grids o, map rep- resentations and temporal dynamics of envi- ronmental elements like pedestrians.
Figure 4: Dual pathway modeling for prediction. Neuron a captures future multi-modality by incor- porating uncertainty distribution. Neuron b com- pensates for information gaps using past variations. Inputs ι and 2, from both pathways are used for the next prediction step.
Motion Planning, Initially. our motion planner generates diverse set of potential trajectories using the bicycle model [69]. Among these, the trajectory that minimizes a predefined cost function selected. This function integrates various factors including learned occupancy probabilities (from segmentation maps generated in the Prediction section) and compliance with traffic regulations to ensure the selected trajectory optimizes for safety and smoothness.
Cost Function. The cost function f employed is a multi-component function, where:
[(T, o, m; w) = fo(T, o, m. w.) + [o; w.) + (,,
(5)
where w = (w,, Wa, wy) represents the learnable parameters associated with each cost component, and 7 denotes set of trajectory candidates. Specifically, Jo evaluates trajectory compliance with static and dynamic obstacles, fo is derived from the prediction decoder assessing future states, and fr addresses metries of ride comfort and progress towards the goal. The aim is select the optimal set of trajectories r* that minimize this cost function. This cost function is adapted from Hu et al. [70] and is detailed in Appendix C
Additionally, trajectories are filtered based on high-level commands (e.g., go forward, turn left, turn right) which tailor the trajectory selection to the immediate navigational intent.
Optimization with Spiking Gated Recurrent Unit. Following the initial selection, the "best" trajectory r" undergoes further refinement using a Spiking Gated Recurrent Unit (SGRU), as inspired by [7I]. In this optimization phase, the hidden state h, of the SGRU incorporates features derived from the front camera's 's encoder. The input state x, is formulated by concatenating the vehicle's

---

Table I: Perception results. We We report the BEV segmentation IoU (%) of intermediate representations and their mean value.
Drivable Area
Method
Spike
Lane
Vehicle
Podestrian
Avg.
VED
72
RAL
X
60.82
16.74
23.28
11.93
28.19
VPN
73
65.97
17.05
28.17
10.26
30.36
PON
74
CVPR
X
63.05
17.19
27.91
13.93
30.52
Sift-Splat
R5
72.23
19.98
31.22
15.02
34.61
IVMP
76
ICRA
X
74.70
20.94
34.03
17.38
36.76
FIERY
LCCV
71.97
33.58
38.00
17.15
40.18
ST-P3
70
ECCV
X
75.97
33.85
38.00
17.15
42,69
SAD (Ours)
64.74
27.78
34.82
15.12
35.62
current position, the corresponding position from the selected trajectory x", and the designated target point. The SGRU model processes inputs as follows:
Tt = @ + bir + Warhu- + bur)
(6)
2t biz + Whzhe-1 + bnz)
(7)
n4 = D(Wan + bn + @ (Whnhe-1 + bjon)
(8)
he (1 2)) @ n4
h4-1
where fu denotes the hidden state time t, I, represents the input, and Tr, Rt, are the reset, update, and new candidate state gates respectively. The Heaviside function ensures sparse and binarized operations in the state of the SGRU, thus preserving the advantages of SNNs.
This optimization step enhances trajectory reliability by mitigating uncertainties inherent in perceptual and predictive analyses, and by integrating dynamic traffic light information directly into the trajectory planning process.
After the optimization, the model can be trained end-to-end, from perception to planning. Details about the stage-wise training process and the end-to-end learning methodology can be found in Appendices D.2 and D.3 respectively.
4 Experiments
We evaluate the proposed model using the nuScenes dataset [78] with 20 epochs with 4 x NVIDIA A100 80GB. done in ST-P3 [70]. For experiments, we consider 1.0s of historical context and predict 2.0s into the future, which corresponds to processing 3 past frames and predicting future frames. Additional experimental details are elaborated in the Appendix B
4.1 Experimental Results on nuScenes
Perception. Our evaluation focuses on the model's ability to interpret map representations and perform semantic segmentation. For map representation, we specifically assess the identification of drivable areas and lanes, which are critical for safe navigation they dictate where the Self- Driving Vehicle (SDV) can travel and help maintain the vehicle's position within the lanes. Semantic segmentation tests the model's ability to recognize dynamic objects, such as vehicles and pedestrians, which are pivotal urban driving scenarios.
We employ the Intersection-over- Union (IoU) metric to quantify the accuracy of our BEV segmen- tation tasks. The results, summarized in Tab. [ show that our SAD method, which is fully implemented with spiking neural networks (SNNs), competes favorably against state-of-the-an non- spiking artificial neural networks (ANNs). Notably, our model achieves a superior mean IoU on the nuScenes dataset compared to existing leading methods such VED 72 VPN 73). PON 774, and Lift-Splat (75). Specifically, our SAD method outperforms the VED loU. This enhancement is significant, considering that our network utilizes spiking neurons across all layers, which contributes to greater computational efficiency. Despite the inherent challenges using SNNs, such the binary nature of spikes and potential information loss compared to traditional
72 model by 7.43% mean
7

---

Table Prediction results. We report semantic segmentation IoU (%) and instance segmentation metrics from the video prediction area. The static method assumes all obstacles static in the prediction horizon.
Future Semantic Seg.
Future Instance Seg.
Method
Spike
IoU t
PQ f
SQ
RQ
Static FIERY (7
X
32.20
27.64
70.05
39.08
X
37.00
30.20
70.20
42.90
ST-P3
70 BOCV
X
38.63
31.72
70.15
45.22
SAD (Ours)
V
32.74
20.00
68.74
29.39
Table 3: Planning results.
L2 (m)
Collision (%) |
Method
Spike
Energy (mJ)
1s
2s
38
1s
2s
3s
NMP
48 CVPR
X
0.61
1.44
3.18
0.66
0.90
2.34
Freespace
CVPR
X
0.56
1.27
3.08
0.65
0.86
1.64
344.11
ST-P3
70
BCCV
X
1.33
2.11
2.90
0.23
0.62
1.27
3520.40
SAD (Ours)
/
1.53
2.35
3.21
0.62
1.26
2.38
46.92
ANNs, results demonstrate that SAD is capable of delivering competitive perception accuracy in autonomous driving scenarios.
Prediction. We assess the predictive capabilities of our model using multiple metrics tailored for video prediction, specifically IoU, existing Panoptic Quality (PQ), Recognition Quality (RQ), and Segmentation Quality (SQ) for evaluating our prediction quality. The definition of these metrics can be found Appendix E.2 The results, presented in Tab. demonstrate that while our model does not employ additional temporal module, the inherent temporal dynamics of spiking neurons facilitate effective information processing. However, our SAD model still shows a gap in performance when compared with state-of-the of-t----t----tttt ANN methods.
Planning. In the planning domain, we evaluate our model using two primary metrics: 1.2 error and collision rate. To ensure fairness, the planning horizon is adjusted to 3.0s. The 1.2 error measures the deviation between the SDV's planned trajectory and the human driver's actual trajectory, providing a quantitative measure of planning accuracy. The collision rate assesses the model's ability safely navigate the driving environment without incidents. Results, detailed in Tab. 3 reveal that our SAD method achieves an L2 error and collision rate comparable to those of the state-of-the-art ANN-based methods, underscoring the safety and reliability of our planning approach.
Energy Efficiency. Neuromorphic hardware is able to take advantage of small activation bit- bit-widths and dynamical sparsity {8] and such, SNNs able to significantly reduce energy consumption during inference provided there are sufficiently sparse activation patterns amongst spiking neurons. As detailed in Tab. 3] we present an estimation of the energy usage of each SOTA model based on dynamical sparsity (detailed calculation methods described in Appendix F). Owing to the utilization of spiking neurons, our model achieves substantial energy reductions: 7.33 x less than the Freespace model [51] and 75.03 x lower compared to the ST-P3 model (70). This exceptional energy efficiency makes our model highly suitable for real-world applications.
Table 4: Ablation Study different modules for the encoder and the decoder on Planning tasks.
Table 5: Ablation study on different timestep align- ment strategies for the encoder and decoder on perception tasks.
Decoder
Prediction
Results
Encoder
Decoder
Results
SQ
SR
SA
SR SA wis Dricable
Lane
Vehicle
Pedestrian
Avg.
67.58
59.28
13.35
0.75
19.77
0.44
V
V
/
61.80
43.84
0.00)
20.92
15.25
0.00
31.78
0.00
4.41
13:46
0142
1.62
31.99
15.25
0.11
68.16
16.57
24.11
25.31
15.24
34,06
8

---

FRONT LEFT
FRONT
FRONT RIGHT
Area
Vitecoo
BACK RAART
BACK
DACK RIGHT GHT
(a): 6-Camera View Input
(b): ANN Output
(d): SNN Output
Figure 5: Qualitative Results of the SAD Model the nuScenes Dataset. (a) displays six camera view inputs utilized by the model. (b) illustrates the planning result of the ANN model, and (c) presents the planning results of our SAD model. The comparison shows that our SAD model can achieve performance comparable to that of the ANN model and successfully generate a safe trajectory.
4.2
Visualization
To evaluate the effectiveness of our SAD model in more interpretable manner, we provide qualitative results using the nuScenes dataset. Fig. | illustrates the outputs of model. The SAD model effectively generates a safe trajectory that enables straight-ahead motion while avoiding collisions with curbsides and the vehicle ahead. Additionally,
conducted a comparative analysis with the ANN model, which demonstrated that our SAD model can achieve comparable performance, ensuring accurate and reliable planning untcomes.More visual results can be found in the Appendix E.3
4.3 Ablation Study
4.3.1
Timestep Strategy
The large space of architecture design decisions came with a large number of ablation studies before determining the best performing model. We conducted ablation study to examine the effects different timestep alignment strategies on the performance of the encoder and decoder in perception tasks. The strategies include 'SR' for the repetition of vision inputs over timesteps, 'SA' where sequential inputs are aligned with the model's inherent temporal dimension, and 'w/o T' where recurrent dynamics in the decoder timesteps. The results are summarized in Tab. 5 The last row is the baseline configuration of our model that serves reference point for these experiments. All ablation experiments run for epochs.
removed, thus disconnecting the hidden states between
Impact of Timestep Repetition in Encoder. (Row 1) When repeating timesteps in the encoder which was pre-trained on the ImageNet-IK dataset for classification tasks, we notice a substantial benefit. This approach leverages the encoder's ability to capture spatial information through repeated images, a technique effective during its pre-training phase. Adopting this strategy in the SAD model enhances perception performance by maintaining consistency with the pre-training approach.
Impact of Timestep Alignment in Decoder. (Row 2) In contrast to the encoder, the decoder, which trained from scratch, benefits from aligning timesteps with the model's inherent temporal dimension ('SA'). This strategy leverages the decoder's capacity to mix temporal information, improving performance over the repeat strategy ('SR'), which fails to show any performance gains.
Single Timestep Input. (Row 3) The purpose of this experiment was study how well spiking neurons perform as temporal mixers. In this setup, all inputs are processed in one timestep without inter-framm frame connections, leading to slight performance decrease compared to our baseline. Therefore, it confirms that spiking neurons inherently possess temporal processing capabilities.
4.3.2 Effectiveness of Different Modules
Tab. outlines the results of an ablation study that assesses the impact of various structural modi- fications to the encoder and decoder on planning tasks. The study differentiates between 'MS' for MS-ResNet structure [68). 'SEW' for SEW-ResNet structure [79]. 'SP' for single pathway model, and 'DP' for dual pathway model. We analyze the planning performance of each configuration. The last row represents the configuration of our final model, which we use as the baseline for comparison.
Decoder Structure Evaluation. (Row 1) This part of the study compares the MS-ResNet and SEW-ResNet structures in their roles decoders, where both of them are mainly used in SNN

---

architectures. The difference between MS-ResNet and SEW-Res can be found in Appendix B Our results indicate that the MS- iS-ResNet structure is more effective for planning tasks, likely due to its enhanced capability to handle the spatial-temporal dynamics required in this context.
Here, explore the performance difference between single and dual pathway prediction models. The dual pathway model integrates information through two distinct processing streams. Our experimental results show that the dual pathway model significantly outperforms the single pathway model in planning tasks.
Pathway Model Comparison. (Row 2)
5 Conclusion
In this work, presented Spiking Autonomous Driving (SAD), the first end-to-end spiking neural network designed for autonomous driving. By integrating perception, prediction, and planning into unified neuromorphic framework, SAD demonstrates competitive performance the nuScenes dataset while exhibiting exceptional energy efficiency compared to state-of-the-ar ANN-based methods. The perception module effectively constructs interpretable bird's eye view representations, the prediction module accurately forecasts future states using novel dual-pathway architecture with spiking neurons, and the planning module generates safe and comfortable trajectories. Crucially, SAD showcases the immense potential of SNNs in complex real-world applications, marking a significant step towards realizing low-power, intelligent systems for safety-critical domains like autonomous vehicles. However, further validation through real-world on-vehicle testing is necessary. Moving forward, we believe this work will inspire further research into neuromorphic computing for sustainable and robust autonomous driving solutions.
Acknowledgements
This work was supported in part by National Science Foundation (NSF) awards CNS-1730158, ACI- 1540112, ACI-1541349, OAC-1826967, OAC-2112167, CNS-2100233 CNS-2120019, the University of California Office of the President, and the University of California San Diego's California Institute for Telecommunications and Information Technology/Qualcomm Institute. Thanks to CENIC for the 100Gbps networks. This project was also supported in in-part by the National Science Foundation RCN-SC 2332166.
References
[1] SAE Committee. Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems, 2014.
[2] Kshitij Bhardwaj. Zishen Wan, Arijit Raychowdhury, and Ryan Goldhahn. Real-time fully unsupervised domain adaptation for lane detection autonomous driving. In 2023 Design,
Automation Test in Europe Conference Exhibition (DATE), pages 1-2, 2023.
[3] Soumya Sudhakar, Vivienne Sze. and Sertac Karaman. Data centers on wheels: Emissions from computing onboard autonomous vehicles. IEEE Miero, 43(1):29,39, 2022.
[4] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-617, 2019.
[5] Jason K Eshraghian, Max Ward, Emre O Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. Proceedings of the IEFE, 2023.
[6] Guoqi Li, Lei Deng, Huajing Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass. Brain inspired computing: A systematic survey and future trends. Authorea Preprints, 2023.
[7] Wolfgang Maass. Networks spiking neurons: the third generation of neural network models. Neural networks, 165991165- 1997.
10

---

[8] Mike Davies, Narayan Srinivasa, Tsung-Han, Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Pradip Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEF Micro, 38(1):82-999 2018.
[9] Mostafa Rahimi Azghadi, Corey Lammie, Jason K Eshraghian, Melika Payvand, Elisa Donati, Bernabe Linares- Barres-,BBrrnoo,, and Giacomo Indiveri. Hardware implementation of deep network accelerators towards healthcare and biomedical applications. IEEE Transactions on Biomedical Circuits and Systems, 14(6): 11111118-11111 1159, 2020.
[10] Charlotte Frenkel, Jean- Didier Legat, and David Bol. Morphic: A 65-nm 738k-syyaasccmmm2 quad-core binary-weight digital neuromorphic processor with stochastic spike-driven online learning. IEEE transactions on biomedical circuits and systems, 13(5):999-10) 1010, 2019.
[11] Fabrizio Ottati, Chang Gao, Qinyu Chen, Giovanni Brignone, Mario R Casu, Jason K Eshraghian, and Luciano Lavagno. To spike or not spike: digital hardware perspec- tive on deep learning acceleration. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2023.
[12] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages October 2021.
[13] Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike- driven transformer. Advances Neural Information Processing Systems, 36, 2024.
[14] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425 2022.
[15] Rui-Jie Zhu, Qihang Zhao, Tianjing Zhang, Haoyu Deng, Yule Duan, Malu Zhang, and Liang- Jian Deng. Teja-snn: Temporal-channel joint attention for spiking neural networks. arXiv preprint arXiv:2206.10111 2022.
[16] Ziqing Wang, Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, and Renjing Xu. Masked spiking transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1761-1771, 2023.
[17] Ziqing Wang, Qidong Zhao, Jinku Cui, Xu Liu, and Dongkuan Xu. Autost: Training- frree neural architecture search for spiking transformers. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3455-3459 IEEE, 2024.
[18] Qiaoyi Su, Yuhong Chou, Yifan Hu, Jianing Li, Shijie Mei, Ziyang Zhang, and Guoqi Li. Deep directly-trained spiking neural networks for object detection. In Proceedings of the IEEE/CVF 2023.
International Conference Computer Vision, pages
[19] Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: yolo: spiking neural network for energy energy-efficient efficientt object detection. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 11270-1122 11277, 2020.
[20] Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, XU Bo, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next- gext-gennratton neuromorphic chips. In The Twelfth International Conference Learning Representations, 2023.
[21] Youngeun Kim, Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering, 2(4):044015, 2022.
[22] Xue-Rui Qiu, Rha Wang, Zheng Luan, Rui-Jie Zhu, Ma-Lu Zhang, and Liang-Jian Deng. Vtsnn: virtual temporal spiking neural network. Frontiers in neuroscience, 17:109 2023.
11

---

[23] Hiromichi Kamata, Yusuke Mukuta, and Tatsuya Harada. Fully spiking variational autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7059-7067,
2022.
[24] Qiugang Zhan, Xiurui Xie, Guisong Liu, and Malu Zhang. Esvae: An efficient spiking variational autoencoder with reparameterizable poisson spiking sampling. arXiv preprint arXiv:2310. 14839, 2023.
[25] Xuerui Qiu, Zheng Luan, Zhaorui Wang, and Rui-Jie Zhu. When spiking neural networks meet temporal attention image decoding and adaptive spiking neuron. 2023.
[26] Jiahang Cao, Ziqing Wang, Hanzhong Guo, Hao Cheng, Qiang Zhang, and Renjing Xu. Spiking denoising diffusion probabilistic models. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 4912-4921 2024.
[27] Rui-Jic Zhu, Qihang Zhao, Guoqi Li, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939 2023.
[28] Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang. Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer trained with two-stage knowl- edge distillation from bert. arXiv preprint arXiv:2308. 3an::2208811122.. 2023.
[29] Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 10998-11006, 2024.
[30] Alexander Henkes, Jason K Eshraghian, and Henning Wessels. Spiking neural networks for nonlinear regression. arXiv preprint arXiv:2210.03515. 2022.
[31] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200-5205, 2021.
[32] Samuel Schmidgall, Jascha Achterberg, Thomas Miconi, Louis Kirsch, Rojin Ziaei, S Hajiseye- drazi, and Jason Eshraghian. Brain-inspired learning in artificial neural networks: a review. arXiv preprint arXiv:2305.11233 2023.
[33] Antonio Vitale, Alpha Renner, Celine Nauer, Davide Scaramuzza, and Yulia Sandamirskaya. Event-driven vision and control for uavs on a neuromorphic chip. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 103-109. IEEE, 2021.
[34] Sebastian Glatz, Julien Martel, Raphaela Kreiser, Ning Qiao, and Yulia Sandamirskaya. Adap- tive motor control and learning in a spiking neural network realised on a mixed-signal neuro- morphic processor. In 2019 International Conference on Robotics and Automation (ICRA), pages 9631-9633 IEEE, 2019.
[35] Rasmus Karnøe Stagsted, Antonio Vitale, Alpha Renner, Leon Bonde Larsen, Anders Lyhne Christensen, and Yulia Sandamirskaya. Event-based pid controller fully realized in neuromor- phic hardware: A one dof study. In 2020 JEEE/RS] international conference on intelligent robots and systems (IROS), pages 10939- 10944. IEEE, 2020.
[36] Rasmus Stagsted, Antonio Vitale, Jonas Binz, Leon Bonde Larsen, Yulia Sandamirskaya, et al. Towards neuromorphic control: A spiking neural network based pid controller for uav. RSS, 2020.
[37] Tim Burgers, Stein Stroobants, and Guido de Croon. Evolving spiking neural networks to mimic pid control for autonomous blimps. arXiv preprint arXiv:2309. 12937, 2023.
[38] Zhenshan Bing, Claus Meschede, Kai Huang, Guang Chen, Florian Rohrbein, Mahmoud Akl, and Alois Knoll. End to end learning of spiking neural network based on r-stdp for a lane keeping vehicle. In 20/8 IEEE international conference on robotics and automation (ICRA), pages 4725 4732. IEEE, 2018
[39] Raz Halaly and Elishai Ezra Tsur. Autonomous driving controllers with neuromorphic spiking neural networks. Frontiers in Neurorobotics, 17:1234962, 2023.
12

---

[40] Jacques Kaiser, J Camilo Vasquez Tieck, Christian Hubschneider, Peter Wolf, Michael Weber, Michael Hoff, Alexander Friedrich, Konrad Wojtasik, Amne Roennau, Ralf Kohlhaas, al. Towards a framework for end-to-end control of simulated vehicle with spiking neural net- works. In 2016 IEEE International Conference on Simulation, Modeling, and Haddrrog Programming for Autonomous Robots (SIMPAR), pages 127-134. IEEE, 2016.
[41] Albert Shalumov, Raz Halaly, and Elishai Ezra Tsur. Lidar-driven spiking neural network for collision avoidance in autonomous driving. Bioinspiration & Biomimetics, 16(6):066016, 2021.
[42] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers, arXiv preprint (Xiv:2336.16927 2023.
[43] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference Robot Learning, pages 66-75. PMLR, 2020.
[44] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End to end learning for self-driving cars. arXiv preprint TrXX:::604.07116 2016.
[45] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4693-4700. IEEE, 2018.
[46] Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemystaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, et al. Urhan driving with conditional imitation learning. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 251 2517 IEEE, 2020.
[47] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9329-9338, 2019.
[48] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8660-8669, 2019.
[49] Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 4144430, Springer, 2020.
[50] Hao Wang, Peng Cai, Rui Fan, Yuxiang Sun, and Ming Liu. End-to-end end interactive prediction and planning with optical flow distillation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2199-2200 2021.
[51] Peiyun Hu, Anthony Huang, John Dolan, David Held, and Deva Ramanan. Safe local mo- tion planning with self-supervised freespace forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[52] Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable raycasting for sulf-supervised occupancy forecasting. In European Conference on Computer Vision, pages 238-254. Springer, 2022.
[53] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 20/9 international conference on robotics and automation (ICRA), pages 8248 8284. IEEE, 2019.
[54] Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative rein- forcement learning for vision-based self-driving. In Proceedings of the European conference on computer vision (ECCV), pages 584-599, 2018.
13

---

[55] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model free reinforce- ment learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7153-7162, 2020.
[56] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde. Gri: General reinforced imitation and its application to vision-based autonomous driving. arXiv preprint arXiv:211.008577 2021.
[57] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7077-7087, 2021.
(58) Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-bas based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[59] Hao Shao, Letian Wang. Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced au- tonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 737. PMLR, 2023.
[60] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21983-21994, 2023.
[61] Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to and-to-end driving models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1240-8249, 2023.
[62] Kashyap Chitta, Aditya Prakash, and Andreas Geiger. Neat: Neural attention fields for end- to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15793-15802, 2021.
[63] Dian Chen, Vladlen Koltun, and Philipp Kr"ahenb"uhl Learning to drive from a world on rails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15590-15599, 2021.
[64] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. End-to-end urban driving by imitating reinforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15222-15232, 2021.
[65] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline, Advances in Neural Information Processing Systems, 35:61 19-6132, 2022.
[66] Jimuyang Zhang, Zanming Huang, and Eshed Ohn-Bar. Coaching teachable student. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7805-7815, 2023.
[67] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-20882 255.255. Ieee, 2009.
[68] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[69] Philip Polack, Florent Altché, Brigitte d' Andréa Novel, and Arnaud de La Fortelle. The kinematic bicycle model: consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEE intelligent vehicles symposium (/V), pages 812-818. 818. IEEE, 2017.
[70] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference Computer Vision, pages 533-549. Springer, 2022.
14

---

[71] Ali Lotfi Rezaabad and Sriram Vishwanath. Long short-term memory spiking networks and their applications. In International Conference on Neuromorphic Systems 2020, pages 1-9, 2020.
[72] Chenyang Lu, Marinus Jacobus Gerardus Van De Molengraft, and Gijs Dubbelman. Monocular semantic occupancy grid mapping with convolutional variational encoder- networks. IEEE Robotics and Automation Letters, 4(2):445 (22::445-552, 2019.
[73] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 5(3):4867-4873, 2020.
[74] Thomas Roddick and Roberto Cipolla. Predicting semantic map representations from images using pyramid occupancy networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11138 11147, 2020.
[75] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer, 2020.
[76] Hengli Wang, Peide Cai, Yuxiang Sun, Lujia Wang, and Ming Liu. Learning interpretable end- to-end vision-based motion planning for autonomous driving with optical flow distillation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13731-13737. IEEE, 2021.
[77] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: Future instance prediction in bird's-eye view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15273-15283 15282, 2021.
[78] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11621-11631, 2020.
[79] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks, Advances in Neural Information Processing Systems, 34:21056-21069, 2021.
[80] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01 2020.
[81] Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(1 11:78)2447888 2021.
[82] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE transactions on pattern analysis and machine intelligence, 2023.
[83] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Video panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9859-9868, 2020.
15

---

APPENDIX
A
Encoder
The encoder consists of 12 layers, each containing 2D convolution layer, batch normalization, and spiking neuron. The output the encoder is a feature map F E 10 and a depth estimation D {0, 1) Xxx xxW where Cý is the number of feature channels, Ca is the number of channels, each of which has a depth associated with it, and (H, W) is the spatial size. Formally, given image sequence I:
(10) where Conv2d represents 2D convolutional layer (stride: 1, 3 kernel size), BN is batch normalization, and MP is max-pooling operator. The feature map F and depth estimation D are then averaged over the sequence T' and combined using an outer product to obtain a camera feature
X (N((N(Conv)d(()),
frustum:
Y = = D,Y E f0, (0.1)GxXCuTXLXXHN
(11)
The frustums from all cameras are transformed into global 3D coordinate system centered at the ego-vehicle's inertial center at time Previous BEV feature maps are merged with the current BEV by applying discount factor a to integrate these layers efficiently.
i. b, + Σ a' Āt-
(12)
where ā, is the BEV time which has an initial condition of ã b1 and the discount factor is = 0.5. We then average across the I dimension to eliminate the repeated temporal dimension and obtain the average firing rate. The resulting feature map is then be passed to the decoder.
B Spiking ResNet Architecture
MS-ResNet 668 and SEW-ResNet [79 are two directly trained residual learning methods for SNNs, primarily aiming to overcome the degradation problem in training deep SNNs. In classic CNNs, ResNet achieves "very deep" neural networks by attaching an identity mapping skip connection throughout the entire network. However, directly copying the classic residual structure SNNs leads to the degradation problem, where the training loss increases as the network deepens. The underlying reason is that CNNs generate continuous values through activation functions such as ReLU, while SNNs output discrete spikes.
To address this issue, SEW-ResNet and MS-ResNet establish shortcut connections between spikes or membrane potentials of different layers, respectively. SEW-ResNet builds shortcuts between the output spikes of different layers, thereby obtaining an identity mapping. On the other hand, MS-ResNet constructs shortcuts between the membrane potentials of spiking neurons in different layers. Fig- figllssslsss the differences between these two methods. By building residual learning on membrane potentials, MS-ResNet achieves higher accuracy than SEW-ResNet.
summary, SEW- ResNet and MS-ResNet effectively alleviate the degradation problem faced by deep SNNs by cleverly constructing residual connections suitable for SNN characteristics. These methods significantly improve the performance of SNNs, further narrowing the gap with ANNs of the same scale, and provide effective ideas for building large-scale, high-perffrmance SNNs.
C Detail of Cost Function
In the method section, we briefly introduced the Cost Function f. a multi nulti-commpnent function designed to evaluate the suitability of planned trajectories for self-driving vehicles. We follow the cost function ST-P3 [70), each component of the cost function addresses a specific aspect of the trajectory planning, detailed below:
Safety Cost. This component ensures that the planned trajectories do not result in collisions. The model must account for the dynamic positioning of other vehicles and road elements, avoiding overlap
16

---

SEW-ResNet
MS-ResNee
BN
Conv
Cony
Conv
BN
Spatial Forward Propagation
Forward Propagation
LIF
Figure 6: MS-ResNet and SEW-ResNet.
with the grids they occupy. Furthermore, safe distance must be maintained, especially at high velocities, to prevent any potential accidents.
Cost Volume. Road environments are complex, with numerous unpredictable elements affecting impractical manually enumerate all possible scenarios and associated costs. To address this, we utilize learned cost volume produced by the prediction module head mentioned earlier. To prevent the cost volume from disproportionately influencing the trajectory evaluation, we clip its values maintain balanced decision-making.
trajectory planning.
Comfort and Progress. To ensure that the trajectories not only safe but also comfortable for passengers, this component penalizes excessive lateral acceleration, jerk, and high curvature. Additionally, the efficiency of the trajectory is critical; thus, paths that effectively progress towards the target destination receive positive reinforcement.
These components collectively ensure that the cost function comprehensively assesses both the safety and quality of the trajectories planned by the self-driving vehicle.
D Training
D.1 Pretraining on Spiking Token Mixer (STM)
In STM, the architecture is similar to Spiking Transformers, consisting of a Spiking Patch Embedding (SPS) layer followed by Transformer-liks architecture. However, unlike Spiking Transformers, the SPS layer in STM is not followed by self-attention layer. We pre-train the STM on ImageNet-I KK for 300 epochs, aligning with other SNNs that are pre-trained on ImageNet. The input size to 224 x 224, and the batch size is set to 128 or 256 during the 310 training epochs with cosine-decay learning rate whose initial value is 0.0005. The optimizer used Lamb. The SPS module divides the image into N = 196 patches. Standard data augmentation techniques, such as random augmentation and mixup, are also employed during training. We compare our method with other SNN methods of similar size trained on ImageNet- IK, shown in Tab. 6

---

Table 6: Performance on ImageNet- [67] Note, "Spike", "Para", and "Step" denote "Spike driven". "Spike-driven', "Parameters", and "Timestep".
Param (M)
Methods
Architecture
Spike
Step
Acc.(%)
ANN2SNN
ResNet-34 80)
/
21.8
250
61.5
VGG-16
65.1
SEW-ResNet
[79
x
25.6
4
67.8
SCNN
MS-ResNet
68
/
21.8
69.4
Att-MS-ResNet
82
x
22.1
1
69.2
Spiking-
Spikformer
[4
16.8
70.2
Transformer
Spike-driven Transformer
13
/
16.8
4
72.3
SMLP
STM(Ours)
12.2
4
72.1
D.2 Stage-wise Training of the End-to-end Model
Although model is end-to-end differentiable, directly implementing end-to-end training can lead to unstable loss convergence. To mitigate this issue, we divide our training process into three stages:
Perception Module Training In the first stage, we focus training the perception module while disabling the prediction and planning modules. This means that the prediction module within the network remains inactive, and the decoder does not output instance flow information related to prediction. During this stage, we use a learning rate of le and train the model for 20 epochs.
Stage 1:
Stage 2: Prediction Module Training In the second stage, we enable the prediction module, which predicts the movement of instances on the BEV for the next three steps. We train the model for an additional 10 epochs during this stage.
Stage 3: Planning Module Training Finally, we start training the planning module, using real trajectories to supervise the model's planning results. In both the prediction and planning stages, set the learning rate to 2e - 4 to fine-tune the model from the perception stage. Throughout all stages, employ the Adam optimizer update the model parameters. By incrementally introducing the prediction and planning modules, we ensure that each component is well-trained before integrating them into the end-to-end framework.
D.3 Overall Loss for End-to-End Learning
Our model integrates perception, prediction, and planning into unified framework optimized end-to-end using composite loss function:
-=Cr + aC + 3
(13)
where a and | are learnable weights. These parameters dynamically adjust to scale the contributions of each task based on the gradients from the respective task losses.
Perception Loss. This component is multi-faceted, covering segmentation for current and previous frames, mapping, and auxiliary depth estimation. For semantic segmentation, a top-k cross-entropy loss employed effectively handle the large amount of background, non-salient content in the BEV images, following the approach in [77]. Instance segmentation utilizes an la loss for centerness supervision and an 4 loss for offsets and flows. Lane and drivable area predictions are evaluated using a cross-entropy loss.
Our prediction module extends the perception task by forecasting future semantic and instance segmentation. adopts the same top-k cross-entropy loss used in perception but applies an exponential discount to future timestamps account for increasing uncertainty in predictions.
Prediction Loss.
Planning Loss. The planning module begins by selecting the initial best trajectory 7" from a set of sampled trajectories as defined in Eq. This trajectory is then refined using SGRU-based network
18

---

to produce the final trajectory output T The planning loss comprises two parts: max margin loss that differentiates between the expert behavior Th (treated as positive example) and other sampled trajectories (treated as negatives), and an | distance loss that minimizes the deviation between the refined trajectory and the expert trajectory.
Further details on these loss components and the training are provided in the Appendix D.2
E Experiments
E.1
Datasets
The nuScenes dataset 778 is comprehensive, publicly available dataset tailored for autonomous driving research. It encompasses 1,000 manually selected driving scenes from Boston and Singa- pore-cities renowned for their dense traffic and complex driving environments. Each scene, lasting 20 seconds, showcases variety of driving maneuvers, traffic conditions, and unforeseen events, reflecting the intricate dynamics of urban driving. The dataset aims to foster the development of advanced methods that ensure safety in densely populated urban settings, featuring dozens of objects per scene. It includes around 1.4 million camera images, 390,000 LIDAR sweeps, 1,4 million RADAR sweeps, and 1.4 million object-bounding boxes across 40,000 keyframes.
Data Preprocessing. In preparation for modeling, preprocess the images by cropping and resizing the original dimensions from 3x9000 1600 | Additionally, we incorporate temporal data by including the past three frames for each sequence, denoted 1 12x224x4880 where t E {1, 2, 3} represents the frame indices, and n E (1..... 6} indexes the cameras.
E.2 Evaluation Metrics
In this section, introduce the metrics used to assess the predictive capabilities of our model for video prediction. Specifically, we utilize Intersection over Union (loU), Panoptic Quality (PQ), Recognition Quality (RQ), and Segmentation Quality (SQ) defined by Kim al. [8]]
Intersection over Union (IoU) The loU metric evaluates the overlap between the predicted and ground truth segmentation masks. It is calculated
Prediction n Ground Truth IoU
(14)
Prediction U Ground Truth
where | denotes the number of pixels in the respective set.
Panoptic Quality (PQ) Panoptic Quality (PQ) is comprehensive metric that combines both segmentation and recognition quality. It accounts for both true positive matches and penalizes false positives and false negatives. PQ is defined as:
Eipgerr IOU (p,g)
PQ [TP 1 FPI + FFN
where TP represents the set of true positive matches between predicted segments and ground truth segments g. FP denotes false positives, and FN denotes false negatives.
Recognition Quality (RQ) Recognition Quality (RQ) measures the accuracy of object recognition and classification. is given by:
TP
RQ TP + 2
FP +
FNN
Segmentation Quality (SQ) Segmentation Quality (SQ) evaluates the quality of the predicted segments' spatial accuracy. It is defined as:
SQ = Σ(p9)TP loU(p.g)
TPP
19

---

BACK RIGHT 1H
FRO
LEFT
FRONT
FRONT RIGHE
Drtvble
BACK RIGHT
BACK
BACK RIGHT
Peceurbons
Parvad
Trojettry
FRNN LEPT
FRONT
RIGHT
BACK RIGHT
BACK
BACK RIGHT
Inpur
Output
(0): Output
Figure 7: More Qualitative Results of the SAD Model the nuScenes Dataset. (a) displays six
camera view inputs utilized by the model. (b) illustrates the planning result of the ANN model, and (c) presents the planning results of our SAD model. The comparison shows that SAD model can achieve performance comparable to that of the ANN model and successfully generate safe trajectory.
E.3 More Visualizations
present three additional visual examples in Fig. 7 further showcase the performance of our SAD model on the nuScenes dataset. These examples demonstrate the model's ability to generate safe and collision-free trajectories in various driving scenarios, highlighting its effectiveness in autonomous driving applications.
F Theoretical Energy Consumption Calculation
In the experimental section of our paper, we utilize energy consumption as a metric to evaluate the efficiency of various models. This appendix outlines the methodology employed compute the theoretical energy consumption of our SNN architecture. The calculation process involves two main steps: first, determining the synaptic operations (SOPs) for each architectural component, and second, estimating the cumulative energy consumption based on these operations.
The synaptic operations within each block of the SNN are calculated using the following equation:
SOPs(I)
x T FLOPs(!)
(15)
where denotes the ordinal number of the block within the SNN, fr represents the firing rate of the input spike train to the block, 7' indicates the time step of the neuron, and FLOPs(() refers to the floating-point operations within the block, primarily consisting of -nd-a-uuuuuaaalttte (MAC) operations.
To compute the SNN's theoretical energy consumption, we consider both MAC and spike-based accumulation (AC) operations, utilizing 45 nm technology. The energy costs are EM EM 4.6 for MAC operations and EAc 0.9 for AC operations. Based the methodologies outlined in 82
we calculate the energy consumption of the SNN as follows:
ESNN -MMA FLOP] SNN CoNY
N
(16)
EAC x
Σ
SOp CoNc
SOP SNNPO
カ=2
m=1
20

---

Here, N and M denote the number of convolutional (Conv) and fully connected (FC) layers, re- spectively. The terms FLOP FNN and SOP SNNCamv SOP SNNFC represent the FLOPs for the first Conv layer and SOPs for the subsequent th Conv and m FC layers.
For the ANN model, the formula for computing the theoretical energy consumption is more straight- forward, all neurons operate in floating point:
EANN = EMAC x FLOPANY
(17)
Here, FLOPANO ANN represents the FLOPs of the ANN model.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction reflect the paper's contributions and scope
clearly.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitation of this work in the Conclusion section.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results that should be proved.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All training details have been revealed in Sec.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have attached the anonymized code for reproducing our product in supple- mentary material.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details have been revealed in Sec.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not have enough computational resources to run it multiple times to generate error bars.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The paper provides detailed information the compute resources used.
22

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/ss//aacppp....popppleeeea
Question:
Answer: [Yes]
Justification: This work follows the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Question: Are creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits creators and mentions the license and terms of for existing assets.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
23


Reviews:
[{"review_id": "qOqND9ygWY", "reviewer": "Reviewer_x6bz", "summary": "This paper introduces a unified Spiking Autonomous Driving (SAD) system based on Spiking Neural Networks (SNNs). The system is trained end-to-end and comprises three main modules: a perception module that processes inputs from multi-view cameras and constructs spatiotemporal bird's-eye views; a prediction module that forecasts future states using a novel dual-pathway approach; and a planning module that generates safe trajectories based on the predictions. The authors evaluated SAD on the nuScenes dataset, where it demonstrated competitive performance in perception, prediction, and planning tasks. By leveraging its event-driven and energy-efficient nature, SAD effectively addresses the energy challenges faced by autonomous driving systems.", "questions": "Referencing the Weakness section, I recognize that some evaluation methods are commonly used within the SNN community. However, due to differences in task formulations, explanations and clarifications are necessary.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper constructs the first end-to-end spiking neural network designed specifically for autonomous driving. By integrating perception, prediction, and planning into a single network, it enhances efficiency and reduces the complexity associated with managing these components separately.\n\n2. The experimental results are comprehensive and persuasive, with a thorough comparison with other works, demonstrating the significant potential of SNNs in complex real-world applications.", "weaknesses": "1. Energy Consumption Analysis: The main motivation for implementing autonomous driving tasks with SNNs in this paper is to enhance energy efficiency, with a corresponding energy efficiency analysis provided in Appendix F and Table 3. Nonetheless, I have reservations about the energy consumption calculations presented for the following reasons:\n   - Since both ANNs and SNNs process the same input data (with SNNs employing direct encoding at the first layer), the difference in energy efficiency can only stem from differences in memory access or the energy consumption of MAC/AC operations. This paper only offers a quantitative comparison of energy consumption during computation (L748-L768).\n   - It is well-known that energy consumption is primarily determined by memory access rather than FLOPS or IPS [1r], yet this aspect is not discussed in the paper, nor is it considered in the energy consumption calculations. This should at least be acknowledged, as it is a common shortcoming in publications claiming reduced energy consumption. Could the authors comment on how memory access impacts the energy consumption of SNNs?\n   - ANN networks typically exhibit similarly sparse ReLU activations. Previous research has shown that even on general-purpose CPU hardware, the sparsity of ReLU can be utilized to accelerate (and reduce energy consumption of) inference [2r]. How would the comparison results between SNNs and ANNs change if ANN sparsity were considered?\n\n[1r]: An Analytical Estimation of Spiking Neural Networks Energy Efficiency. ICONIP 2022\n\n[2r]: Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference. ICML 2020\n\n\n2. Inference Speed: In comparisons with six ANN methods, SNN performance only surpassed half of the existing methods. Assuming SNNs have lower energy consumption, we would consider using SNNs for autonomous driving tasks. What about their inference speed in practical deployments? Autonomous driving tasks rely on high dynamic resolution and timely responsiveness, such as detecting and avoiding obstacles on highways.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful comments and questions raised by the reviewers. Your feedback has provided us with valuable insights that will help improve our manuscript. We are grateful for the opportunity to address these points and clarify certain aspects of our work. In the following responses, we aim to address each of your concerns comprehensively while highlighting the strengths and potential impact of our research on SNNs for autonomous driving tasks.\n\n> Since both ANNs and SNNs process the same input data (with SNNs employing direct encoding at the first layer), the difference in energy efficiency can only stem from differences in memory access or the energy consumption of MAC/AC operations. This paper only offers a quantitative comparison of energy consumption during computation (L748-L768). It is well-known that energy consumption is primarily determined by memory access rather than FLOPS or IPS [1r], yet this aspect is not discussed in the paper, nor is it considered in the energy consumption calculations. This should at least be acknowledged, as it is a common shortcoming in publications claiming reduced energy consumption. Could the authors comment on how memory access impacts the energy consumption of SNNs?\n\nResponse: Thank you for your insightful question. We acknowledge that this issue indeed exists. The field of SNNs has not specifically focused on memory access, despite the fact that data movement is indeed the primary bottleneck in modern deep learning. We will certainly address this point in our revised manuscript.\n\nSNNs utilize binary activations, which naturally lead to activation sparsity. This sparsity can be leveraged to reduce memory access frequency and enable more efficient inference, potentially allowing for partial offloading to CPU DRAM [R1]. Additionally, the lower precision of activations in SNNs contributes to reduced memory bandwidth requirements.\n\nFurthermore, neuromorphic hardware inspired by in-memory computing concepts, such as RRAM-based architectures, shows promise in eliminating traditional memory access costs altogether when paired with SNNs [R2]. These factors combined contribute significantly to the overall energy efficiency of SNN implementations, beyond what is captured by comparing only the MAC/AC operations.\n\nCollectively, these characteristics of SNNs and their potential hardware implementations suggest that the energy efficiency gains may be even more substantial when considering memory access, though we agree that this aspect warrants more thorough investigation and quantification in future research.\n\n> ANN networks typically exhibit similarly sparse ReLU activations. Previous research has shown that even on general-purpose CPU hardware, the sparsity of ReLU can be utilized to accelerate (and reduce energy consumption of) inference [2r]. How would the comparison results between SNNs and ANNs change if ANN sparsity were considered?\n\nResponse: Thanks for your question. In our examination of ANNs, we focused on ST-P3, the state-of-the-art ANN for these tasks. We found that the main challenge in utilizing sparsity across the entire network stems from the varied use of activation functions. Specifically, only the encoder and decoder head employ ReLU, which exhibits sparsity. Other crucial components such as parts of the decoder, temporal module, and planning module use different activation functions like tanh (for GRU) and GeLU (for partial decoder blocks). These functions are not inherently sparse, resulting in a model that is more dense than sparse overall. \n\nThis heterogeneity in activation functions makes it difficult to fully leverage the potential benefits of sparsity throughout the ANN. In contrast, our SNN model utilizes sparse and binary activations across all layers, enabling us to fully exploit both the sparsity and binary nature of the network. This comprehensive approach to sparsity in SNNs provides a more consistent basis for comparison and potential performance advantages.\n\n> Inference Speed: In comparisons with six ANN methods, SNN performance only surpassed half of the existing methods. Assuming SNNs have lower energy consumption, we would consider using SNNs for autonomous driving tasks. What about their inference speed in practical deployments? Autonomous driving tasks rely on high dynamic resolution and timely responsiveness, such as detecting and avoiding obstacles on highways.\n\nResponse: Thank you for your question. As stated in our paper, our primary goal is to demonstrate the potential of SNNs to handle the complex requirements of low-power autonomous driving. Neuromorphic chips can achieve extremely low latency (about 200 times less) compared to conventional chips, while also maintaining low power consumption. For example, Speck [R3] can achieve <0.1 ms latency when implementing SNN on-chip, compared to 24.7 ms latency when not implemented on-chip. Intel's Loihi chip also shows superior latency when executing similar tasks [R4]. We plan to implement our algorithm on these chips to fully utilize their capabilities.\n\nIn conclusion, we thank the reviewers for their insightful comments, which have allowed us to provide a more comprehensive view of our work. \n\n---\n[R1]:Song, Yixin, et al. \"Powerinfer: Fast large language model serving with a consumer-grade gpu.\" arXiv preprint arXiv:2312.12456 (2023).\n\n[R2]:El Arrassi, Asmae, et al. \"Energy-efficient SNN implementation using RRAM-based computation in-memory (CIM).\" 2022 IFIP/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC). IEEE, 2022.\n\n[R3]: Yao M, Richter O, Zhao G, et al. Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nature Communications, 2024.\n\n[R4]: Viale A, Marchisio A, Martina M, et al. Carsnn: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor, 2021 IJCNN.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTQjpXGRed", "reviewer": "Reviewer_GVJq", "summary": "This paper presents Spiking Neural Network to reduce the energy consumption by normal neural network in autonomous driving. The network includes end-to-end architecture including perception, prediction and planning. It evaluates the model using nuScenes dataset and achieves comparable performance in three modules yet drawing much lower energy consumption.", "questions": "- Do the author know if any self-driving company use spiking neural network in their real operation? This can justify the impact of this work. \n- Energy is saved but how about memory, inference time?", "limitations": "I think the work addresses the proposed question about reducing energy consumption. Besides, I do not have expert knowledge in spiking neural networks, thus I shall refrain from talking about limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The author proposes a good architecture and describe in detail all of the modules: perception, prediction and planning. I enjoy the reading and found it very good to list in details of architecture, especially the results, visualization and ablation study. In general, I think this stands a good paper. The reason I rate it at 5 is due to not familiar with the work in energy consumption using spike neural networks.", "weaknesses": "I do not write this due to not being familiar with the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions regarding the practical implementation of SNN in self-driving systems. While it's true that SNNs are not yet widely deployed in commercial self-driving operations, our research aims to pave the way for their future adoption by demonstrating their viability and potential advantages in this domain.\n\n> Do the author know if any self-driving company use spiking neural network in their real operation? This can justify the impact of this work.\n\nResponse: While there are several companies exploring the use of SNNs in autonomous driving, the exact information on their progress is currently not in the public domain. To the best of our knowledge, this is the first work to use an SNN to perform complex motion planning given multi-camera input.\n\nMercedes Benz has begun [implementing neuromorphic chips](https://www.eetimes.com/mercedes-applies-neuromorphic-computing-in-ev-concept-car/) in its concept car, EQXX, specifically for voice detection. Similarly, [BMW has announced plans](https://www.greaterzuricharea.com/en/news/synsense-and-bmw-exploring-intelligent-cockpits) to use neuromorphic chips in their vehicles. However, we've observed that in most cases, these chips are being used to solve relatively simple tasks. This aligns with the limitations of SNNs that we discussed in our paper, where they are often perceived as unable to solve complex problems.\n\nSecondly, Intel's simultaneous development of both the Loihi 2 neuromorphic chip and Mobileye's self-driving processors presents a unique opportunity for integrating SNNs into autonomous driving systems. This combination of technologies within the same company creates a potential pathway for implementing SNN-based self-driving models on neuromorphic hardware.\n\nLastly, the primary advantage of deploying SNN models on neuromorphic chips like Loihi 2 for self-driving applications would be significantly reduced power consumption. This is crucial for electric and autonomous vehicles, where energy efficiency directly impacts range and overall performance. Therefore, while not yet widely implemented, the potential benefits of SNNs in self-driving technology make it a promising area for future development and application.\n\n> Energy is saved but how about memory, inference time?\n\nResponse: You raise an important point about memory usage and inference time.\n\nIn our current stage of research, we've primarily validated the performance of our SNN-based self-driving model on standard hardware like GPUs. At this point, SNNs don't show significant advantages in terms of memory usage or inference time compared to traditional neural networks on these platforms.\n\nHowever, the real potential of SNNs for self-driving applications is in their deployment on neuromorphic hardware such as Intel's Loihi 2 chip. The asynchronous data processing capabilities of Loihi 2 are expected to greatly enhance efficiency across multiple dimensions, including memory usage and inference time.\n\nIt's worth noting that previous work has already demonstrated the feasibility and benefits of implementing self-driving tasks on neuromorphic hardware. For instance, [R1] successfully applied SNNs to a self-driving classification task on the Loihi chip. While this was a specific subtask of autonomous driving, it provided strong evidence that SNNs can indeed work effectively on neuromorphic hardware for automotive applications.\n\nIn future stages of our research, we plan to implement our model on Loihi 2 or a similar neuromorphic platform. This transition from standard GPUs to specialized neuromorphic hardware is expected to unlock the full potential of SNNs for self-driving applications, addressing current limitations in memory usage and inference time while maintaining the energy efficiency advantages.\n\nWe appreciate the reviewer's thoughtful questions and comments. Your feedback has allowed us to clarify important aspects of our work and its potential impact on the field of autonomous driving.\n\n---\n\n[R1]: Viale A, Marchisio A, Martina M, et al. Carsnn: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor, 2021 IJCNN.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TiLs0W8OSw", "reviewer": "Reviewer_EVdw", "summary": "This paper presents an end-to-end SNN model for the autonomous driving to address the energy challenges. This model consists of three main modules: perception, prediction, and planning. The model is evaluated in the nuScenes dataset.", "questions": "see weekness.", "limitations": "None", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduce the first SNN designed for end-to-end autonomous driving, integrating perception, prediction, and planning into a single model.\n\n2. well-written.", "weaknesses": "1. The novelty is limited. The paper just uses the SNN to autonomous driving and does not provide any special designs.\n\n2. Only one dataset is used.\n\n3. The energy conputation does not consider the data moving.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort the reviewer has dedicated to evaluating our work. Your feedback is valuable in helping us improve and clarify our research.\n\n>The novelty is limited. The paper just uses the SNN to autonomous driving and does not provide any special designs.\n\nResponse: We appreciate the reviewer's comments but respectfully disagree with the assessment regarding novelty and the lack of special designs. Our research makes several significant contributions to the field:\nFirstly, unlike tasks such as object detection, semantic segmentation, and classification that have been previously addressed by SNNs, we tackle autonomous driving - a task widely recognized as highly complex in the visual domain. To our knowledge, this application of SNNs to autonomous driving is the first of its kind in the field. If there are existing papers on this topic, we kindly request the reviewer to point them out.\nSecondly, the claim that there are no special designs is inaccurate. Our 3.1 module, \"Distinct Temporal Strategies for Encoder and Decoder,\" combines Sequential Alignment with Sequential Repetition, which is a novel approach in the SNN domain while maintaining a spike-driven architecture. Furthermore, the training process of the Spiking Token Mixer, a core component in our Encoder, is entirely unique. As detailed in Appendix B.1, we avoided using any attention or convolutional components, as well as MLP-Mixer type token mixers. Instead, our approach achieved a performance of 72.1% on ImageNet, surpassing other Spiking Transformers.\nThirdly, the design of the Spiking GRU is an original contribution not previously proposed in other papers. Our ablation study demonstrates how our architecture outperforms alternative designs, some of which struggle to achieve meaningful performance.\nContrary to the reviewer's assertion, we adhere to Occam's razor principle, seeking the most appropriate design rather than pursuing novelty for its own sake. We firmly believe that truly effective designs, not merely novel ones, are what advance the machine learning community.\n\n> Only one dataset is used.\n\nResponse: We'd like to address this concern from several angles. Firstly, comparable datasets for end-to-end autonomous driving are scarce. Several prominent works in this field, such as [R1] and [R2], have also exclusively utilized the nuScenes dataset, highlighting its significance and broad acceptance in the research community. Moreover, it's not entirely accurate to say we only used one dataset. As detailed in Appendix B.1, we also tested the crucial part of our method's performance on ImageNet to verify its robustness and generalizability.\nWe believe that the combination of in-depth analysis on a specialized autonomous driving dataset and performance verification on a general, large-scale dataset provides a comprehensive evaluation of our method's effectiveness and potential for broader applications.\n\n>The energy conputation does not consider the data moving.\n\nResponse: Thank you for your observation. You're correct that our energy computation does not consider data movement. This approach is consistent with standard practices in SNNs power estimation. Typically, these estimations focus on computation using FLOPS [R1][R2][R3].\nIt's important to note that this is a common limitation in the SNN field as a whole. However, the field is evolving, and new SNN hardware based on in-memory computing is emerging[R4]. This development is part of the broader trend in neuromorphic hardware design, which aims to address energy efficiency issues, including those related to data movement.\n\nIn conclusion, we would like to emphasize that our work represents a novel and significant contribution to the field of SNNs and autonomous driving. We have introduced unique architectural designs, applied SNNs to a complex real-world problem, and provided comprehensive evaluations on both specialized and general datasets. While we acknowledge that there is always room for improvement and further exploration, we believe our research pushes the boundaries of what is possible with SNNs in autonomous driving applications. We hope that our clarifications have addressed the reviewer's concerns and demonstrated the value and novelty of our work. We remain open to further discussion and are committed to advancing this important area of research.\n\n--- \n\n[R1]:Zhou, Zhaokun, et al. \"Spikformer: When spiking neural network meets transformer.\" arXiv preprint arXiv:2209.15425 (2022).\n\n[R2]:Yao, Man, et al. \"Spike-driven transformer.\" Advances in neural information processing systems 36 (2024).\n\n[R3]:Zhu, Rui-Jie, et al. \"Spikegpt: Generative pre-trained language model with spiking neural networks.\" TMLR 2024.\n\n[R4]: El Arrassi, Asmae, et al. \"Energy-efficient SNN implementation using RRAM-based computation in-memory (CIM).\" 2022 \nIFIP/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC). IEEE, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QxhXUwRLi5", "reviewer": "Reviewer_j4mM", "summary": "The authors adapted ST-P3 into a version with a binary spiking neuron and then stated that autonomous driving based on a spiking neural network can address the energy challenges. The authors stated that this neuromorphic technology can be a step toward sustainable and safety-critical automotive technology.", "questions": "What is the advantage of the SNN solution from the perspective of a safety-critical application? As this work is oriented towards this specific application, it is better to clearly state the advantages. Do you have specific safety considerations in the design of this model? Do you include any related work on SNN robustness in this paper?\n\n\n\nDual pathways and other architecture designs manifest a problem of mapping on neuromorphic hardware. I suspect that SGRU is as energy efficient as simple LIF. Eqs. 6, 7, 8, and 9 imply heavy matrix computation. Can you provide an explanation for this?\n\n\n\nANN seems more robust than SNN (Figs. 5, 7). In the figures of ANN outputs, are the vehicle markers clearer and more accurate? What do you think SNN will do to lead to this result? Do you have any solutions to this? Can you include experiments on this to address this problem? (This question is important.)", "limitations": "See questions and weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This work has completed the end-to-end SNN-based autonomous driving pipeline, which is a heavy job.\n\nThe work has a clear architecture. This work is easy to follow.\n\nThe idea of using SNN in driving technology is interesting.", "weaknesses": "The evaluation is not as effective as its reference work, ST-P3, which has evaluations of open loop validation and close loop validation.\n\nThe implementation plan on neuromorphic hardware is not clear in this paper, which I think it’s the most important aspect for energy-efficient applications.\n\nNo comparison with recent state-of-the-art ANN models (later than ST-P3), as the necessity of autonomous driving using SNN is an open question. It is better to include more comparison and further discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough examination of our work and their insightful comments. We acknowledge the challenges highlighted in your review and appreciate the opportunity to address them. In the following responses, we aim to clarify our contributions, explain our methodological choices, and discuss the implications of our findings for the field.\n\n\n> What is the advantage of the SNN solution from the perspective of a safety-critical application? As this work is oriented towards this specific application, it is better to clearly state the advantages. Do you have specific safety considerations in the design of this model? Do you include any related work on SNN robustness in this paper?\n\nResponse: Thank you for your question. We'd like to clarify our focus and address your points:\n\nThe primary aim of our work is to demonstrate the potential of SNNs to handle the complex requirements of low-power autonomous driving. Our current research is centered on establishing the feasibility and efficiency of SNN-based models for this application.\n\nTo our knowledge, there is ongoing research into robust SNNs [R1], demonstrating their potential in safety-critical applications. This existing work provides strong evidence for the viability of SNNs in scenarios where safety is paramount, such as autonomous driving.\n\nPrior to this work, the application of SNNs to autonomous vehicle planning had not been achieved, and so integrating robustness, interpretability, and safety-critical features are a logical next step beyond efficiency. \n\nMoreover, we appreciate your suggestion about including related work on SNN robustness. This is a valuable addition that we plan to incorporate in the final version of our paper.\n\n\n> Dual pathways and other architecture designs manifest a problem of mapping on neuromorphic hardware. I suspect that SGRU is as energy efficient as simple LIF. Eqs. 6, 7, 8, and 9 imply heavy matrix computation. Can you provide an explanation for this?\n\nResponse: Thank you for highlighting this point. The dual pathways and other complex architectural designs present challenges when mapping onto neuromorphic hardware. However, it's worth noting that designs similar to ResNet have been extensively explored in the field of Spiking Neural Networks (SNNs) and successfully implemented in hardware. This suggests that dual pathway architectures are quite straightforward to realize in neuromorphic systems. Given that most modern neuromorphic hardware utilize many smaller cores, parallel layers would be executed in separate cores, with their results merged in another core - quite similarly to residual/skip connections.\n\nRegarding the energy efficiency of SGRU compared to simple LIF neurons, your intuition may be correct. The key lies in understanding the nature of the computations in Equations 6, 7, 8, and 9. While these equations might initially appear to involve heavy matrix computations, the actual implementation in SGRU is more efficient due to the binary nature of the signals.\n\nIn SGRU, the input $x_t \\in \\{0,1\\}$ represents spikes. This binary representation simplifies the computations $W_{ir}x_t$, $W_{iz}x_t$, and $W_{in}x_t$. Similarly, since $h_t = (1 - z_t) \\odot n_t + z_t \\odot h_{t-1}$, where $z_t \\in \\{0,1\\}$ and $n_t \\in \\{0,1\\}$, it follows that $h_t$ is also a binary value. Consequently, the terms $W_{hr}h_{t-1}$, $W_{hz}h_{t-1}$, and $W_{hn}h_{t-1}$ become spike-driven operations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L1LNi0seB9", "reviewer": "Reviewer_S2mt", "summary": "This paper introduces Spiking Autonomous Driving (SAD), an end-to-end spiking neural network (SNN) designed for autonomous driving.  SAD integrates perception, prediction, and planning into a unified neuromorphic framework.  It achieves competitive performance on the nuScenes dataset and significantly outperforms traditional artificial neural network (ANN) methods in terms of energy efficiency. The authors anticipate that this work will spur further research into neuromorphic computing for sustainable and robust autonomous driving solutions.", "questions": "1. I wonder if the numbers in your proposed SNNs are spikes or not. $\\mathbb{R}$ stands for real number field.\n2. Can you show the detailed energy results of all steps in Table 3? I think it is incomplete to show the planning step only.\n3. In Line 238, I wonder why you chose GRU to refine the trajectory. Does GRU have unique advantages on this issue?", "limitations": "The authors mentioned limitations in the Conclusion Section, focusing on further validation through real-world on-vehicle testing. However, it would be better for authors to discuss limitations in more detail.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Autonomous driving can be seen as an energy-limited scene, so I think it is meaningful to introduce SNNs to this task. To the best of my knowledge, this paper is among the first to utilize SNNs to AD.\n2. This paper provides a unified framework on how to apply SNNs toAD. The authors employ a dual-pathway architecture and I think this is an innovative design in the use of spiking neurons.\n3. SAD achieves competitive results on the BEV segmentation IoU and semantic segmentation IoU, indicating the practical viability of the proposed method. The results in Table 3 have really surprised me, for its great energy-efficiency.", "weaknesses": "1. Notation typos. It seems that the authors make mistakes in equations in Line 148, 155, 157, and 169. For example, I think $X \\in$ \\{0,1\\} $^{C\\times T\\times L\\times H\\times W}$ instead of $X \\in \\mathbb{R}^{C\\times T\\times L\\times H\\times W}$ because $X$ should be a spike train rather than a floating-point matrix. \n2. I think energy efficiency is the key to this paper. However, the authors talk little about this issue in  Section 4.1. Can you provide me more about how you calculate the energy consumption and your detailed energy experiments of all 3 steps (i.e., perception, prediction, and planning)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful comments on our manuscript. We appreciate the time and effort you've invested in providing valuable feedback. We have carefully considered your points and are pleased to address them below.\n\n>I wonder if the numbers in your proposed SNNs are spikes or not.  stands for real number field.\n\nResponse: \nThank you for your observation. You are correct; the SNNs indeed represent spikes. We should clarify that the domain should be $\\{0,1\\}$ instead of $\\mathbb{R}$, accurately reflecting the spiking nature of the neurons.\n\n>Can you show the detailed energy results of all steps in Table 3? I think it is incomplete to show the planning step only.\n\nResponse:  We apologize for any confusion. To clarify, Table 3 presents the overall energy results, not just a single stage. Here's a detailed breakdown of the energy consumption for each stage:\n\n   | Step       | Energy (mJ) |\n   | ---------- | -------------- |\n   | Perception | 41.09          |\n   | Prediction | 4.53           |\n   | Planning   | 1.30           |\n   | Overall    | 46.92          |\n\n\n>In Line 238, I wonder why you chose GRU to refine the trajectory. Does GRU have unique advantages on this issue?\n\nResponse:We chose GRU for its effectiveness in handling temporal and spatial mixing. While both GRU and LSTM are suitable for this task, GRU offers a slight advantage in terms of computational efficiency. It has a lower token mixer complexity compared to LSTM, as it uses one less hidden state. This makes GRU faster while still providing comparable performance for our specific use case.\n\nWe hope these clarifications and additional details address your concerns and improve the quality of our manuscript. Once again, we sincerely appreciate your valuable feedback, which has helped us enhance the clarity and completeness of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXKZd3FGcx", "reviewer": "Reviewer_XRtX", "summary": "Although SNNs have potential of neuromorphic computing in sustainable and safety-critical autonomous technology, they still lack evidence in complex real-world computer vision applications. In this work, the authors propose a unified end-to-end SNN called SAD that consists of three models to generate safe trajectories, which presents improved energy efficiency. The researchers complete the validation on the nuScenes dataset, which not only verifies the good performance, but also proves the effectiveness to a certain extent.", "questions": "1. In related works, the authors should add a relevant description of the connection between this research and End-to-end Autonomous Driving, as in the previous paragraph “We push...in this paper ”.\n\n2. In Section 4.1, the authors mention that \"The results, as summarized in Tab. 1, show that our SAD method, ..., competes favorably against state-of-the-art, non-spiking artificial neural networks (ANNs)”. But I only see a specific figure of 7.43%. Please add descriptions to support it.\n\n3. This work lacks a discussion of the robustness of the proposed method, I would like to recommend adding experiments on more datasets.\n\n4. In Table 4 and Table 5, the authors do not show complete ablation experiments, e.g., SEW+SP in Table 4, SA+SR in Table 5, etc. Please provide the necessary details.", "limitations": "Some details in the manuscript are not clearly stated and algorithmic performance needs more convincing.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed idea is novel and relevant to the NeurIPS community.\n\n1. SNNs have attracted a lot of attention in recent years due to their better performance and low energy consumption, but the application potential remains to be explored. In this work, the authors investigate an end-to-end SNN-based approach applying in autonomous driving, which shows impressive performance similar to traditional neural network approaches. This increases the impact of the work.\n\n2. The proposed method can help reduce energy consumption and plan a safe path, which shows that SNNs have great potential to be applied in the field of autonomous driving systems.\n\n3. In experiments, the authors compare the proposed method with traditional deep learning works in recent five years, and obtain competitive results.", "weaknesses": "Comparisons with state-of-art works in the last three years are lacking in the results of perception, prediction, and planning in Section 4.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough and constructive feedback on our manuscript. We appreciate the positive comments on the novelty and relevance of our work, as well as the recognition of its potential impact in the field of autonomous driving. We have carefully considered all the points raised and have addressed them in detail in the following sections. We kindly ask the reviewer to find our responses to their specific questions and concerns in the content below.\n\n>In related works, the authors should add a relevant description of the connection between this research and End-to-end Autonomous Driving, as in the previous paragraph “We push...in this paper ”.\n\nResponse: Thank you for pointing this out. We will add a paragraph that connects our research to end-to-end autonomous driving in the final version. A draft passage is provided here: \"While previous work has shown SNNs' effectiveness in autonomous control tasks, our research extends to the more complex challenge of end-to-end autonomous driving. We build upon these foundations, pushing SNNs to handle the challenging, real-time decision-making required for full autonomy in dynamic, real-world environments. This work bridges the gap between simplified control tasks and the holistic approach needed for true end-to-end autonomous driving.\n\n>In Section 4.1, the authors mention that \"The results, as summarized in Tab. 1, show that our SAD method, ..., competes favorably against state-of-the-art, non-spiking artificial neural networks (ANNs)”. But I only see a specific figure of 7.43%. Please add descriptions to support it.\n\nResponse: Thank you for bringing this to our attention. You're right that we should provide more comprehensive support for our statement about competing favorably against state-of-the-art ANNs. We'll revise this section to include more detailed comparisons. A draft update is provided here: \"The results, as summarized in Tab. 1, show that our SAD method, which is fully implemented with spiking neural networks (SNNs), competes favorably against several state-of-the-art, non-spiking artificial neural networks (ANNs). Specifically: Our SAD method achieves a mean IoU of 35.62%, which outperforms VED (28.19%), VPN (30.36%), PON (30.52%), and Lift-Splat (34.61%). While more recent methods like IVMP (36.76%), FIERY (40.18%), and ST-P3 (42.69%) achieve higher mean IoUs, our SNN-based approach comes close to their performance, especially considering the inherent energy-efficiency of using SNNs.\"\n\n>This work lacks a discussion of the robustness of the proposed method, I would like to recommend adding experiments on more datasets.\n\nResponse: We appreciate your suggestion. We agree that testing on additional datasets would strengthen our work. While time constraints prevent us from conducting these experiments for this paper, we acknowledge this limitation and plan to address it in future work. Specifically, we aim to extend our experiments to include the CARLA dataset.\n\n>In Table 4 and Table 5, the authors do not show complete ablation experiments, e.g., SEW+SP in Table 4, SA+SR in Table 5, etc. Please provide the necessary details.\n\nResponse: Thank you for your observation. We appreciate the suggestion for more comprehensive ablation experiments. However, we would like to clarify that our ablation studies are designed to compare individual changes in configuration against our proposed model (shown in the last row of each table). The purpose of these experiments is to demonstrate the impact of each specific component or strategy. Combining multiple changes simultaneously (e.g., SEW+SP in Table 4 or SA+SR in Table 5) would make it difficult to isolate the effect of individual components. Moreover, as our results show, changing one configuration already leads to a decrease in performance. It's reasonable to expect that altering multiple configurations simultaneously would likely result in an even greater performance drop. \n\nWe thank the reviewer once again for their valuable feedback and insightful questions. We believe that addressing these points has helped to clarify and strengthen our work. We have provided additional context for our research, expanded on our results, acknowledged limitations, and explained our experimental design choices. We hope that these responses adequately address the reviewer's concerns and further demonstrate the significance and potential impact of our work in advancing SNN-based approaches for autonomous driving. We are committed to incorporating these improvements in the final version of our paper, should it be accepted. We appreciate the reviewer's time and expertise in evaluating our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission presents Spiking Autonomous Driving (SAD), the first ever end-to-end framework that uses spiking neural networks (SNNs) to build sub-blocks that target perception, prediction and planning tasks. This framework is trained end-to-end using a single cost function. SAD demonstrates comparable accuracies to existing artificial neural network (ANN) based baselines highlighted in the paper across the three tasks targeted using the nuScenes dataset. In addition thanks to the event-driven nature and single bit spike encoding of SNNs SAD achieves significant reduction in the energy consumption (up to ~70x) compared to other state-of-the-art Autonomous Driving ANNs. 

 The energy estimates presented are analytically calculated based on Op count and energy per op. This is good first order estimate, but it ignores the costs of operations like Batch Normalization, Heaviside step function, Tensor fusion etc. Also, its unclear how the authors account for the energy of the motion planning block (bicycle model) for SAD and other baselines. Regardless, the event-driven nature of SNNs along with the significant reduction in bits per Op for multiply-accumulate ops, in my opinion will lend SAD a notable energy efficiency benefit. Many reviewers pointed out that this analytical energy estimation misses the cost of memory accesses and data movement. The authors addressed those concerns in their rebuttal aptly by citing the fewer bits used by SNNs for their accumulation operations and recent SNN HW implementations using non-volatile memories that further lower the cost of data-movement. 

The reviewers also raised concerns related to lack of recent baselines (post ST-P3), lack of closed loop validation (unlike ST-P3) and an insufficient discussion on the robustness of SNNs for deployment in a safety-critical application like Autonomous Driving. While the first two concerns remain unanswered, authors intend to add robustness analysis to their future work section. 

Despite these limitations, the paper does present a novel result by training the first ever SNN based end-to-end autonomous driving framework, while demonstrating comparable accuracy to its ANN counterparts as acknowledged by reviewers, j4mM and XRtX. For this reason, I think this paper should be accepted.
Decision: Accept (poster)