SpikSSD: Better Extraction and Fusion for Object Detection with Spiking Neuron Networks

Authors: Yimeng Fan, Changsong Liu, Mingyang Li, Wei Zhang
Primary area: applications to neuroscience & cognitive science
Keywords: Spiking Neural Networks, Object Detection, Membrane-based Deformed Shortcut, Spiking Bi-direction Fusion Module

Abstract:
As the third generation of neural networks, Spiking Neural Networks (SNNs) have gained widespread attention due to their low energy consumption and biological interpretability. Recently, SNNs have made considerable advancements in computer vision. However, efficiently conducting feature extraction and fusion under the spiking characteristics of SNNs for object detection remains a pressing challenge. To address this problem, we propose the SpikSSD, a novel Spiking Single Shot Multibox Detector. Specifically, we design a full-spiking backbone network, MDS-ResNet, which effectively adjusts the membrane synaptic input distribution at each layer, achieving better spiking feature extraction. Additionally, for spiking feature fusion, we introduce the Spiking Bi-direction Fusion Module (SBFM), which for the first time realizes bi-direction fusion of spiking features, enhancing the multi-scale detection capability of the model. Experimental results show that SpikSSD achieves 40.8\% mAP on the GEN1 dataset and 76.0\% mAP@0.5 on the VOC 2007 dataset with only around 10\% firing rate, outperforming existing SNN-based approaches at ultralow energy consumption. This work sets a new benchmark for future research in SNN-based object detection. Our code is publicly available in supplementary materials.

Decision: reject
Reviewer scores: 3, 6, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730113704772

thread_type:
official_review

<thread_object>
creator: reviewer_v2pm

<content>
summary:
This paper proposes a spiking neural network backbone MDS-ResNet, and a spiking object detection model SpikSSD. Based on EMS-ResNet, the proposed MDS-ResNet modifies the shortcut connection in the residual blocks. The SpikSSD combines the proposed bi-directional feature fusion and MDS-ResNet. Experimental results show that the proposed SpikSSD outperforms existing spiking object detection methods.

strengths:
1. This paper analyzes in detail the gradient vanishing or explosion for MDS-ResNet.
2. The proposed SpikSSD model achieves state-of-the-art mAP metric performance on GEN1 and VOC2007 datasets compared to existing spiking object detection methods.

weaknesses:
1. In the method section, the authors spend a lot of words introducing the proposed backbone MDS-ResNet. However, MDS-Blocks 1, 2, and 4 are already utilized in MS-ResNet and EMS-ResNet, while only MDS-Block 3 is newly proposed, and it closely resembles MDS-Block 2. Moreover, the gradient analysis of MDS-ResNet is similar to EMS-ResNet. Therefore, I believe that the MDS-ResNet architecture should not be regarded as a main contribution of this paper. I suggest that the authors detail why MDS-Block is more effective for object detection tasks than MS-Block and EMS-Block.
2. The SpikSSD is only evaluated on VOC 2007 and GEN1 datasets, while EMS-ResNet is evaluated mainly on COCO 2017 dataset. I wonder if SpikSSD also works well on COCO 2017. If possible, please provide the experimental results on COCO 2017 dataset.
3. Figure 1 illustrates the firing patterns of MDS-ResNet and EMS-ResNet. However, it does not indicate on which task or dataset the evaluation is performed. Furthermore, it is not clear whether this phenomenon holds for all tasks or only for specific tasks.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730426476929

thread_type:
official_review

<thread_object>
creator: reviewer_troc

<content>
summary:
This paper introduces SpikSSD, a Spiking Single Shot Multibox Detector that leverages a fully spiking backbone network (MDS-ResNet) and a novel Spiking Bi-direction Fusion Module (SBFM) for object detection in Spiking Neural Networks (SNNs). The model shows strong performance on the GEN1 and PASCAL VOC 2007 datasets, achieving a mean average precision (mAP) comparable to or better than existing SNN-based models with a low firing rate.

strengths:
The paper highlights the low energy consumption of SpikSSD, reporting approximately 10% firing rates, which is notable in comparison to conventional ANNs.

weaknesses:
The results are primarily limited to GEN1 and VOC 2007, which may not fully represent the performance across diverse object detection tasks. The paper could be strengthened by including more diverse datasets or a comprehensive analysis of SpikSSD's generalization capabilities. While the paper introduces the MDS to stabilize membrane synaptic inputs, the theoretical foundations are only briefly discussed. A more rigorous exploration of why MDS effectively stabilizes membrane potential and prevents gradient vanishing/explosion would strengthen the paper’s contributions. Additionally, it’s unclear how the MDS compares with existing methods for managing synaptic input variance in SNNs. Although the results are promising, the comparisons are limited to SNN-based models. The paper would benefit from a broader comparison with non-SNN state-of-the-art object detectors on similar datasets to better highlight SpikSSD's competitiveness in real-world applications. Given that SNNs are often measured against ANN counterparts, direct ANN comparisons could showcase where SNNs might still be falling short. While SBFM achieves bi-directional fusion, the paper lacks a detailed justification of why this method is uniquely advantageous in SNNs as opposed to standard fusion techniques. It would also be helpful to clarify the specific benefits of using a spiking-only architecture over incorporating mixed approaches if they offer potential improvements in performance. The paper does not include ablation studies to validate the specific contributions of MDS-ResNet and SBFM. Without this, it is difficult to assess whether the performance gains are directly attributable to these components. Such studies could better clarify the impact of each proposed module.

questions:
Certain sections lack clarity, particularly regarding the mechanisms of MDS and SBFM. Visualizations of neuron firing patterns and more illustrative figures could help elucidate these mechanisms. Additionally, providing more detail in the methodology could allow for better reproducibility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730555300869

thread_type:
official_review

<thread_object>
creator: reviewer_p8ri

<content>
summary:
This paper proposes an SNN-based single-shot multi-box detector SpikSSD. In the SpikSSD, this paper proposes a backbone MDS-ResNet. The MDS-ResNet adjusts the distribution of the input membrane synaptic to avoid gradient vanishing and explosion. Besides, this paper also proposes a Spiking Bi-direction Fusion Module (SBFM) to fuse spiking features in a bi-directional way. The proposed method is tested on GEN1 and VOC 2007 datasets, where the SpikSSD achieves good performance with only around 10% firing rate.

strengths:
1. The proposed MDS-ResNet adjust uses tdBN in the shortcut to adjust the distribution of input membrane synaptic to alleviate the problems brought by gradient vanishing and explosion.
2. This paper gives a theoretical analysis of gradient vanishing/explosion for MDS-ResNet.

weaknesses:
1. This paper claims they propose a backbone MDS-ResNet based on spiking neural networks. Is the backbone specifically designed for object detection? If so, what part of MDS-ResNet is specifically designed for objection detection? If not, it should be evaluated on classification datasets such as ImageNet [1] to show the generalization of the proposed backbone [2-5].
2. In the area of object detection, COCO [6] is a general dataset, why not evaluate the proposed method on this dataset?
3. What's the motivation for the proposed bi-directional fusion? What challenges are handled by the bi-directional fusion? It would be better to introduce more about that.
4. The experimental table of the paper shows the proposed method has a low energy. How is the low energy consumption realized? On general computational devices or specifically designed devices? If the low energy depends on specifically designed devices, how is the low energy realized?

[1] J. Deng, et al., Imagenet: A large-scale hierarchical image database. CVPR 2009.  
[2] Y. Li, et al., Differentiable spike: Rethinking gradient-descent for training spiking neural networks. NeurIPS 2021.  
[3] W. Fang, et al., Deep residual learning in spiking neural networks. NeurIPS 2021.  
[3] H. Zheng, et al., Going deeper with directly-trained larger spiking neural networks. AAAI 2021.  
[4] X. Yao, et al., GLIF: A unified gated leaky integrate-and-fire neuron for spiking neural networks. NeurIPS 2022.  
[5]  W. Fang, et al., Parallel spiking neurons with high efficiency and ability to learn long-term dependencies. NeurIPS 2023.  
[6] T. Lin, et al., Microsoft COCO: common objects in context. ECCV 2014.

questions:
Please see the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625102259

thread_type:
official_review

<thread_object>
creator: reviewer_ff5q

<content>
summary:
This paper presents an MDS architecture for solving the problem of SNN target detection. This approach enables the model to further utilize spike operations to reduce computing power. The authors validated it on multiple target detection datasets.

strengths:
1. The author's analysis of the spike operation in the algorithm is very accurate, and based on this, he designed a spike-friendly MDS architecture.
2. The method proposed by the authors has good performance and energy efficient. (more elegant than EMS method~)
3. The author provides a relatively complete theoretical analysis.

weaknesses:
1. This article appears to be incremental work.
2. See Question part.

questions:
1. Can you make a more detailed comparison of the EMS and MDS architectures in terms of formulas? And show why the EMS architecture cannot be fused to the spike operation?
2. Regarding the calculation of energy consumption, can you give a discussion on how to deploy it on real hardware? Similarly, I would like to see a comparison between MDS and other model architectures?
3. What do you think of the method of gradient identity analysis? Are there any problems with this method's series of assumptions in discrete models such as SNN?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644303917

thread_type:
official_review

<thread_object>
creator: reviewer_wnby

<content>
summary:
The paper presents SpikSSD, an SNN-based object detection network. SpikSSD introduces a spiking residual network (MDS-ResNet) for feature extraction and Spiking Bi-direction Fusion Module (SBFM) for multi-scale feature fusion. Experiments on GEN1 and VOC dataset indicates the best performance of SNN-based object detection.

strengths:
1.SpikSSD shows improvements in energy efficiency, achieving a 10% firing rate with competitive accuracy, which is beneficial for power-constrained applications.
2.Sufficient mathematical proof of the network structure mathematically, which proves the liability of the proposed MDS-ResNet.
3.Experiments on GEN1 and VOC proves the performance of SpikeSSD. On both dataset SpikeSSD proves to be the best SNN-based object detection network.

weaknesses:
1.For object detection on images, the authors only provide experiment results on VOC dataset, which is insufficient since MS-COCO dataset is the mainstream dataset for this task. VOC dataset has much fewer images, which lacks of utility. There should be experiments on MS-COCO dataset and real-world images for validation qualitatively.

2.SBFM is too complicated for SNN-based network. SBFM fuses different scales of feature maps, but it is too complicated for training. To meet the requirement of SSD detection head, multiple features need to be generated. SBFM has a PAN-like structure but with 5 feature maps. There could be a more efficient way of feature fusion other than the proposed SBFM.

3.The MDS-ResNet primarily extends SNN modules on mature ANN networks, which lacks new design or new insight on SNN. Core ideas like membrane-based shortcuts and spiking feature fusion have seen prior application.

SpikSSD provides a model that is interesting but the contributions are incremental, offering refinements on existing modules rather than making novel breakthroughs. Additionally, without real-world experiments, it is hard to identify the claimed efficiency gains. The paper would benefit from hardware validation and further simplification for real-world deployment or application, rather than merely experimenting on certain dataset.

questions:
1.There could be experiments on MS-COCO dataset and real-world images for validation qualitatively.

2. Could explain the following part with efficient manner? "SBFM has a PAN-like structure but with 5 feature maps. There could be a more efficient way of feature fusion other than the proposed SBFM."
</content>
</thread_object>

</thread>
<thread>
creation date:
1732369658171

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To All Reviewers

comment:
We sincerely thank all reviewers for their valuable feedback and suggestions on our paper. Here, we would like to address the concerns regarding the performance of SpikSSD on the COCO 2017 dataset.

On the COCO 2017 dataset, as demonstrated in Tab. R1, SpikSSD not only achieves the second-highest mAP@0.5 among all SNN-based methods but also ranks first among directly-trained SNN-based methods while consuming less energy.
Notably, while SUHD achieves slightly better results on COCO 2017, our approach outperforms it on the VOC 2007 dataset, demonstrating the overall superiority of SpikSSD across different frame-based datasets. 
Moreover, when compared to ANN-based methods, SpikSSD-L demonstrates comparable performance while requiring merely 1/3 (YOLOV5s) or even 1/37 (DETR) of the energy consumption.

| Method                | Model           | Param(M) | mAP@0.5  | Energy(mJ) |
| :-------------------- | :-------------- | :------: | :------: | :--------: |
| ANNs                  | YOLOv5s         |   7.3    |   54.8   |   38.41†   |
| ANNs                  | **DETR**        | **41.0** | **55.7** | **430.10** |
| ANNs to SNNs          | Vthfast+Vthacc  |   8.9    |   21.1   |     -      |
| ANNs to SNNs          | Spiking-YOLO    |   8.9    |   25.7   |     -      |
| ANNs to SNNs          | SpiCalib        |   23.9   |   45.4   |     -      |
| ANNs to SNNs          | **SUHD**        |  **-**   | **54.6** |   **-**    |
| Directly trained SNNs | Hybrid-YOLO     |    -     |   26.2   |     -      |
| Directly trained SNNs | EMS-YOLO        |   26.9   |   50.1   |   29.00    |
| Directly trained SNNs | **SpikeFormer** | **75.0** | **51.2** | **140.80** |
| Directly trained SNNs | **SpikSSD-S**   | **16.3** | **49.2** | **11.65**  |
| Directly trained SNNs | **SpikSSD-L**   | **21.3** | **52.4** | **15.59**  |

**Table R1: Comparison with state-of-the-art models on the COCO 2017 dataset.** † indicates that energy is recalculated using our energy consumption method for fair comparison.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes SpikSSD, a Spiking Single Shot Multibox Detector. It leverages a full-spiking backbone network, MDS-ResNet, and a Spiking Bi-direction Fusion Module (SBFM). Experimental results show that SpikSSD outperforms existing SNN-based approaches with a low firing rate.

After the rebuttal period, 5 reviewers rate 3, 3, 3, 6, 6, respectively. All reviewers agree that the proposed SpikSSD has excellent performance and low energy consumption, which are the strengths of this paper. However, there are some criticisms. Reviewer wnbY thinks that the contributions are incremental. Reviewer TRoC thinks that the mechanisms of MDS and SBFM lack clarity. Reviewer V2pm believes that the motivation of MDS-ResNet is too complex. I think these criticisms are justified. In my opinion, this paper is not well-written. I suggest that the authors further clarify the proposed mechanisms and highlight the contributions. For example, it would be helpful to add a figure comparing with the existing methods which do not have bidirectional fusion to highlight the contribution of SBFM.

While the authors argue that Reviewers TRoC and V2pm overlooked the key contributions of the proposed SBFM and focused on MDS-ResNet, I believe this may be due to poor content organization. As Reviewer V2pm states, "The authors spend a lot of words introducing the proposed backbone MDS-ResNet". Sections 3.2 and 3.3 are significantly longer than Section 3.4, making the paper focus on MDS-ResNet rather than SBFM. I suggest that the authors reorganize the paper to highlight the contribution of SBFM.

Overall, I believe this paper would benefit from further revision. Therefore, the final decision is to reject this paper.