SPACT18: Spiking Human Action Recognition Benchmark Dataset with Complementary RGB and Thermal Modalities

Authors: Yasser Ashraf, Ahmed Sharshar, Velibor Bojkovic, Bin Gu
Primary area: datasets and benchmarks
Keywords: Spiking Camera, Spiking Neural Network, Video Understanding, Video Action Recognition

Abstract:
Spike cameras, bio-inspired vision sensors, asynchronously fire spikes by accumulating light intensities at each pixel, offering ultra-high energy efficiency and exceptional temporal resolution. Unlike event cameras, which record changes in light intensity to capture motion, spike cameras provide even finer spatiotemporal resolution and a more precise representation of continuous changes. In this paper, we introduce the first video action recognition (VAR) dataset using spike camera, alongside synchronized RGB and thermal modalities, to enable comprehensive benchmarking for Spiking Neural Networks (SNNs). By preserving the inherent sparsity and temporal precision of spiking data, our three datasets offer a unique platform for exploring multimodal video understanding and serve as a valuable resource for directly comparing spiking, thermal, and RGB modalities. This work contributes a novel dataset that will drive research in energy-efficient, ultra-low-power video understanding, specifically for action recognition tasks using spike-based data.

Decision: reject
Reviewer scores: 5, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730082451898

thread_type:
official_review

<thread_object>
creator: reviewer_yfhy

<content>
summary:
This paper introduces SPACT18, a novel video action recognition (VAR) dataset captured using a spike camera, alongside synchronized RGB and thermal modalities.

strengths:
Introducing a spike camera-based dataset for video action recognition is a novel contribution. The dataset could become a valuable resource for researchers exploring energy-efficient spiking models for real-time applications, particularly in edge devices or autonomous systems.

weaknesses:
1. The experimental setup and analysis are not sufficiently thorough.  The dataset is aimed at benchmarking Spiking Neural Networks (SNNs) for action recognition tasks, but the paper does not include any direct evaluation using SNN-based methods. Since the core claim is that this dataset enables effective benchmarking for SNNs, its advantages in temporal processing remain unproven without such an evaluation. Including results from SNN-specific architectures or methodologies (eg. directly trained SNN models and ANN-SNN conversion methods) would strengthen the paper’s contribution to the field.

2. The dataset's scale and diversity are limited compared to established RGB video action recognition benchmarks such as HMDB-51, UCF-101, and Kinetics-400, which feature far more action classes (51, 101, and 400, respectively). SPACT18 includes only 18 action classes and a significantly lower number of training clips. This restricts its utility for developing models that generalize well across diverse action recognition tasks, limiting the dataset’s value for broader use in the community.

3. Overclaiming on Spike Camera superiority. The paper claims that spike cameras offer significant advantages over event cameras. However, this claim is not supported by sufficient evidence or direct comparisons. To substantiate such claims, the authors should provide side-by-side evaluations of spike and event camera performance, particularly in relevant tasks like video action recognition, or refer to studies that have explored these comparisons in depth.

4. The paper overstates the complexity of action recognition relative to tasks such as optical flow estimation, motion estimation, depth estimation, or image reconstruction [L133-138]. Action recognition, being a classification task, is often less complex than these tasks, which involve per-pixel regression and dense predictions. It would be more accurate to frame action recognition as an important but distinct problem, rather than more complex, to avoid misrepresentation.

questions:
Could you clarify the advantages of using spike cameras over event cameras, RGB cameras, and thermal data? From Table 4, it appears that performance using spike data is consistently lower, and Figure 6 shows that spike data is quite blurry, which may make it harder for models to learn meaningful dynamic information. How do you address these limitations, and in what situations does spike data excel over other modalities?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538589387

thread_type:
official_review

<thread_object>
creator: reviewer_jxr8

<content>
summary:
This paper introduces an action recognition dataset incorporating spike, RGB, and thermal modalities. Additionally, it presents a spiking data compression algorithm designed for preprocessing and compressing the dataset. The dataset is evaluated using state-of-the-art lightweight ANN and SNN models.

strengths:
- The dataset is original and diverse, featuring 44 participants and 18 activities, making it a sufficiently challenging resource for validating models.
- With three modalities—spike, RGB, and thermal—the dataset is highly informative and suitable for researchers in image processing, spiking neural networks, and multimodal analysis.
- The writing is clear, and the paper is well-structured.
- The authors validate the dataset using popular architectures, demonstrating its applicability.

weaknesses:
1. The paper lacks a thorough comparison with existing action recognition datasets, particularly in terms of modality, participant diversity, types of activities, sample size, and duration. Given that the primary contribution of this work is the proposed dataset, a clear comparison would help emphasize its advantages over existing datasets.
2. The paper cites only one spike-based action recognition dataset (Amir et al., 2021). However, there exist other spike-based action recognition datasets, such as (Wang et al., 2024, Liu et al., 2021, Miao et al., 2019). It would be beneficial for the authors to consider these datasets in their comparison and explore whether any other relevant action recognition datasets are available.
3. Could you provide the confusion matrix for one or several models in Table 3? Since some activities in the proposed dataset overlap with those in [Amir et al., 2021], a confusion matrix could help determine if the additional activities are sufficiently discriminative and enhance the dataset's value.
4. The authors clearly specify the recording devices and versions for the thermal and RGB data. However, the spike camera is not similarly described. Could you include details about this device for completeness?
5. I have a question regarding the proposed compression method. Specifically, what distinguishes the approach of calculating frequency first and then using IF neurons from directly employing leaky LIF neurons? Would using leaky LIF neurons potentially offer a simpler and more direct solution?

Wang et al., 2024, DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition, ECCV.
Liu et al., 2021, Event-based Action Recognition Using Motion Information and Spiking Neural Networks, IJCAI.
Miao et al., 2019, Neuromorphic vision datasets for pedestrian detection, action recognition, and fall detection. Frontiers in Neurorobotics.

The authors addressed some of my concerns and I raised my score.

questions:
Can you address the 1st, 3rd, 4th and 5th points I mentioned in the weakness section?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636903385

thread_type:
official_review

<thread_object>
creator: reviewer_jr4n

<content>
summary:
SPACT18 has developed a novel VAR dataset for fully incorporating SNNs into VAR tasks. It is commendable that the author considered multiple variables in the process of constructing the dataset. In addition, the author's introduction is complete and logically fluent, with good narrative quality. In the methodology section, the author provides a complete introduction to the data construction process, its basic data architecture, and the most important event data alignment steps. However, in the experimental section, the author only tested the effectiveness of two typical algorithms and used the ANN-SNN conversion to test the performance of the SNN algorithm, which is incomplete. In future work, a larger number of ANN algorithms should be supplemented for testing to verify the stability of the distribution of the SPACT18 dataset. In addition, algorithm testing based entirely on SNN is also necessary. In summary, this article provides a large-scale VAR dataset based on events and innovatively proposes an event alignment strategy, which is in line with the logic and innovation required for conference papers.

strengths:
This article proposes SPACT18- a novel event based large-scale VAR task dataset. The dynamic response characteristics of events and the low power consumption of SNNs provide new possibilities for the development of VAR. Meanwhile, the method section innovatively proposes an alignment method for event pulses, which is a unique compression method suitable for VAR tasks. This article has made certain progress and contributions to VAR tasks.

weaknesses:
The experimental part appears slightly thin. Using only two typical VAR algorithms for verification is insufficient for verifying the distribution of RGB and thermal modal data. At the same time, the validation of the SNN algorithm is still based on the ANN-SNN conversion method, which cannot fully utilize the parsing ability that SNN should have.

questions:
(1) Has a detailed analysis been conducted on the impact of external factors such as age, gender, and race on the distribution of the dataset and the effectiveness of VAR;
(2) Does the architecture of the SPACT18 dataset include commonly used data collection methods (RGB/thermal mode) and analyze the differences between SPACT18 and other existing datasets;
(3) Due to the fact that video understanding typically adopts frame by frame input, i.e. still in the form of a single frame image sequence. Is there an analysis of the contribution of each frame to the video action recognition task? I believe that different cumulative states of the event stream can be equivalent to one frame of video image, and the contribution of each frame will be particularly prominent in the event camera;
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687423952

thread_type:
official_review

<thread_object>
creator: reviewer_ddyr

<content>
summary:
This paper proposes the first video action recognition using spike camera, alongside synchronized RGB and thermal modalities.

strengths:
1. This paper is simple and easy to understand.

weaknesses:
1.  The authors emphasize that the temporal resolution of spike cameras is superior to that of event cameras.  But I think that's wrong. The proof is not a hardware paper but an algorithm paper, which I don't think is a good starting point. The paper [1]  would have been able to do 3.6us  temporal resolution compared to the spike camera even in 2011.

[1] Leñero-Bardallo J A, Serrano-Gotarredona T, Linares-Barranco B. A 3.6$\mu $ s Latency Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor[J]. IEEE Journal of Solid-State Circuits, 2011, 46(6): 1443-1455.

2. In line 60,  the thermal modality is used very suddenly, just to provide a comprehensive multimodal representation, and why this modality is important, I think, needs to be elaborated.

3. The references are out of date and need a more comprehensive review.

4. Different models should be utilized in the experiment to give more comparable results.

5. There is a lack of comparison of model experiments for SNNs that direct train.

6. Please reference these paper to set up and complete the experiment part [2],[3]. 

[2] Wang X, Wang S, Tang C, et al. Event stream-based visual object tracking: A high-resolution benchmark dataset and a novel baseline[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 19248-19257.
[3] Duan Y. LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 25637-25647.

questions:
None
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper has undergone review by four referees, with two suggesting acceptance and two others expressing reservations. Regarding the research motivation, the paper proposes using a Spike Camera for motion detection; however, the camera’s strength lies in capturing high-speed movement, not in performing well under low-light conditions. The inclusion of thermal infrared and RGB cameras could address this issue, but the paper’s focus on the Spike Camera is less convincing. Taking into account the concerns raised by the other reviewers, the decision is to reject the manuscript.