Self-Supervised Pre-training of Spiking Neural Networks by Contrasting Events and Frames

Authors: raghav  singhal
jan  finkbeiner
emre  neftci
forschungszentrum  jülich
rwth  aachen
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-22 17:29:06

Abstract: Artificial Neural Network (ANN) pre-training, followed by fine-tuning, is an established procedure to solve real-world problems where labeled data is scarce. This paper aims to adapt this established procedure to the domain of event-based vision and Spiking Neural Networks (SNNs). Event-based sensors, inspired by the retina, capture visual scenes with low latency and high dynamic range, making them suitable for many real-world vision problems. SNNs, inspired by biological neural networks, when implemented on neuromorphic hardware, enable energy-efficient and low-latency processing, making them well-suited for fully event-based pipelines. However, the lack of sufficiently large labeled datasets hinders the pre-training of SNNs. Here, we leverage joint frame and event data to forego labeling. We achieve this using self-supervised contrastive learning, where an ANN and SNN pair are jointly trained to assimilate (contrast) (un)related frame-event stream pairs. We show that the pre-trained SNN model reaches higher accuracy on several downstream visual classification benchmarks. These results signify that pre-training large-scale SNNs using raw data output from event cameras is possible and paves the way toward foundation SNN models.

Main paper: 
 introduction: 
 Inspired by the brain's remarkable capabilities, neuromorphic systems strive to capture its key mechanisms for low-power, versatile, and fast information processing [1] [2] [3] . Similar to the brain, neuromorphic hardware aims to directly leverage the physics of the hardware, leading to a many-fold increase in efficiency, parallelism, and response speed compared to conventional computers. While the algorithms for training such networks have progressed dramatically in the last decade [4] [5] [6] [7] [8] , their applications still lag behind their conventional ANN counterparts. A key reason for this is the combined lack of suitable datasets, benchmarks, and training hardware, which collectively prohibit the highly practical workflow used in deep learning.Pre-training is the standard practice in deep learning, thanks to improved deployment speed, generalizability, and robustness. This enables a common backbone network to extract meaningful low-level features for the task at hand. Pre-training relies critically on the availability of large datasets which capture various factors, such as background, noise, lighting, and inherent object variations [9] . This practice has become even more common in recent self-supervised and foundational models, whose training from scratch is often beyond the computational resources of most practitioners [10, 11] .Frames Framesf θ Y h θ Z Events Events Events f ′ ϕ ′ U ′ T g ′ ϕ ′ Y ′ h ′ ϕ ′ Z ′ L VICRegFigure 1 : Given a batch of event streams and their corresponding frame-based images, we generate representations for images Y by passing them through an encoder f θ . In parallel, event streams are encoded via f ϕ ′ to produce a sequence of event representations U ′ T , which are aggregated using an accumulator g ′ ϕ ′ , yielding representations Y ′ . These representations are fed into expanders h θ and h ′ ϕ ′ to produce embeddings Z and Z ′ , which are jointly optimized using the VICReg loss. f ′ ϕ ′ and g ′ ϕ ′ (in green) are spiking in nature, while the other modules (in blue) are non-spiking.Typical pre-training approaches are difficult to implement in neuromorphic computing due to the lack of labeled datasets on the scale of ImageNet and the higher training cost. While the latter is being addressed by more suitable hardware that emphasize locality and sparsity [12] , the dataset labeling is a tedious process in videos and event streams. The lack of a neuromorphic SNN backbone is a critical obstacle to overcome before enabling their use in real-world applications. For example, in automotive traffic applications, the available event-based datasets are small compared to those of frame cameras [13] . This severely limits research in using novel vision sensors in real-world situations. A solid eventbased vision backbone can mitigate this limitation by requiring a comparatively small fine-tuning dataset, which may be recorded in a well-controlled and consensual scenario.Inspired by prior work in contrastive learning, we report here on a self-supervised algorithm to train a SNN on data recorded by neuromorphic vision sensors with simultaneous frame and event output. A majority of recent self-supervised techniques rely on data augmentations to achieve multiple views of the data [14, 15] . These different views are crucial to compute and minimize the similarity (contrast) losses [16] [17] [18] [19] [20] [21] [22] [23] [24] . Effective data augmentation techniques are critical in this context, however their application to event-based data has not been explored as extensively or rigorously as it has been for conventional image-based data. On the other hand, multimodal contrastive learning learns by associating pairs of data samples if they refer to the same object or concept. For example, CLIP [10] learns by associating a caption to an image if they are related while contrasting them if they are unrelated. Motivated by this approach, we show that it is possible to train an SNN using contrastive self-supervised learning by associating frame and event outputs to each other if they originate from the same visual sequence and disassociate them otherwise. This result is meaningful because very large datasets can be easily gathered using such cameras and pre-trained to provide a solid backbone for neuromorphic vision applications.Our approach uses a joint architecture, as depicted in Figure 1 , in which one branch is a conventional ANN and the other branch is an equivalent SNN. The ANN part embeds the frame output, while the SNN embeds the event output of the camera. The two branches are trained by minimizing the VICReg loss. The SNN follows Leaky Integrate and Fire (LIF) dynamics which are commonly used in software simulations and many digital and mixed-signal hardware emulations [25] [26] [27] [28] [29] . SNNs are compatible with modern ANN architectures such as Convnets [6, 30, 31] and transformers [32] . They can be trained using backpropagation through time using the well-established surrogate gradients approach [8] . We pre-train our model using a diverse corpus of frame-event pairs, synthesized from several open-source datasets, including DSEC [13] , Brisbane-Event-VPR [33] , DAVIS-DATA [34] , MVSEC [35] , and TUM-VIE [36] . Fine-tuning on downstream tasks, namely N-CARS [37] and CIFAR10-DVS [38] , yields substantial performance improvements, demonstrating the effectiveness of our pre-training strategy. Our results demonstrate performance gains from self-supervised pre-training of SNNs, when fine-tuned on diverse downstream tasks. 
methods: 
 Core Methodology. The overall architecture of our method is shown in Figure 1 . We begin by processing a batch of event streams alongside their corresponding frame-based images. This data pairing, while originating from the same visual modality, can be viewed as multimodal since it captures different perspectives of the same sensory input. Image representations, Y , are generated by passing the images through an encoder, f θ . In parallel, event streams are processed by a SNN encoder, f ′ ϕ ′ , resulting in a sequence of spike-based representations, U ′ T . To integrate these temporal spike representations, U ′ T is fed into an accumulator module, g ′ ϕ ′ , modeled as a Leaky Integrate layer with an infinite firing threshold. The final membrane potential, representing the cumulative spike activity across all time steps, forms the representation, Y ′ . Both sets of representations, Y and Y ′ , are subsequently processed through their respective expander networks, h θ and h ′ ϕ ′ , to produce the embeddings Z and Z ′ . These embeddings are optimized jointly using the VICReg loss function [39] , ensuring robust alignment between the frame-based and event-based modalities (details about the loss can be found in [39] ). VICReg is well-suited for this task as it independently regularizes each branch; it is thereby suitable for multimodal signal alignment tasks and helps to enhance the flexibility of joint-embedding self-supervised learning in such scenarios. During evaluation by fine-tuning on various downstream tasks, we only use the pre-trained SNN encoder, f ′ ϕ ′ , and discard the other modules. Pre-Training Dataset Processing. The diversity and quality of data used during pre-training are critical for achieving robust model performance. We pre-train our model on a diverse corpus of frame-event pairs drawn from several event camera datasets, including DSEC [13] , Brisbane-Event-VPR [33] , DAVIS [34] , MVSEC [35] , and TUM-VIE [36] . Importantly, no ground truth labels are required, as our approach solely relies on the alignment between frames and event streams. Given the varying frame sampling rates across datasets, careful preprocessing is essential. For each dataset, we extract the last frame within each sampling period and aggregate the corresponding event stream into 10 time bins, ensuring consistency across the diverse sources. Through this process, we collected a total of 181,207 event-frame pairs across all datasets. A summary of the datasets is given in Table 1 . DAVIS-DATA [34] alongside their corresponding greyscale frame-based images, encompassing scenarios such as office environments, urban settings, and simple objects and textures. The DSEC [13] and Brisbane-Event-VPR [33] datasets focus on event-based driving, capturing footage under a range of challenging lighting conditions. TUM-VIE [36] contains a diverse collection of handheld and head-mounted recordings in both indoor and outdoor contexts, including high dynamic range scenes. MVSEC [35] comprises recordings captured in various scenarios, including footage from a hexacopter, a car, and a motorcycle, all under different illumination levels and environmental conditions.We expect that the diverse scenes represented in this collection of datasets will promote improved knowledge transfer during the subsequent fine-tuning process on other datasets.SNN Details. The spiking neurons in our SNN follow the LIF model dynamics [40] . The dynamics are described by the following discrete-time equations:EQUATIONHere, X[t] is the input current at time step t, H[t] represents the membrane potential before spiking, and V [t] is the membrane potential after a spike occurs. The neuron fires when H[t] crosses the threshold V th , which is determined by the Heaviside step function Θ(x). The spike output at time step t is denoted by S[t], while V reset is the reset potential after a spike. The constant β governs the membrane decay. Since the step function is non-differentiable, we apply the surrogate gradient method [8] , to approximate the derivative of the function, using the arctan surrogate function [41] .Implementation Details. We use PyTorch and SnnTorch [42] to simulate spiking neurons for training and Tonic [43] for dataset processing. The ANN and SNN encoders are the ResNet-18 model [44] and its adapted spiking variant [45] , respectively. In the spiking ResNet-18 model, we replace all ReLU activations with leaky integrate-and-fire (LIF) modules and substitute max-pooling operations with average pooling layers. The membrane decay potential β is set to 0.5, and the reset potential V reset to 0. Both the ANN and SNN encoder networks consist of 512 output units. The expander networks comprise two-layer fully-connected models with batch normalization and ReLU activations, where each expander layer has 4096 units, as larger expander dimension sizes have been shown to enhance performance [46] . Notably, neither the encoder nor the expander modules share weights. We resize each event and frame data sample to a spatial resolution of 224 × 224 pixels. Following the training protocols established by BYOL [47] , Barlow Twins [46] , and VICReg [39] , we jointly train the models using the LARS optimizer [48] , with a learning rate defined as lr = base_lr × batch_size /256, where base_lr is set to 0.2 and the batch size is 128. The learning rate follows a cosine decay schedule [49] , starting from 0 with 10 warm-up epochs, ultimately reaching a final value of 0.002. We use the same VICReg loss regularization coefficients as used in the original paper [39] , and pre-train the models for 200 epochs.Evaluation on Downstream Tasks. We evaluate our pre-trained SNN model on two downstream benchmark event-based datasets for visual classification: CIFAR10-DVS [38] and N-CARS [37] . We resize the spatial resolution of all samples to 128 × 128 pixels across both datasets. CIFAR10-DVS, derived from the original CIFAR10 dataset, consists of 10,000 DVS recordings. We use a 9:1 trainvalidation split, allocating 9,000 samples for training and 1,000 for validation. N-CARS, a real-world event-based dataset for binary object classification, comprises 15,422 training samples (7,940 cars, 7,482 background) and 8,607 test samples (4,396 cars, 4,211 background). For both datasets, we bin events into 10 time steps and set the membrane potential decay factor β to 0.5. We optimize our models using Adam [50] with an initial learning rate of 0.001, following a cosine decay schedule [49] , and a weight decay of 10 -4 . The results are reported as the average over 3 different runs. Table 2 presents the performance of the spiking ResNet-18 model when fine-tuned on varying percentages of the CIFAR10-DVS and N-CARS training sets, with a linear classifier appended. Our proposed pre-training scheme consistently leads to a significant improvement in performance compared to the baseline without pre-training (fully supervised training). This improvement is especially notable when fine-tuning on smaller subsets of the training data, demonstrating the intended effectiveness of our method in low-data regimes. Our results show successful demonstration of self-supervised transfer learning in SNNs, with substantial gains observed on downstream datasets that were not part of the original training set. This highlights the model's capacity to transfer knowledge and leverage representations learned during pre-training to enhance performance on unseen tasks. Additionally, the success of this approach opens new avenues for using SNN architectures in neuromorphic vision applications, laying a strong foundation for the development of versatile and generalizable SNN backbones. Our method also mitigates the challenges associated with limited exploration of data augmentation techniques for event-based data by simply enriching the event data with already existing frame-based images.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Contrastive learning for Spiking
main_review: main_review: Though I am not an expert in Spiking Neural Networks (SNNs), the paper "Self-Supervised Pre-training of Spiking Neural Networks by Contrasting Events and Frames" presents a compelling and innovative approach. It addresses a key challenge in neuromorphic computing: the scarcity of labeled event-based datasets for pre-training SNNs. By leveraging contrastive learning between frame and event streams, the authors propose a self-supervised learning strategy that pairs an ANN with an SNN, enabling efficient pre-training without requiring labels. The method demonstrates significant improvements on downstream tasks like CIFAR10-DVS and N-CARS, especially when data is limited, indicating its effectiveness in low-data scenarios. The architecture and methodology are clearly outlined, and the results suggest potential for broader applications in neuromorphic vision. While the paper presents a solid contribution, further exploration into other datasets and architectures could help establish the approach's versatility and impact on the field. Overall, the paper appears to offer valuable insights into enhancing SNN performance through self-supervised learning.

I guess this could be an intersting contribution for UNIREPS, being the topic align the representations of ANN and SNN
title: title: Self-supervised contrastive learning for event-based images via alignment to conventional images
main_review: main_review: Spiking Neural Networks (SNNs) are a favorable paradigm in the domain of event-based computer vision. This paper proposes a self-supervised contrastive learning method for event-based vision to overcome the lack of labeled data. In particular, the proposed method has a dual pipeline with the conventional non-spiking model and a SNN that are optimized using the VICReg loss. The last stage of the dual architecture has a block to align spiking and non-spiking representations. Transfer learning experiments show that the SNN that have been pretrained using the proposed method performs better in the downstream tasks (classification) than a SNN without pretraining. Overall, the motivation of this paper is appealing but I believe, the proposed direction has a clear limitation. In particular, it requires to have paired data rather than trying to perform contrastive learning directly on the event-based data which is not available in all practical setups. Moreover, the "proxy" alignment approach can result in a complicated contrastive learning procedure with its own issues. Lastly, the proposed method has no brave academia contribution where the latter is the goal for workshop extended abstracts.
title: title: A good paper with excessive citations
main_review: main_review: 1. 77 cited articles for a 4 page workshop paper seems excessive. In line 45, citing 9 articles [16-24] seems unnecessary.
2. Are there other pre-training schemes in previous literature for SNNs?
3. How do you explain the test accuracy going up 4% with a 10% train set, then 9% with a 50% train set, when the test accuracy went up 9% with a 1% train set in the CIFAR10-DVS dataset? In the N-CARS dataset, the delta is decreasing consistently with the train set size increase.