<figure><img src="image_1.png" /></figure>

HAL
open science

# Cross-Modal Neuromorphic Semantic Segmentation based on Knowledge Distillation

Dalia Hareb, Jean Martinet, Benoit Miramond

► To cite this version:

Dalia Hareb, Jean Martinet, Benoit Miramond. Cross-Modal Neuromorphic Semantic Segmentation based on Knowledge Distillation. International Joint Conference in Neural Network, Jun 2025, Rome, Italy. pp.1-9, 10.1109/IJCNN64981.2025.11227429 . hal-05192745

HAL Id: hal-05192745

https://hal.science/hal-05192745v1

Submitted on 30 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Cross-Modal Neuromorphic Semantic Segmentation based on Knowledge Distillation

Dalia Hareb

I3S & LEAT, Univ. Cote d'azur, CNRS

Sophia Antipolis, France
hareb@i3s.unice.fr

Jean Martinet

I3S, Univ. Cote d'azur, CNRS

Sophia Antipolis, France
jean.martinet@univ-cotedazur.fr

Benoit Miramond

LEAT, Univ. Cote d'azur, CNRS

Sophia Antipolis, France
benoit.miramond@univ-cotedazur.fr

**Abstract**—Event-based cameras, with their high temporal resolution and low energy consumption, offer significant advantages over conventional cameras for computer vision tasks. However, current semantic segmentation algorithms for event-based data face challenges in achieving optimal performance for two main reasons: 1) the mismatch between sparse event streams (even when encoded into pseudo-frames) and the dense data structure of traditional frame-based images for which most existing implementations were designed, and 2) the lack of texture information in events, which solely detect temporal variations in brightness. To address this challenge, we propose a novel Cross-Modal (CM) Knowledge Distillation (KD) approach. Our method transfers knowledge from a high-performing Artificial Neural Network (ANN) processing fused grey-scale images and events to a Spiking Neural Network (SNN) – a bio-inspired, energy-efficient computing paradigm – operating on event data alone. Experiments on the DDD17 and DSEC-semantic datasets demonstrate that our approach significantly improves semantic segmentation results while reducing SNN energy consumption. This work represents the first application of cross-modal knowledge distillation to neuromorphic semantic segmentation, paving the way for more efficient event-based vision systems.

**Index Terms**—semantic segmentation, event-based cameras, spiking neural network, cross-modal knowledge distillation

## I. INTRODUCTION

Evolved from biological retinas, event-based cameras [1] have become attractive visual sensors in the field of computer vision as they overcome some drawbacks associated with standard frame cameras thanks to their high temporal resolution (in the order of µs versus s for frame cameras), high dynamic range ( > 120dB versus 60dB), low latency (10ms on the lab bench, and sub-millisecond in the real world) and low power consumption (100mW max) [2]. Such features are essential for computer vision tasks in challenging scenarios, such as exiting a tunnel in bright sunlight or when children unexpectedly jump in front of a car. These benefits arise from the asynchronous measurement of per-pixel brightness changes, resulting in the generation of a stream of events denoted as (x, y, p, t), where x and y represent the spatial coordinates, p the polarity (positive for pixel brightness increase and negative otherwise), and t the time of occurrence.

Given the asynchronous and sparse nature of events generated by these cameras, we turn to spiking neural networks for their processing. SNNs are a biologically inspired paradigm in artificial intelligence that more closely mimics the functioning

of the human brain. They employ an asynchronous sparse event-driven computation paradigm for processing data in a spatial-temporal manner [3], making them particularly suited for processing events. This type of neural network uses spikes as units of information and these spikes are generated only when the internal state of a neuron, known as the membrane potential, exceeds a specified threshold. Upon reaching this threshold, the membrane potential undergoes a hard reset to 0 or a soft reset, where the threshold value is subtracted. If the threshold is not reached, the neuron, depending on its intrinsic characteristics (Integrate-and-Fire 'IF' or Leaky Integrate-and-Fire 'LIF' [4]), either maintains its current state or undergoes a decay over time determined by a given factor. Consequently, each layer in an SNN produces a binary tensor, indicating whether a spike has been triggered (1) or not (0). Given this binary representation, SNNs use only additions, distinguishing them from ANNs where multiplications and additions are involved in the computation. This, coupled with their asynchronous nature, contributes to their energy efficiency, resulting in lower power consumption while being implemented in dedicated hardware. For instance, Lemaire et al. [5] demonstrated energy savings of up to 8 times in SNNs compared to ANNs. Similarly, Khacef et al. [6] achieved comparable performance between SNNs and ANNs with lower hardware implementation costs. Additionally, Dampfhoffer et al. [7] reduced energy consumption in spiking GRUs through regularization methods. This low power consumption addresses a crucial challenge in adopting ANNs, characterized by substantial computational costs and high memory consumption, especially in embedded scenarios such as Internet of Things (IoT) devices, automotive applications, and Augmented Reality/Virtual Reality (AR/VR) systems. Nonetheless, this binary representation presents challenges during backpropagation training, as it can lead to unstable gradients and weight updates of 0 or infinity. The Surrogate Gradient (SG) method [8] addresses this issue by employing a differentiable approximation, although it may introduce estimation errors that can slow convergence relative to traditional ANNs.

Despite the advantages of event-based cameras and SNNs, a significant challenge remains: while event-based cameras excel at capturing dynamic changes, they often fall short in providing the dense semantic information necessary for robustness and accuracy in computer vision applications.
Furthermore, most existing computer vision algorithms are primarily designed for frame-based images, neglecting the unique data format generated by event-based sensors. To tackle these limitations, we propose a novel method in this study that harnesses the complementary strengths of event-based and frame-based cameras. Initially, we employed grey-scale images processed by ANNs to enhance SNN performance on event data through knowledge distillation. However, our ablation study (Sec. IV-C) revealed that transferring features from dense to sparse data was suboptimal. Consequently, we developed a cross-modal knowledge distillation approach in which the teacher network processes features from grey-scale images that are fused with event data using an ANN-based Dual-path Network (ADN). This method enables the teacher to transfer valuable knowledge during training to the SNN-based student, thereby enhancing its performance in semantic segmentation tasks when processing event data during inference. Our work presents the first method that effectively integrates inputs from both event-based and frame-based cameras to tackle semantic segmentation challenges using SNNs through cross-modal knowledge distillation.

The paper is structured as follows. First, a review of relevant literature in event-based semantic segmentation, as well as knowledge distillation in SNNs, is conducted in Sec. II. Following this, a detailed description of the proposed kd model is presented in Sec. III. This includes an overview of both the teacher and student networks, as well as the methodology employed for knowledge transfer between them. This approach is evaluated in Sec. IV using the training hyperparameters outlined on the DDD17 and DSEC datasets described in Sec. IV-A and Sec. IV-B, respectively. For a deeper understanding of the effectiveness of each component of the approach, an ablation study is conducted on DDD17 in Sec. IV-C. Besides, an analysis of the semantic segmentation networks' sparsity with and without knowledge distillation is presented in Sec. 4. Sec. V concludes this paper and draws the perspectives of this work.

## II. RELATED WORK

### A. Event-based Semantic Segmentation

Event-based cameras have shown great potential in robotics and computer vision, particularly in challenging scenarios where traditional cameras falter. Researchers have explored various approaches to process events for semantic segmentation tasks. Alonso et al. [9] employed an Xception-based encoder-decoder architecture with a 6-channel input structure, achieving remarkable performance, especially in situations of excessive exposure. Gehrig et al. [10] improved upon this work by augmenting training data with synthetic events from video sources. Subsequent studies by Wang et al. [11], Sun et al. [12], and Wang et al. [13] explored event-to-image transformation, unsupervised domain adaptation, and knowledge distillation, respectively. Hybrid networks combining SNN and ANN components have also been proposed, such as those by Zhang et al. [14] and Das et al. [15]. Notably, Kim et al. [16] pioneered the use of SNNs exclusively, developing an

encoder-decoder architecture with skip connections and batch normalization through time. Building on this, Hareb et al. [17] proposed an improved fully spiking network, while Zhang et al. [18] introduced very recently a spike-based hierarchical search method to optimize the encoder architecture for both event streams and event streams-image fusions.

### B. Knowledge Distillation in SNN

Knowledge distillation, a model compression technique where a smaller "student" model learns from a larger "teacher" model, has seen limited application in the SNN domain. This method typically employs additional loss functions beyond the standard prediction and ground-truth loss, focusing on final outputs (response-based), intermediate outputs (feature-based), or their relationships (relation-based). In the context of SNNs, Kushawaha et al. [19] distilled spikes from a larger SNN to a smaller one, while Guo et al. [20] addressed gradient issues using online knowledge distillation. Both studies applied these techniques to single-modality classification tasks using RGB images.

Our approach advances event-based semantic segmentation by combining the strengths of SNNs and cross-modal knowledge distillation. Unlike previous ANN-based or hybrid methods, we use a fully spiking network while incorporating frame-based information during training to leverage the complementary strengths of both event-based and frame-based cameras. We transfer knowledge from a network processing fused grey-scale images and event data with the ADN network to an event-only SNN by integrating feature- and response-based KD techniques. This method differs from earlier SNN knowledge distillation works by focusing on cross-modal transfer in semantic segmentation rather than single-modality classification. By leveraging multi-modal data in training while maintaining efficient event-only processing during inference, our approach uniquely balances high performance with the inherent advantages of SNNs for event data processing.

## III. METHOD

In this section, we introduce the preprocessing applied to the events. Then, we describe the network and the loss functions designed for applying cross-modal distillation between the ANN-based teacher and the SNN-based student.

### A. Input Pre-processing

We convert an event stream initially represented in the Address Event Representation (AER) format, denoted as $(x, y, t, p)$, into voxel grids following the method introduced by [21]. These voxel grids are constructed from non-overlapping windows capturing $N$ events within intervals of $50ms$ or $250ms$, divided into $T$ timesteps. Then, the time dimension within each resulting event volume, denoted as $\Delta T$, is discretized into $B$ bins by scaling the timestamps to the range $[0, B-1]$ and applying a bilinear sampling kernel $k_b(a)$ as follows:

$$V(x, y, t) = \sum_{i=1}^{i=N} p_i k_b(x - x_i) k_b(y - y_i) k_b(t - t_i^*) \quad (1)$$
<figure><img src="image_3.png" /><figcaption>Fig. 1. Network Overview: A. Teacher: ADN-fused event and image features (Fig. 2). B. Knowledge distillation sub-networks. C. Student: Event processing enhanced by distilled knowledge. Before each classifier, an Upsampling operation is applied</figcaption></figure>

$$t_i^* = (B - 1)(t_i - t_0) / \Delta T \quad (2)$$

$$k_b(a) = \max[0, 1 - |a|] \quad (3)$$

Finally, we obtain a structure with dimensions $T \times B \times W \times H$, where $W$ and $H$ represent the input resolution. This event format serves as input for both the teacher and student networks. The key difference lies in their processing: SNN (student): Processes temporal information by forwarding the $B \times W \times H$ input $T$ times before backpropagation, while ANN (teacher): Processes the entire $T \times B \times W \times H$ structure at once.

## B. Cross Modal Knowledge Distillation

Our approach targets the knowledge transfer from a teacher network fed with grey-scale images fused with events, to a student network fed solely with events. Fig. 1 shows an overview of our method which is described below.

The teacher network, shown in Fig. 1.A, adopts a non-spiking PSPNet architecture based on ResNet18/ResNet10, initially proposed by Zhao et al. [22], known for its outstanding performances in semantic segmentation tasks across diverse datasets such as PASCAL VOC 2012 [23] and Cityscapes [24]. The teacher network receives input from a feature map generated by an ANN-based Dual-path Network. This network's architecture draws inspiration from the Spiking Spatially-Adaptive Modulation (SSAM) module [18] but uses non-spiking neurons instead of spiking ones. As illustrated in Fig.2, the architecture consists of two main components: i) A $3 \times 3$ convolution layer with a Batch Normalization (BN)

<figure><img src="image_4.png" /><figcaption>Fig. 2. ADN overview</figcaption></figure>

and ReLU activation is applied to a grey-scale image followed by a multi-scale dilated convolution operation with a BN and ReLU performed in parallel. ii) A $1 \times 1$ convolution layer with BN and ReLU is applied to the $T \times B \times H \times W$ voxel grid generated after concatenating the event windows.

The student network, represented in Fig. 1.C, is a fully spiking network using ResNet18/ResNet10 as backbones, incorporating Spike-Element-Wise (SEW) blocks proposed by Fang et al. [25]. Unlike basic Spiking ResNet blocks that add output feature maps to the identity before the spiking activation, SEW blocks perform this addition between output spikes and the identity. This approach enables residual learning in deep SNNs while mitigating exploding/vanishing gradient issues. Post-backbone, the network applies two sequential convolutions with batch normalization and spiking neurons. A spike accumulator then sums spike voltages across timesteps, followed by upsampling. Unlike the teacher, the PSPNet is not utilized to preserve the fully-spiking characteristic of our
SNN, crucial for implementation on dedicated neuromorphic hardware such as Loihi [26], SpiNNaker [27] or SPLEAT [28], [29], a neuromorphic architecture designed specifically for deploying SNNs on ASIC and FPGA hardware. We employ the rectangular function [30] as surrogate function, defined as follows:

$$f(u) = \frac{1}{\gamma} \mathrm{sign} \left( |u - v_{th}| < \frac{\gamma}{2} \right) \quad (4)$$

where $u$ is the neuron membrane potential, $\gamma$ is a hyper-parameter to control the surrogate gradient's width and sharpness, and $v_{th}$ is the threshold.

**Losses and Optimization:** Inspired by the approach outlined in [20], we incorporate additional branches used only during training after each stage (the first 3) of the ResNet architecture. Each branch comprises three sequential layers: Convolution, BN, and an activation function. We utilize a ReLU activation function for the teacher network, while we employ a LIF neuron for the student network. Subsequently, a head layer follows, consisting of a convolutional layer instead of a fully connected layer. This adaptation aims to tailor the network for semantic segmentation tasks rather than classification. The output channels of the head layer correspond to the number of classes within the dataset targeted for segmentation. These additional branches serve for the application of feature-based and response-based KD techniques by enabling the transfer of intermediate knowledge from the teacher to the student network through the following loss functions:

1) The Kullback-Leibler Divergence (KLD) loss, originally proposed by [31], is utilized to quantify the dissimilarity between the probability distribution of the student outputs $y_{SNN}$ and a reference probability distribution, which corresponds to the teacher outputs $y_{ANN}$.

$$L_{KLD} = \sum_{i}^{N=4} KLD(y_{ANN,i}, y_{SNN,i}) \quad (5)$$

2) The L2 loss from hints is employed to minimize the disparity in magnitude norms between the features of the ANN and the SNN using the following:

$$L2\_norm = \sum_{i}^{N=4} \|y_{ANN,i} - y_{SNN,i}\|^2 \quad (6)$$

Given that these losses are computed not only between the three intermediate outputs of each branch but also between the final outputs of both networks, $N = 4$. Moreover, these branches contribute to intermediate semantic segmentation, assessed using Cross Entropy (CE), which is computed between the predicted semantic segmentation and the ground-truth labels, defined as:

$$L_{CE} = \sum_{i}^{N=4} CE(y_{SNN,i}, labels) \quad (7)$$

The overall loss function consists of a weighted sum of these three losses as follows:

$$L = (1 - \alpha)L_{CE} + \alpha L_{KLD} + \beta L2\_norm \quad (8)$$

where $\alpha$ and $\beta$ are hyper-parameters used to control the distillation loss.

## IV. EXPERIMENTS AND EVALUATION

During the offline knowledge distillation process, we train the teacher network by employing the CE loss function, with a batch size of 16, for 60 epochs, and a learning rate of 0.02 optimized with Adam. Then, we train the student network using the outputs generated by the pre-trained teacher. Throughout both training processes, we employ the Adam optimizer [32] with a learning rate initialized to 0.02, a batch size of 16 for DDD17 and 4 for DSEC, and conduct the training over 70 epochs. Following [20], the distillation hyper-parameters $\alpha$ and $\beta$ are configured to 0.1 and $1 \times 10^{-6}$, respectively. A hard reset is used for the SNN network.

### A. Evaluation on DDD17

The DAVIS Driving Dataset (DDD17) for semantic segmentation is specifically tailored for automotive scenarios, encompassing 12 hours of driving data captured with a DAVIS camera [1]. This dataset offers per-pixel aligned and temporally synchronized events alongside grey-scale frames. For ground-truth annotations, we rely on semantic pseudo-labels generated by Alonso et al. [9], obtained by pre-training the Xception network [33] on frames extracted from the DAVIS dataset. The provided labels merge several classes, resulting in six distinct classes: flat (road and pavement), background (construction and sky), object, vegetation, human, and vehicle. This dataset is divided into training and validation sets containing 15950 and 3890 grey-scale images, respectively, with dimensions of $260 \times 346$, cropped to $200 \times 346$. We construct two voxel grids as described in Sec. III-A, using events captured over $50ms$ and $250ms$ intervals, each divided into $T = 5$ timesteps, with spatial dimensions of $W = 346$ and $H = 260$. The number of bins is set to $B = 5$ for the $50ms$ voxel grid and $B = 10$ for the $250ms$ voxel grid. The ground truth used for training and evaluation is used at the same rate, corresponding to the accumulated intervals.

<table><caption>TABLE I SEMANTIC SEGMENTATION RESULTS ON DDD17 WITH (W/) AND WITHOUT (W/O) KNOWLEDGE DISTILLATION. RESNET10\* W/ KD (LAST ROW) IS TRAINED USING 0.002 AS A LEARNING RATE INSTEAD OF 0.02. $T$ REFERS TO THE TEACHER BACKBONE.</caption><thead><tr><th colspan="4">Interval = 250ms, B = 10</th></tr><tr><th>Network</th><th>MIoU w/o KD</th><th>MIoU w/ KD</th><th></th></tr></thead><tbody><tr><td>ResNet18</td><td>47.43%</td><td><b>53.90%</b> (+6.47), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10</td><td>49.40%</td><td><b>52.60%</b> (+3.20), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10*</td><td>49.40%</td><td><b>52.57%</b> (+3.17), <i>T</i> = ResNet10</td><td></td></tr></tbody><thead><tr><th colspan="4">Interval = 50ms, B = 5</th></tr><tr><th>Network</th><th>MIoU w/o KD</th><th>MIoU w/ KD</th><th></th></tr></thead><tbody><tr><td>ResNet18</td><td>47.81%</td><td><b>50.90%</b> (+3.09), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10</td><td>46.70%</td><td><b>50.48%</b> (+3.78), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10*</td><td>46.70%</td><td><b>49.12%</b> (+2.42), <i>T</i> = ResNet10</td><td></td></tr></tbody></table>
**Network with and without CM KD comparison:** Our main results are presented in Tab.I. The first column indicates the backbone employed in the student network, while the second column displays the MIOU achieved during the validation process without knowledge distillation. The third column is split into two sections: the backbone used for the teacher model and the corresponding MIOU of the student obtained through KD. These results are reported across two experiments conducted with different voxel grids. Our analysis reveals consistent performance improvements when employing knowledge distillation with cross-modality data across various backbones and event intervals (see Fig. 3 for qualitative results). This underscores that the performance gains are not solely attributable to knowledge distillation as a technique, but specifically to the cross-modal knowledge transfer. This crucial distinction is thoroughly examined and confirmed in our ablation study, detailed in section IV-C, where we demonstrate that a mono-modality approach using only grey-scale images for knowledge transfer does not achieve comparable results. Furthermore, we observe that larger teacher networks yield superior performance gains for the student during knowledge distillation and longer event intervals lead to a substantial increase in learned knowledge, reaching up to 6%. This last observation may be attributed to the denser voxel grid obtained with a 250ms interval, averaging 100K events per sample in the validation set compared to 20K events with a 50ms interval, which is more conducive for distillation with a teacher model fed with dense inputs.

By taking a closer look at the class-wise MIOU results summarized in Tab.II, we observe that the proposed approach, leveraging knowledge distillation, enhances the pixel classification across the six distinct classes. Particularly notable are the classes whose MIOU falls below 50% without KD. This highlights the value of integrating grey-scale image knowledge when event data alone is insufficient, thus leveraging the complementarity of both modalities. Moreover, while comparing the class-wise MIOU between the different intervals, we notice that the larger the interval of the captured events, the better

<table><caption>TABLE III
AVERAGE NUMBER OF EVENTS PER SAMPLE IN THE VALIDATION SET CORRESPONDING TO EACH CLASS WITHIN TIME INTERVALS OF 50ms AND 250ms FOR DDD17.</caption><thead><tr><th>Interval</th><th>Back</th><th>Flat</th><th>Vehicle</th><th>Vegetation</th><th>Human</th><th>Object</th></tr></thead><tbody><tr><td>250ms</td><td>50637</td><td>2494</td><td>7812</td><td>15476</td><td>767</td><td>464</td></tr><tr><td>50ms</td><td>9838<br>(19%)</td><td>386<br>(15%)</td><td>1769<br>(22%)</td><td>3160<br>(20%)</td><td>220<br>(28%)</td><td>166<br>(35%)</td></tr></tbody></table>

<table><caption>TABLE IV
AVERAGE NUMBER OF PIXELS PER SAMPLE IN THE VALIDATION SET PER CLASS IN DDD17.</caption><thead><tr><th>Interval</th><th>Back</th><th>Flat</th><th>Vehicle</th><th>Vegetation</th><th>Human</th><th>Object</th></tr></thead><tbody><tr><td>#Pixels</td><td>47.22K</td><td>7.49K</td><td>3.36K</td><td>10.55K</td><td>87</td><td>476</td></tr></tbody></table>

the MIOU. This can be explained by examining the number of events corresponding to each class as outlined in Tab.III. Specifically, the number of events decreases by a factor of 5 for the interval of 50ms, compared to the 250ms. However, exceptions are observed for the "Human" and "Object" classes, where despite an increase in the number of events with longer intervals, the average number of pixels in the validation set (refer to Tab.IV) remains very low, at least ×10 less than other classes. This disparity in pixel count could explain why, even after transferring knowledge from grey-scale images, there is no improvement in the MIOU.

**State-of-the-art comparison:** Tab.V compares our approach with SOTA ANN, hybrid (SNN+ANN), and SNN networks processing 50ms event intervals, except for [18] which uses a continuous stream of 4 Stacking Based on Time (SBTs) [34] (200ms total). Our method outperforms existing SNN-based networks [16] and [17] for 50ms intervals and surpasses [18] when using a 250ms interval, establishing itself as the new benchmark. However, event-frame fusion approaches like SpikeEDN [18] and Halsie [15] show performance gains of 22.09% and 10.18% respectively over our event-only method.

<table><caption>TABLE II
CLASS-WISE MIOU RESULTS USING VOXEL GRIDS CONSTRUCTED WITHIN TIME INTERVALS OF 50ms AND 250ms FOR DDD17.</caption><thead><tr><th colspan="6">Interval= 250ms</th></tr><tr><th></th><th>Background</th><th>Flat</th><th>Vehicle</th><th>Vegetation</th><th>Human</th><th>Object</th></tr></thead><tbody><tr><td>w/o KD</td><td>87.55%</td><td>73.27%</td><td>48.76%</td><td>43.30%</td><td>29.78%</td><td>1.96%</td></tr><tr><td>w/ KD</td><td>88.90%<br>(+1.35)</td><td>75.36%<br>(+2.09)</td><td>57.66%<br>(+8.9)</td><td>49.27%<br>(+5.97)</td><td>40.23%<br>(+10.75)</td><td>5.20%<br>(+3.24)</td></tr></tbody><thead><tr><th colspan="6">Interval= 50ms</th></tr><tr><th></th><th>Background</th><th>Flat</th><th>Vehicle</th><th>Vegetation</th><th>Human</th><th>Object</th></tr></thead><tbody><tr><td>w/o KD</td><td>86.90%</td><td>69.17%</td><td>42.95%</td><td>39.78%</td><td>44.03%</td><td>4.08%</td></tr><tr><td>w/ KD</td><td>88.35%<br>(+1.45)</td><td>74.34%<br>(+5.17)</td><td>51.02%<br>(+8.07)</td><td>43.03%<br>(+3.25)</td><td>42.78%<br>(-1.25)</td><td>5.91%<br>(+1.83)</td></tr></tbody></table>

<table><caption>TABLE V
METHODS COMPARISON USING EVENTS (E) OR EVENTS AND FRAMES (E+F) DURING INFERENCE ON DDD17.</caption><thead><tr><th>Method</th><th>Type</th><th>Params [M]</th><th>Input</th><th>MIOU</th></tr></thead><tbody><tr><td>HALSIE [15]</td><td>Hybrid</td><td>1.82</td><td>E+F</td><td>60.66%</td></tr><tr><td>SpikingEDN [18]</td><td>SNN</td><td>8.62</td><td>E+F</td><td>72.57%</td></tr><tr><td>ESS [12]</td><td>ANN</td><td>12.91</td><td>E</td><td>61.37%</td></tr><tr><td>EvDistill [13]</td><td>ANN</td><td>29.09</td><td>E</td><td>54.81%</td></tr><tr><td>HALSIE [15]</td><td>Hybrid</td><td>1.82</td><td>E</td><td>46.09%</td></tr><tr><td>Zhang et al. [14]</td><td>Hybrid</td><td>1.90</td><td>E</td><td>51.42%</td></tr><tr><td>Deep-Lab [16]</td><td>SNN</td><td>4.14</td><td>E</td><td>33.70%</td></tr><tr><td>FCN [16]</td><td>SNN</td><td>13.60</td><td>E</td><td>34.20%</td></tr><tr><td>EvSegSNN [17]</td><td>SNN</td><td>8.55</td><td>E</td><td>45.54%</td></tr><tr><td>SpikingEDN [18]</td><td>SNN</td><td>8.6</td><td>E</td><td>53.15%</td></tr><tr><td>Ours (50ms)</td><td>SNN</td><td>6.54</td><td>E</td><td>50.48%</td></tr><tr><td>Ours (250ms)</td><td>SNN</td><td>14.79</td><td>E</td><td><strong>53.90%</strong></td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Fig. 3. Qualitative results on DDD17 Dataset. From left to right, we visualize the accumulated events into an interval of 50ms, its corresponding grey-scale images that fed the teacher, the ground-truth, the prediction of the network (ResNet18) without knowledge distillation and finally, the prediction of the same network used as a student in the proposed KD model.</figcaption></figure>

This disparity can be attributed to the integration of frame data during inference, as evidenced by the significant performance gap between event-only and event-frame fusion variants. Finally, our SNN-based method currently falls short of ANN [12] and hybrid [14] likely due to the inherent challenges in training SNNs, as discussed in the introduction. The performance gap observed in our approach using a 50ms interval with SpikeEDN can be attributed to three main factors. First, the use of a continuous stream of 4 SBTs. Second, the use Adaptive iterative LIF (AiLIF) to process input events. Third, the incorporation of a sophisticated encoder, whose architecture was determined through neural architecture search (NAS). This search process involved using cells and nodes while optimizing operations through training. Additionally, the gap may also stem from the integration of the Atrous Spatial Pyramid Pooling (ASPP) network between the encoder and decoder.

<table><caption>TABLE VI
SEMANTIC SEGMENTATION RESULTS ON DSEC-SEMANTIC WITH (W/) AND WITHOUT (W/O) KD. *T* REFERS TO THE TEACHER BACKBONE.</caption><thead><tr><th colspan="4">Interval = 250ms, B = 10</th></tr><tr><th>Network</th><th>MIoU w/o KD</th><th>MIoU w/ KD</th><th></th></tr></thead><tbody><tr><td>ResNet18</td><td>45.09%</td><td><strong>48.32%</strong> (+3.23), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10</td><td>46.06%</td><td><strong>48.66%</strong> (+2.60), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10</td><td>46.06%</td><td><strong>48.97%</strong> (+2.91), <i>T</i> = ResNet10</td><td></td></tr></tbody><thead><tr><th colspan="4">Interval = 50ms, B = 5</th></tr><tr><th>Network</th><th>MIoU w/o KD</th><th>MIoU w/ KD</th><th></th></tr></thead><tbody><tr><td>ResNet18</td><td>49.12%</td><td><strong>51.14%</strong> (+2.02), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10</td><td>49.45%</td><td><strong>51.82%</strong> (+2.37), <i>T</i> = ResNet18</td><td></td></tr><tr><td>ResNet10</td><td>49.45%</td><td><strong>50.18%</strong> (+0.73), <i>T</i> = ResNet10</td><td></td></tr></tbody></table>

## B. Evaluation on DSEC-semantic

The DSEC-semantic dataset, developed by Sun et al. [12] using Tao et al.'s hierarchical method [35] addresses label quality issues in DDD17. This dataset consists of 4021 training samples and 1397 validation samples, captured across urban and rural environments in Switzerland with automotive-grade standard cameras and high-resolution event cameras. It includes 11 classes: background, building, fence, person, pole, road, sidewalk, vegetation, car, wall, and traffic sign. The event stream is transformed into a voxel grid similar to those constructed for DDD17, with a resolution of 640 × 480, cropped to 640 × 440 due to limitations in frame-based camera capture of the bottom 40 rows.

**Network with and without CM KD comparison:** Based on the results in Tab.VI, integrating Cross-Modal KD during training improves performance during inference across different teacher and student backbone configurations. The degree of improvement varies depending on the intervals used for voxel grid construction. In contrast to DDD17 findings, shorter intervals yield more significant performance gains, regardless of KD usage. This trend may be influenced by the average number of events in the validation set, which is approximately 500K for 50ms and 2.5M for 250ms, potentially introducing a higher noise level into the data.

**State-of-the-art Comparison:** Our fully spiking network demonstrates competitive performance against event-based SOTA methods on the challenging DSEC dataset (Tab.VII). Despite the inherent difficulties in training fully spiking networks, our approach achieves results comparable to conventional ANN networks [12], which typically benefit from more established training techniques. This achievement under-
<table><caption>TABLE VII
METHODS COMPARISON USING EVENTS (E) OR EVENTS AND FRAMES (E+F) DURING INFERENCE ON DSEC.</caption><thead><tr><th>Method</th><th>Type</th><th>Params [M]</th><th>Input</th><th>MIoU</th></tr></thead><tbody><tr><td>HALSIE [15]</td><td>Hybrid</td><td>1.82</td><td>E+F</td><td>52.43%</td></tr><tr><td>SpikingEDN [14]SNN</td><td></td><td>23.09</td><td>E+F</td><td>58.32%</td></tr><tr><td>ESS [12]</td><td>ANN</td><td>12.91</td><td>E</td><td>53.29%</td></tr><tr><td>SpikingEDN [14]SNN</td><td></td><td>6.27</td><td>E</td><td><strong>53.17%</strong></td></tr><tr><td>Ours (50ms)</td><td>SNN</td><td>6.54</td><td>E</td><td>51.82%</td></tr><tr><td>Ours (250ms)</td><td>SNN</td><td>6.54</td><td>E</td><td>48.97%</td></tr></tbody></table>

scores the potential of SNNs in complex event-based vision tasks. Among SNN-based methods, the recently proposed SpikeEDN [18] outperforms our approach on DSEC, likely due to the same factors contributing to the performance gap on DDD17. Additionally, frame-events methods continue to show superior performance, echoing the trend observed in IV-A. Interestingly, the performance gap between our method and frame-events approaches is smaller on DSEC compared to the DDD17 dataset. This reduction can be attributed to DSEC's superior ground-truth quality and more diverse class set (11 classes versus DDD17's 6). The narrower gap on this more complex dataset highlights our method's robustness and scalability. It's worth noting that this comparison involves a limited number of works compared to DDD17's due to DSEC's recent introduction.

### C. Ablation Study

To highlight the importance of cross-modality over mono-modality used by the teacher, we conducted experiments on DDD17 using two setups: one with event data only and the other with grey-scale images only fed to the teacher model. In the first scenario, the teacher model was fed with event data, while in the second, it was fed with grey-scale images. In the first approach, the MIoU achieved by the teacher model did not surpass 54%, indicating inadequate performance for effective knowledge transfer to the student model, which achieved a MIoU of 49% without any knowledge transfer. In the second scenario, training the teacher model on grey-scale images resulted in a MIoU of 72%, a performance comparable to

that obtained with data (i.e. events and grey-scale images) fusion. The learned knowledge was then distilled to the student model. Detailed results of these experiments are presented in Tab.VIII. Although the student trained with a mono-modal teacher performed better than without KD, it still fell behind cross-modal KD, possibly due to inherent differences between images and voxel grid structures. Consequently, transferring the information from images to events may face limitations, particularly as the intermediate and final outputs transferred by the teacher differ from the ones generated by the student.

### D. Sparsity

Unlike ANNs, which rely on dense matrix multiplication, SNNs perform sparse computations driven by events. This characteristic suggests potential advantages for low-power computing, confirmed by measuring 2 metrics: i) The spike activity that quantifies the number of spikes emitted by LIF neurons during inference, with lower values indicating better energy efficiency [5]. ii) The network's spike rate which is obtained by dividing the total number of spikes by the number of neurons. To evaluate the impact of introducing knowledge distillation during SNN training, we examined these metrics on the same network with and without KD, utilizing DDD17 input voxel intervals as previously (50ms and 250ms). The results, depicted in Fig. 4, reveal variations of the spike activity across layers and ResNet stages. These variations are notably higher for all outputs of each stage, which can be attributed to the SEW blocks use. These blocks employ feature maps consisting of binary matrices that are summed between residual and direct connections, significantly reducing the sparsity (number of 0s) in the feature maps. Interestingly, although the spike activity tends to be slightly higher in the first two layers and the initial ResNet stages when employing KD, this pattern is reversed for subsequent stages and layers, especially for stage 4. This observation remains consistent regardless of the interval size used to construct the voxel grid. Moreover, when assessing the network's spike rate, we observe a spike rate of 26.02% for the network without KD and 21.46% with KD. This suggests that, besides improving performances in terms of MIoU, employing knowledge distillation reduces the power consumption of the SNN.

<table><caption>TABLE VIII
SEMANTIC SEGMENTATION RESULTS ON DDD17 WITHOUT (W/O) AND WITH (W/) CROSS-MODAL (CM) AND MONO-MODAL (MM) KD. THE TEACHER'S BACKBONE IS RESNET18.</caption><thead><tr><th colspan="4">Interval = 250ms, B = 10</th></tr><tr><th>Network</th><th>MIoU w/o KD</th><th>MIoU w/ CM</th><th>MIoU w/ MM</th></tr></thead><tbody><tr><td>ResNet18</td><td>47.43%</td><td><strong>53.90%</strong></td><td>50.80%</td></tr><tr><td>ResNet10</td><td>49.40%</td><td><strong>52.60%</strong></td><td>49.83%</td></tr></tbody><thead><tr><th colspan="4">Interval = 50ms, B = 5</th></tr><tr><th>Network</th><th>MIoU w/o KD</th><th>MIoU w/ CM</th><th>MIoU w/ MM</th></tr></thead><tbody><tr><td>ResNet18</td><td>47.81%</td><td><strong>50.90%</strong></td><td>50.64%</td></tr><tr><td>ResNet10</td><td>46.70%</td><td><strong>50.48%</strong></td><td>48.10%</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 4. Layer-wise sparsity of SNNs on DDD17. The blue and red bars represent w/o KD and w/ KD, respectively</figcaption></figure>
V. CONCLUSION

In this paper, we present a novel approach that efficiently fuses event-based and frame-based cameras to capitalize on their complementary advantages. By employing cross-modal knowledge distillation during training, our approach allows a spiking neural network that only processes event data to improve its performance with guidance from an artificial neural network (ANN) trained on both event and frame inputs. Our experiments on the DDD17 and DSEC datasets demonstrate the effectiveness of this method, showing consistent performance improvements across different event intervals when using knowledge distillation. Importantly, our approach sets a new standard on the DDD17 dataset compared to existing SNN-based event processing methods. On the DSEC dataset, we achieved second place, due to the innovative techniques of the top-performing method, which we plan to investigate further to enhance our work. Looking forward, we also intend to optimize our model for deployment on SPLEAT, which is designed for handling non-binary tensors filled with integer values, aligning well with our network's characteristics. This deployment will take advantage of the energy efficiency of SNNs, thanks to their fully event-driven nature and low spike rate.

ACKNOWLEDGMENT

This work was supported by the project NAMED (ANR-23-
CE45-0025-01) of the French National Research Agency.

This work was supported by the French government through
the France 2030 investment plan managed by the National
Research Agency (ANR), as part of the Initiative of Excellence
Université Côte d'Azur under reference number ANR-15-
IDEX-01.

This work is supported by a public grant overseen by the French National Research Agency (ANR) as part of the 'PEPR IA France 2030' programme (Emergences project ANR-23-PEIA-0002).

The authors are grateful to the OPAL infrastructure from
Université Côte d'Azur for providing resources and support.

REFERENCES

[1] C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbrück, “A 240× 180 130 db 3 µs latency global shutter spatiotemporal vision sensor,” *IEEE Journal of Solid-State Circuits*, vol. 49, no. 10, pp. 2333–2341, 2014.

[2] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis et al., “Event-based vision: A survey,” *IEEE transactions on pattern analysis and machine intelligence*, vol. 44, no. 1, pp. 154–180, 2020.

[3] W. Maass, “Networks of spiking neurons: the third generation of neural network models,” *Neural networks*, vol. 10, no. 9, pp. 1659–1671, 1997.

[4] M. A. Nahmias, B. J. Shastri, A. N. Tait, and P. R. Prucnal, “A leaky integrate-and-fire laser neuron for ultrafast cognitive computing,” *IEEE journal of selected topics in quantum electronics*, vol. 19, no. 5, pp. 1–12, 2013.

[5] E. Lemaire, L. Cordone, A. Castagnetti, P.-E. Novac, J. Courtois, and B. Miramond, “An analytical estimation of spiking neural networks energy efficiency,” in *International Conference on Neural Information Processing*. Springer, 2022, pp. 574–587.

[6] L. Khacef, N. Abderrahmane, and B. Miramond, “Confronting machine-
learning with neuroscience for neuromorphic architectures design,” in
*2018 International Joint Conference on Neural Networks (IJCNN)*.
IEEE, 2018, pp. 1–8.

[7] M. Dampfhoffer, T. Mesquida, E. Hardy, A. Valentian, and L. Anghel, “Leveraging sparsity with spiking recurrent neural networks for energy-efficient keyword spotting,” in *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2023, pp. 1–5.

[8] E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” *IEEE Signal Processing Magazine*, vol. 36, no. 6, pp. 51–63, 2019.

[9] I. Alonso and A. C. Murillo, “Ev-segnet: Semantic segmentation for event-based cameras,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, 2019, pp. 0–0.

[10] D. Gehrig, M. Gehrig, J. Hidalgo-Carrió, and D. Scaramuzza, “Video to events: Recycling video datasets for event cameras,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2020, pp. 3586–3595.

[11] L. Wang, Y. Chae, and K.-J. Yoon, “Dual transfer learning for event-based end-task prediction via pluggable event to image translation,” in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2021, pp. 2135–2145.

[12] Z. Sun, N. Messikommer, D. Gehrig, and D. Scaramuzza, “Ess: Learning event-based semantic segmentation from still images,” in *European Conference on Computer Vision*. Springer, 2022, pp. 341–357.

[13] L. Wang, Y. Chae, S.-H. Yoon, T.-K. Kim, and K.-J. Yoon, “Evdistill: Asynchronous events to end-task learning via bidirectional reconstruction-guided cross-modal knowledge distillation,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2021, pp. 608–619.

[14] H. Zhang, X. Fan, and Y. Zhang, “Energy-efficient spiking segmenter for frame and event-based images,” *Biomimetics*, vol. 8, no. 4, p. 356, 2023.

[15] S. Das Biswas, A. Kosta, C. Liyanagedera, M. Apolinario, and K. Roy, “Halsie: Hybrid approach to learning segmentation by simultaneously exploiting image and event modalities,” in *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, 2024, pp. 5964–5974.

[16] Y. Kim, J. Chough, and P. Panda, “Beyond classification: Directly training spiking neural networks for semantic segmentation,” *Neuromorphic Computing and Engineering*, vol. 2, no. 4, p. 044015, 2022.

[17] D. Hareb and J. Martinet, “Evsegsnn: Neuromorphic semantic segmentation for event data,” *arXiv preprint arXiv:2406.14178*, 2024.

[18] R. Zhang, L. Leng, K. Che, H. Zhang, J. Cheng, Q. Guo, J. Liao, and R. Cheng, “Accurate and efficient event-based semantic segmentation using adaptive spiking encoder–decoder network,” *IEEE Transactions on Neural Networks and Learning Systems*, pp. 1–15, 2024.

[19] R. K. Kushawaha, S. Kumar, B. Banerjee, and R. Velmurugan, “Distilling spikes: Knowledge distillation in spiking neural networks,” in *2020 25th International Conference on Pattern Recognition (ICPR)*. IEEE, 2021, pp. 4536–4543.

[20] Y. Guo, W. Peng, Y. Chen, L. Zhang, X. Liu, X. Huang, and Z. Ma, “Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization,” *Pattern Recognition*, vol. 142, p. 109639, 2023.

[21] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “Unsupervised event-based learning of optical flow, depth, and egomotion,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2019, pp. 989–997.

[22] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 2881–2890.

[23] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” *International journal of computer vision*, vol. 88, pp. 303–338, 2010.

[24] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2016, pp. 3213–3223.
[25] W. Fang, Z. Yu, Y. Chen, T. Huang, T. Masquelier, and Y. Tian, "Deep residual learning in spiking neural networks," *Advances in Neural Information Processing Systems*, vol. 34, pp. 2156–2169, 2021.

[26] M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain et al., "Loihi: A neuromorphic manycore processor with on-chip learning," *Ieee Micro*, vol. 38, no. 1, pp. 82–99, 2018.

[27] S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, "The spinnaker project," *Proceedings of the IEEE*, vol. 102, no. 5, pp. 652–665, 2014.

[28] E. Lemaire, M. Moretti, L. Daniel, B. Miramond, P. Millet, F. Feresin, and S. Bilavarn, "An fpga-based hybrid neural network accelerator for embedded satellite image classification," in *2020 IEEE International Symposium on Circuits and Systems (ISCAS)*, 2020.

[29] N. Abderrahmane, B. Miramond, E. Kervennic, and A. Girard, "Spleat: Spiking low-power event-based architecture for in-orbit processing of satellite imagery," in *2022 International Joint Conference on Neural Networks (IJCNN)*. IEEE, 2022, pp. 1–10.

[30] Y. Wu, L. Deng, G. Li, J. Zhu, Y. Xie, and L. Shi, "Direct training for spiking neural networks: Faster, larger, better," in *Proceedings of the AAAI conference on artificial intelligence*, vol. 33, no. 01, 2019, pp. 1311–1318.

[31] G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," *arXiv preprint arXiv:1503.02531*, 2015.

[32] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv preprint arXiv:1412.6980*, 2014.

[33] F. Chollet, "Xception: Deep learning with depthwise separable convolutions," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 1251–1258.

[34] L. Wang, Y.-S. Ho, K.-J. Yoon et al., "Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2019, pp. 1081–1090.

[35] A. Tao, K. Sapra, and B. Catanzaro, "Hierarchical multi-scale attention for semantic segmentation," *arXiv preprint arXiv:2005.10821*, 2020.