# RGB-Event Fusion for Moving Object Detection in Autonomous Driving

Zhuyun Zhou, Zongwei Wu, Rémi Boutteau, Fan Yang, Cédric Demonceaux, Dominique Ginhac

► To cite this version:

Zhuyun Zhou, Zongwei Wu, Rémi Boutteau, Fan Yang, Cédric Demonceaux, et al. RGB-Event Fusion for Moving Object Detection in Autonomous Driving. ICRA International Conference on Robotics and Automation 2023, IEEE, May 2023, London, United Kingdom. 10.1109/ICRA48891.2023.10161563. hal-04048853

HAL Id: hal-04048853

https://hal.science/hal-04048853

Submitted on 28 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
RGB-Event Fusion for Moving Object Detection in Autonomous Driving

Zhuyun Zhou¹, Zongwei Wu*¹,², Rémi Boutteau³, Fan Yang¹, Cédric Demonceaux¹,⁴, Dominique Ginhac¹

**Abstract**—Moving Object Detection (MOD) is a critical vision task for successfully achieving safe autonomous driving. Despite plausible results of deep learning methods, most existing approaches are only frame-based and may fail to reach reasonable performance when dealing with dynamic traffic participants. Recent advances in sensor technologies, especially the Event camera, can naturally complement the conventional camera approach to better model moving objects. However, event-based works often adopt a pre-defined time window for event representation, and simply integrate it to estimate image intensities from events, neglecting much of the rich temporal information from the available asynchronous events. Therefore, from a new perspective, we propose RENet, a novel RGB-Event fusion Network, that jointly exploits the two complementary modalities to achieve more robust MOD under challenging scenarios for autonomous driving. Specifically, we first design a temporal multi-scale aggregation module to fully leverage event frames from both the RGB exposure time and larger intervals. Then we introduce a bi-directional fusion module to attentively calibrate and fuse multi-modal features. To evaluate the performance of our network, we carefully select and annotate a sub-MOD dataset from the commonly used DSEC dataset. Extensive experiments demonstrate that our proposed method performs significantly better than the state-of-the-art RGB-Event fusion alternatives. The source code and dataset are publicly available at: https://github.com/ZZY-Zhou/RENet.

# I. INTRODUCTION

With the development of deep learning methods, researchers have reported great success in modeling the motion behaviors of different traffic participants [1]–[3]. However, most existing methods are frame-based and their performances are highly dependent on video quality, i.e., the frame rate and the image quality [4]–[6]. The former (frame rate) is essential to model the temporal consistency and the motion behavior, while the latter (image quality) influences the accuracy of object detection within a single frame. Therefore, despite significant advances in deep networks, the inherent limits of frame-based cameras are one of the main performance bottlenecks for moving object detection.

Recent advances in bio-inspired event cameras have drawn great research attention for autonomous vehicles [7]–[9]. The

<figure><img src="image_2.png" /><figcaption>Fig. 1: Comparison with RGB-Event Fusion Methods. (1) Most existing works take a single-range event as input [14]–[17] with a single-directional fusion design [15], [18], which neglects the rich temporal cues and cannot thoroughly explore the cross-modal relationship. (2) Our proposed network can benefit from multi-scale temporal events to better deal with object motion. We further adopt a bi-directional fusion design to better model multi-modal features and attentively form the shared representation.</figcaption></figure>

rich temporal information provided by events can help to better avoid collisions with moving traffic agents and thus yield safer autonomous driving. In fact, when dealing with complex lighting scenes, such as driving during night or passing through a tunnel, the performance of frame-based methods may degrade severely [10]–[13]. Meanwhile, event cameras are more robust in these adverse visual conditions since they can quickly adapt to light changes thanks to its low latency. Nevertheless, event data often provide wider object boundaries than RGB frames. This is mainly due to the synchronization between RGB and events since RGB frames are obtained during a short exposure time, while event cameras work in an asynchronous mode at a higher temporal resolution. Therefore, how to fully leverage the heterogeneous but mutually complementary modalities is yet an open research topic.

In the literature, a common practice [15], [18]–[20] is to gradually merge event features in the RGB stream to guide the feature modeling as shown in Fig. 1. Despite the achieved plausible results, these works are often uni-directional and cannot thoroughly leverage modality-specific cues during the encoding stages, i.e., RGB-only, event-only, and fused features. Moreover, most existing works choose a pre-defined time interval as a hyper-parameter for event representation, which yields two major shortages: first, the fixed event representation cannot adapt to different scenarios without handcraft adjusting; secondly, a small range of event intervals, such as the exposure time of RGB camera, cannot fully leverage the rich temporal cues from the event

This work was supported by the French National Research Agency through ANR CERBERE (ANR-21-CE22-0006).

¹Zhuyun Zhou, Zongwei Wu, Fan Yang, Cédric Demonceaux, and Dominique Ginhac are with ImViA, University of Burgundy (Université de Bourgogne), Dijon, France. {Zhuyun_Zhou@etu., Zongwei_Wu@etu., fanyang@, Cedric.Demonceaux@, dginhac@} u-bourgogne.fr

²Zongwei Wu is also with CVL, ETH Zurich.

³Rémi Boutteau is with Univ Rouen Normandie, LITIS, UR 4108, F-76000 Rouen, France. remi.boutteau@univ-rouen.fr

⁴Cédric Demonceaux is also with Université de Lorraine, CNRS, Inria, LORIA, Nancy, France.

*Corresponding author: zongwei.wu.97@gmail.com
data, while a long range of event intervals adds significant misaligned response.

To address the aforementioned issues, in this paper, we propose a novel RGB-Event network, named RENet, for moving object detection. Different from previous works basing on single-range events, as shown in Fig. 1, our event stream takes multi-range events as inputs to better benefit from the rich temporal cues. Specifically, for each event representation, we first apply convolutions to project the input event maps to a latent space. We then apply pooling with different kernel sizes to deal with events from different time ranges, i.e., larger pooling windows for longer-range events. Our motivation comes from the observation that the event frames of long-time ranges introduce local motion blur around the object boundaries. The motion blurs are essential for the detection of moving objects, while being not well-aligned with the image frame obtained during the exposure time. Therefore, we apply different sizes of max-pooling to preserve the most informative motion features and improve the scale invariance within a local region, yielding a simple yet efficient way to aggregate rich temporal events in a coarse-to-fine manner.

Once the multi-range events are fused in the latent space, we feed them through the dual-residual networks along with RGB features. To attentively integrate multi-modal features, we introduce a bi-directional calibration module that first improves each input feature by attending to the channel and the spatial dimensions of the input features and then realizes a cross-modal calibration. Finally, the cross-calibrated features are merged and further integrated with the hierarchical features, if any, to form the shared output. To the best of our knowledge, we are the first to apply attention modules for RGB-Event (moving) object detection tasks. Additionally, we provide DSEC-MOD, a new dataset dedicated to moving object detection from the widely used RGB-Event dataset DSEC [21]. DSEC-MOD contains moving objects with automatically labeled and manually checked bounding boxes. Extensive comparisons performed on DSEC-MOD demonstrate the effectiveness of RENet against state-of-the-art RGB-Event fusion alternatives.

To summarize, our main contributions are three-fold:

* We design a multi-range aggregation module to fully leverage the rich temporal information from events, which are crucial for moving object detection.

* We introduce a novel middle fusion design for RGB-Event fusion that models both modality-specific and shared representations.

* We propose a novel DSEC-MOD dataset with automatically labeled and manually verified annotation to encourage the development of moving object detection with events. We benchmark various state-of-the-art (SOTA) fusion modules on our dataset and show that RENet significantly outperforms the fusion alternatives.

## II. RELATED WORK

**Event Datasets:** Unlike traditional datasets for global shutter RGB cameras, the number of available public event datasets

is much fewer, and the types of tasks that may be practiced with these datasets are also limited. For example, several datasets [21]–[24] only provide depth information as ground truth, including the two most widely used DSEC [21] and MVSEC [24]. To encourage the development of segmentation and detection tasks for autonomous driving, recent researches propose to induce from these two datasets and create sub-datasets for specific tasks. For example, MVSEC-NIGHT [22] induces from MVSEC and generates car annotations with the help of the pre-trained YOLO-v3 [25] for night scenarios. Sharing a similar motivation, [15] uses the pre-trained YOLO-v5 [26] to create the bounding boxes for objects from the DSEC dataset. There are also several other datasets targeting segmentation tasks, such as the DET dataset [27] which provides annotation for lane detection. However, these works focus more on 2D segmentation, with few works taking the temporal consistency and object motion into account. For autonomous driving, understanding object motions is crucial for driving safety. Therefore, we propose an RGB-Event dataset, called DSEC-MOD, to encourage further research on moving object detection.

**Event Processing:** Event cameras have recently drawn great attention thanks to their asynchronous characteristic. As bio-inspired sensors, several event-by-event researches aim to develop novel spiking neural network [28]–[30] to process this novel sensor information [31], [32]. However, there are few works dealing with RGB-Event inputs, especially for multi-modal sensor fusion.

From another perspective, events can be naturally considered as 4D inputs (x, y, p, t), where (x, y) stands for the spatial resolution as that of images, p stands for the polarity, and t stands for the temporal axis. Until nowadays, there is no conventional event representation for video applications as suggested in previous works [18], [33]. In the literature, there are three popular representations: frame-like methods [34]–[37] that focus more on dealing with (x, y, p) information, time-surface methods [38]–[40] that focus more on (x, y, t) cues, and voxel-based methods [19], [41]–[43] that deal with all dimensions at the same time (x, y, p, t). While voxel-based methods can fully explore both space-time information, its 4D structure requires higher computational cost compared to frame-based and time-surface methods, which are not suitable for real-time applications such as autonomous driving. In this paper, we propose a novel event representation for moving object detection. We build upon frame-based approaches and additionally integrate the time information through a temporal multi-scale aggregation, resulting in a simple yet effective manner to explore spatio-temporal cues.

**RGB-Event Fusion:** With the development of event cameras, RGB-Event multi-modal fusion has drawn increasing research attention. Some works adopt a pre-trained network to realize event-frame conversion. Then, several works directly process the generated video in the same way as dealing with RGB video to achieve the target application. A recent work [44] applies knowledge distillation to guide the learning on videos generated from events. However, we argue that there is a significant difference between knowledge
<figure><img src="image_3.png" /><figcaption>Fig. 2: **Architecture.** Our proposed network consists of an Event-based Temporal Multi-scale Aggregation (E-TMA see Sec. III-A) and an attentive fusion with Bi-Directional Calibration (BDC see Sec. III-C). E-TMA aims to fully benefit from the temporal cues to improve the feature modeling, while BDC leverages cross-modal cues to firstly calibrate the noisy responses and attentively fuse them. The decoder and loss are adapted from [46].</figcaption></figure>

distillation and multi-modal fusion. Other works take RGB and events from the input side and realize features during the feature modeling. [16], [45] proposes a late-fusion design to integrate events into RGB streaming to guide the feature decoding. Meanwhile, [15] adopts a middle-fusion design through simple concatenation convolution. [18] further leverages the transformer's attention to better guide the feature fusion. Sharing a similar idea, we also follow the middle-fusion design to fuse RGB and events. Different from [15], [18] with uni-directional fusion design, as shown in Fig. 1, we explicitly model both modality-specific features and shared representation. Furthermore, we introduce a cross-modal calibration module to attentively correct the noisy response in each modality before feature fusion. Different from [18] which focuses on the channel axis, our attention module further attends to the spatial dimension, which plays an important role for object localization within a frame.

### III. OUR WORK

Fig. 2 presents the overall framework of our network RENet. For simplicity, we only illustrate the case for single-frame RGB-Event inputs. In practice, inputs use several frames from video as well as the associated events.

Our network is composed of a tailored stemming layer to merge temporal multi-scale events (E-TMA see Sec. III-A), a dual encoder network to extract features, a bi-directional calibration module to firstly improve and then generate the shared representation (BDC see Sec. III-C), and a decoder to output the bounding box of moving objects. Specifically, the multi-scale events are first fed into our proposed E-TMA to fully leverage the rich temporal motion cues. Then, we adopt a dual but discrepant residual streaming design to extract semantic maps from each modality. To realize the multi-modal fusion, we attend to both the channel and spatial dimensions of each modality and then enable a cross-modal calibration. This process aims to neglect inherent single-modal noisy response and improve the feature modeling. Then, the enhanced multi-modal features together with the hierarchical features are fused in a coarse-to-fine manner.

Finally, the encoded features are fed into the MOD detector [46] to produce the bounding box. The details will be introduced in the following sections.

#### A. E-TMA: Event-based Temporal Multi-scale Aggregation

We observe from Fig. 3 that objects can slightly move from one position to another, i.e., there are more events on object boundaries while the temporal range of events becomes larger. However, the position changes are not significant nor sudden. Inspired by this observation, we design an aggregation module to establish the temporal relationship between multi-range event features $E'_i$. The aggregation must have two properties. First, it must maintain the homogeneous semantic cues across different ranges since the overall environment remains the same. Second, it must also be sensitive to spatially and temporally moving positions. Most existing works often process events within a pre-defined range, neglecting rich dynamic characteristics of event cameras.

To address this issue, we propose a multi-scale event aggregation module. Here, the term **Scale** does **Not** stand for the spatial resolution as conventionally used in image-based tasks. Instead, it stands for the different **Temporal** ranges widely used in the signal processing domain. Mathematically, during a specific temporal pool $T = \{T_i | i \in N^*\}$ under the rule that $T_i \le T_{i+k}$ with $k \in N$, the groups of events $\epsilon_i$ are generated respectively. In our case, we take three scales of events as inputs, i.e, the events obtained from the RGB frame exposure time ($\epsilon_1$), the events obtained from an intermediate scale which is double the exposure time in our case ($\epsilon_2$), and the events obtained during one RGB frame time ($\epsilon_3$). To deal with multi-scale events, we first project the input events into a latent feature space. The projection module, denoted as $\eta$, is similar to the first VGG layer which is composed of a combination of Convolution, Batch Normalization, and Relu activation. Each scale of events is fed into the same projection module to generate the event feature map. In other words, the projection weights are shared for each scale event input. The projection step can be expressed as:

$$E'_i = \eta(\epsilon_i), i \in 1, 2, 3. \quad (1)$$
<figure><img src="image_4.png" /><figcaption>Fig. 3: E-TMA: Event-based Temporal Multi-scale Aggregation. Input events are first projected into a latent space. Then, different sizes of pooling are applied according to the event range. Finally, event features are merged together to learn the shared output with enhanced temporal consistency.</figcaption></figure>

To model the objects moving within a local region, pooling operations with different window sizes are applied to the event features: features from long-range events are processed with coarser pooling, while features from small-range events are processed with finer pooling. Formally, let the pooling with different window sizes $pool_i = \{pool_1; pool_2; pool_3|size訊 pool_1) < size訊 pool_2) < size訊 pool_3)\}$, we apply max pooling on top multi-scale event features $E'_i$:

$$e'_i = pool_i(E'_i), i \in 1, 2, 3. \qquad (2)$$

Note that due to the different sizes of the pooling kernels, the pooled features do not share the same resolution. To merge them together, upsampling is applied to smaller-shape features to adjust the resolution. Finally, all features are concatenated together and fed into a convolution to generate the event representation $f_e$. We have:

$$f_e = Conv(Concat(e'_1; up(e'_2); up(e'_3))), \qquad (3)$$

where up denotes the upsampling to match the resolution. The aggregated feature is later fed into the residual encoder [47] for semantic modeling.

Our event modeling is significantly different from previous works. Several works simply take a pre-defined range of events as input [15], [16], which cannot fully leverage the rich temporal cues. A recent concurrent work [18] also proposes to deal with multi-scale events. However, they simply accumulate the polarities of multi-scale events from the input side. Despite its demonstrated effectiveness on image deblurring, this method fails to explicitly distinguish the object boundary and the background, yielding a noisy representation for object detection tasks. Differently, we propose to merge the multi-scale events at the latent space with the help of pooling operations. The choice of pooling is mainly based on its spatial scale and rotation invariant property which fits our application extremely well, as discussed previously. We particularly leverage the max pooling operation to extract the most informative cues in the event features. The motivation is based on the intuition

that the object boundary should provide a strong response to distinguish the traffic participant from the background. To the best of our knowledge, we are the first to introduce a learnable aggregation module to extract, preserve, and fuse the most informative event features across different temporal scales. We show in Table I the superiority of our proposed event representation approach over our counterparts.

## B. Discrepant Two-Streaming Encoder

Most existing multi-model models apply dual encoders for feature extraction. These encoders do not share weight but share the same architecture. However, we observe that RGB and event maps are significantly different, i.e., the former (RGB) provides dense textual information at each pixel, while the latter (event) only provides sparse activation for the pixels with brightness changes. Moreover, through our empirical experiments, as shown in Table I, we observe that event-only streaming yields significantly lower performance compared to RGB-only streaming. This observation motivates us to design a discrepant two-streaming encoder for feature extraction as shown in Fig. 2, i.e., a deeper encoder for RGB processing with ResNet-101 and a shallower encoder for event processing with Resnet-18. In such a way, we distinguish RGB and event modalities that the event stream plays an assist and auxiliary role. This discrepant design can also help to reduce computational costs.

To realize the multi-modal fusion, RGB and event features must share the same size. However, due to the discrepant encoders, the channel size for our RGB and event features are not the same. Therefore, before realizing the feature fusion, we apply a projection on the encoded event features to match the channel size of RGB features.

## C. BDC: Bi-Directional Calibration

RGB-Event encoder fusion is a relatively novel research topic with very few papers compared to other multi-modal fusions such as RGB-Depth and RGB-Point cloud. Previous RGB-Event works or simply merge multi-modal features through simple concatenation convolution [15], or attend to the channel dimension with the help of transformer attention [18]. However, these methods do not explicitly tackle the spatial cues which are essential for segmentation/detection tasks. A recent tentative on knowledge transfer [44] has also shown great success for event processing. Nevertheless, we argue that there is a huge difference between multi-modal fusion and knowledge distillation where the latter cannot fully leverage the cross-modal features compared to the former. Therefore, we seek to design an attentive fusion that can leverage the informative cues from both spatial and channel axis during the encoding stage.

For simplicity, we take the fusion at the highest semantic level as an example. Formally, let the RGB feature $f_R \in \mathbb{R}^{C \times h \times w}$ and the projected event feature $f_E \in \mathbb{R}^{C \times h \times w}$. We first apply a transformation module based on $1 \times 1$ convolution for activation. We obtain the activated feature maps $f_r$ and $f_e$ by:

$$f_r = Conv_{1 \times 1}(f_R); \quad f_e = Conv_{1 \times 1}(f_E). \qquad (4)$$
Then we attentively fuse RGB and event features in a coarse-to-fine manner. Specifically, we first use a pixel-wise multiplication along with addition to coarsely enhance the most informative features in each modality. Formally, the enhanced features $f'_r$ and $f'_e$ are computed by:

$$f'_r = f_r \otimes f_e + f_r; \quad f'_e = f_r \otimes f_e + f_e. \quad (5)$$

Then we refine the features by explicitly attending to the channel and spatial axes in a separate and successive manner. For each axis, we enable a bi-directional calibration, i.e., we learn the features from one modality and apply it to the other. Then, we merge calibrated multi-modal features while maintaining the most informative features. For simplicity, we show the example with channel calibration. Let CA be the channel attention module from [48]. The cross-calibration along the channel axis can be formulated as:

$$f_r^{CA} = CA(f'_e) \otimes f'_r + f'_r; \quad f_e^{CA} = CA(f'_r) \otimes f'_e + f'_e. \quad (6)$$

Sharing the same protocol, we further refine the features spatially. Let SA be the spatial attention from [48], we obtain the final enhanced features $f_r^{enh}$ and $f_e^{enh}$ as follows:

$$f_r^{enh} = SA(f_e^{CA}) \otimes f_r^{CA} + f_r^{CA}; \quad f_e^{enh} = SA(f_r^{CA}) \otimes f_e^{CA} + f_e^{CA}. \quad (7)$$

These designs aim to find the most truthful and confident channels/pixels within each modality-specific feature through the deep network. Moreover, cross-calibration can selectively calibrate the noisy features while maintaining the informative cues, yielding a simple yet efficient manner to boost the feature modeling with the help of complementary cues. Finally, the calibrated features are attentively merged through a convolution that learns the contribution weights from the most informative components to form the shared output:

$$f = \text{Conv}_{3 \times 3}(\text{Concat}(f_r^{enh} \otimes f_e^{enh}; \max(f_r^{enh}, f_e^{enh}))). \quad (8)$$

Further, the fused output is integrated with hierarchical features, i.e., the output from the previous layer (if any).

# IV. EXPERIMENTS

## A. Our RGB-Event Moving Object Dataset

To the best of our knowledge, DSEC is the largest RGB-Event dataset for autonomous driving. However, DSEC only proposes the depth ground truth, without object bounding box annotation. [15] proposes to generate object annotation with the pre-trained YOLO-V5 network. However, the used dataset is not available. Moreover, the generated annotation takes both statistics and dynamic objects into account. Therefore, to promote the research of moving object detection with events, we introduce a novel DSEC-MOD dataset.

Specifically, we first calibrate the multi-modal input. Following [21], we project the RGB frames to the event-based coordinates with the help of camera matrices. In this process, RGB and event maps have the same field of view and the same resolution. Then, based on the calibrated RGB videos, we manually label the annotation for moving objects. In our dataset, 8 object classes are taken into account: car, truck, bus, train, pedestrian, cyclist, motorcyclist, and others. Since

we aim to build a challenging dataset with multiple moving objects in the scene, our sequences contain at least 3 different types of moving objects. In total, our DSEC-MOD dataset contains 16 sequences (13314 frames), with 11 sequences (10495 frames) for training and 5 other sequences (2819 frames) for testing. To the best of our knowledge, our moving object dataset is the largest of its kind.

## B. Experimental Setup

**Implementation Details:** We choose ResNet-101 and ResNet-18 as our backbones. The input frames are resized to 288 × 288. We apply classical data augmentations such as photometric transformation, scale, and location jittering. We follow [46] to use adam optimizer with the initial learning rate 5e-4, then decreasing by a factor of 10 at the 10th epochs and 15th epochs. The whole model is trained for 30 epochs.

**Evaluation Metrics:** We use frame mean average precision (F. mAP) and video mean average precision (V. mAP) for evaluation. Frame mAP evaluates the detection quality in a single frame, while video mAP also considers the linking of bounding boxes from a temporal perspective [54]–[56].

## C. Comparison

**Comparison with SOTA fusion alternatives:** To the best of our knowledge, there are very few works on RGB-Event fusion during the encoder stage. Moreover, we are the first to conduct a study on moving object detection with RGB-Event input. Therefore, to purely and fairly analyze our performance, we replace our proposed modules with the state-of-the-art fusion alternatives from RGB-Event domain: FPN-Fusion [15], EFNet [18]; and from RGB-D domain: SAGate [51], DCF [52], SPNet [53]. We also compare with famous self-attention modules such as SENet [49], ECAnet [50], and CBAM [48]. Specifically, we maintain the same multi-scale event input with our E-TMA (if with event input), same backbone, same decoder, loss, and same training settings as ours. The only difference is in the RGB-Event fusion module. Note that for all listed SOTA alternatives, we apply the attention modules through middle fusion.

As shown in Table I, compared to both self-attention and cross-modal attention works, our RENet outperforms its counterparts by a wide margin, validating our proposed fusion design. Specifically, compared to FPN-Fusion [15] which is based on middle fusion with concat-conv, our fusion is based on attention which can better deal with informative features across different modalities. Different from EFnet [18] which attends only to the channel dimension, our fusion leverages both channel and spatial attention and realizes in addition a cross-calibration to improve the feature modeling.

**Comparison under different illumination conditions:** To better understand the contribution of event cameras, we analyze the performance gain under different illumination conditions. The detailed comparison can be found in Table II. It can be seen that with the complementary events, our proposed modules can significantly and consistently boost the baseline performance under different lighting conditions.
<table><caption>TABLE I: **Comparison with SOTA fusion alternatives.** 0.5 and 0.2 stand for the thresholds of the intersection of union.</caption><thead><tr><th>Model Type</th><th>Fusion Module</th><th>Pub. & Year</th><th>F. mAP @0.5 (%)</th><th>V. mAP @0.2 (%)</th><th>V. mAP @0.5 (%)</th></tr></thead><tbody><tr><td rowspan="4">Basic Setting</td><td>RGB only</td><td>/</td><td>30.75</td><td>12.82</td><td>4.97</td></tr><tr><td>Event only</td><td>/</td><td>14.66</td><td>4.50</td><td>0.10</td></tr><tr><td>Early Fusion</td><td>/</td><td>32.27</td><td>17.81</td><td>5.43</td></tr><tr><td>Late Fusion</td><td>/</td><td>34.27</td><td>19.08</td><td>6.41</td></tr><tr><td rowspan="3">Self-Attention</td><td>SENet [49]</td><td>CVPR'18</td><td>29.28</td><td>17.51</td><td>3.77</td></tr><tr><td>CBAM [48]</td><td>ECCV'18</td><td>36.22</td><td>16.36</td><td>5.63</td></tr><tr><td>ECANet [50]</td><td>CVPR'20</td><td>34.49</td><td>18.79</td><td>6.73</td></tr><tr><td rowspan="3">Cross-Attention</td><td>SAGate [51]</td><td>ECCV'20</td><td>33.62</td><td>17.80</td><td>5.34</td></tr><tr><td>DCF [52]</td><td>CVPR'21</td><td>32.20</td><td>18.84</td><td>4.39</td></tr><tr><td>SPNet [53]</td><td>ICCV'21</td><td>32.70</td><td>14.71</td><td>6.57</td></tr><tr><td rowspan="3">RGB-Event</td><td>FPN-Fusion [15]</td><td>ICRA'22</td><td>32.28</td><td>14.97</td><td>5.74</td></tr><tr><td>EFNet [18]</td><td>ECCV'22</td><td>35.33</td><td>18.05</td><td>6.30</td></tr><tr><td>RENet</td><td>(Ours)</td><td><b>38.38</b></td><td><b>19.60</b></td><td><b>7.06</b></td></tr></tbody></table>

<table><caption>TABLE II: **Empirical comparison** under overall and specific illumination conditions.</caption><thead><tr><th>Illum.</th><th>Comp.</th><th>F. mAP@0.5</th><th>V. mAP@0.2</th><th>V. mAP@0.5</th></tr></thead><tbody><tr><td rowspan="2">Overall</td><td>RGB</td><td>30.75</td><td>12.82</td><td>4.97</td></tr><tr><td>+ Ours</td><td><b>+7.63</b></td><td>19.60</td><td><b>+6.78</b></td><td>7.06</td><td><b>+2.09</b></td></tr><tr><td rowspan="2">Day</td><td>RGB</td><td>42.08</td><td>16.81</td><td>9.01</td></tr><tr><td>+ Ours</td><td>45.58</td><td>16.84</td><td><b>+0.03</b></td><td>11.15</td><td><b>+2.14</b></td></tr><tr><td rowspan="2">Twilight</td><td>RGB</td><td>24.44</td><td>13.69</td><td>6.15</td></tr><tr><td>+ Ours</td><td>36.02</td><td>25.82</td><td><b>+12.13</b></td><td>7.04</td><td><b>+0.89</b></td></tr><tr><td rowspan="2">Night</td><td>RGB</td><td>17.44</td><td>15.79</td><td>0.76</td></tr><tr><td>+ Ours</td><td>25.54</td><td>18.76</td><td><b>+2.97</b></td><td>4.93</td><td><b>+4.17</b></td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 4: **Qualitative comparison.** Our network generates more accurate detection with fewer failure cases. Best zoomed in.</figcaption></figure>

**Qualitative Comparison:** Fig. 4 depicts the qualitative comparison between our method and the RGB baseline. We can visualize the significant improvement in detecting objects with the help of events, especially in detecting small objects on the scene that are extremely challenging for RGB baseline. Moreover, when RGB baseline results in false positive due to the inferior lighting condition, our RGB-Event network can reason about more robust detection.

### D. Ablation Studies

We also conduct ablation studies to analyze the contribution of each component. Specifically, we gradually add our proposed modules on top of the RGB baseline. It can be seen that each proposed method is helpful and can enable

<table><caption>TABLE III: **Ablation study on key components.** R-E stands for the RGB-Event baseline where we take a pre-defined range of events as input and features are merged through a simple concat-conv without any form of attention.</caption><thead><tr><th>#</th><th>R-E</th><th>CA</th><th>SA</th><th>E-TMA</th><th>Acc. [18]</th><th>F. mAP@0.5</th></tr></thead><tbody><tr><td>1</td><td>✓</td><td></td><td></td><td></td><td></td><td>34.25</td></tr><tr><td>2</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td>35.56</td></tr><tr><td>3</td><td>✓</td><td>✓</td><td></td><td>✓</td><td></td><td>36.53</td></tr><tr><td>4</td><td>✓</td><td></td><td>✓</td><td>✓</td><td></td><td>37.75</td></tr><tr><td>5</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td><td>36.34</td></tr><tr><td>6</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td><b>38.38</b></td></tr></tbody></table>

performance gain. We also replace our proposed multi-scale event aggregation with the recently proposed multi-scale accumulation [18]. The main difference is that our methods aggregate multi-scale events at the semantic level in a coarse-to-fine manner, while [18] simply concatenates multiple event frames from the input side. Empirically, we can see that the performance drops significantly, which demonstrates the superiority of our proposed module in modeling event data for (moving) object detection.

## V. CONCLUSION

In this paper, we proposed a novel fusion architecture for moving object detection with RGB-Event inputs. Different from previous works based on single-scale event inputs, we introduce a multi-scale events aggregation to explicitly leverage the rich temporal cues from the asynchronous event sensor. Furthermore, we gradually fuse the heterogeneous RGB and event features in a coarse-to-fine manner, yielding a simple yet efficient cross-calibration mechanism with the help of different forms of attention. Finally, we propose a novel DSEC-MOD dataset to encourage further research on RGB-Event fusion for moving object detection. Extensive comparisons of our proposed dataset validate the effectiveness and robustness of our approach compared to the state-of-the-art fusion alternatives. In our future work, we will test our methods on more recent approaches. We will also collect a real-world RGB-Event dataset for autonomous driving.

## REFERENCES

[1] M. Siam, H. Mahgoub, M. Zahran, S. Yogamani, M. Jagersand, and A. El-Sallab, "Modnet: Motion and appearance based moving
object detection network for autonomous driving,” in *International Conference on Intelligent Transportation Systems (ITSC)*, 2018.

[2] H. Rashed, M. Ramzy, V. Vaquero, A. El Sallab, G. Sistu, and S. Yogamani, “Fusemodnet: Real-time camera and lidar based moving object detection for robust low-light autonomous driving,” in *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)*, 2019.

[3] S. A. Baur, D. J. Emmerichs, F. Moosmann, P. Pinggera, B. Ommer, and A. Geiger, “Slim: Self-supervised lidar scene flow and motion segmentation,” in *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2021.

[4] A. Mackin, F. Zhang, and D. R. Bull, “A study of high frame rate video formats,” *IEEE Transactions on Multimedia (TMM)*, vol. 21, no. 6, pp. 1499–1512, 2018.

[5] P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “St-greed: Space-time generalized entropic differences for frame rate dependent video quality prediction,” *IEEE Transactions on Image Processing (TIP)*, vol. 30, pp. 7446–7457, 2021.

[6] G. Sun, Y. Liu, H. Ding, T. Probst, and L. Van Gool, “Coarse-to-fine feature mining for video semantic segmentation,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.

[7] G. Chen, H. Cao, J. Conradt, H. Tang, F. Rohrbein, and A. Knoll, “Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception,” *IEEE Signal Processing Magazine (SPM)*, vol. 37, no. 4, pp. 34–49, 2020.

[8] J. Li, S. Dong, Z. Yu, Y. Tian, and T. Huang, “Event-based vision enhanced: A joint detection framework in autonomous driving,” in *2019 IEEE International Conference on Multimedia and Expo (ICME)*, 2019.

[9] A. I. Maqueda, A. Loquercio, G. Gallego, N. García, and D. Scaramuzza, “Event-based vision meets deep learning on steering prediction for self-driving cars,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018.

[10] W. Zhen and S. Scherer, “Estimating the localizability in tunnel-like environments using lidar and uwb,” in *International Conference on Robotics and Automation (ICRA)*, 2019.

[11] Z. Wu, G. Allibert, C. Stolz, C. Ma, and C. Demonceaux, “Modality-guided subnetwork for salient object detection,” in *International Conference on 3D Vision (3DV)*, 2021.

[12] Z. Wu, S. Gobicettipalayam, B. Tamadazte, G. Allibert, D. P. Paudel, and C. Demonceaux, “Robust rgb-d fusion for saliency detection,” *International Conference on 3D Vision (3DV)*, 2022.

[13] J. Lin and F. Zhang, “R3 live: A robust, real-time, rgb-colored, lidar-inertial-visual tightly-coupled state estimation and mapping package,” in *International Conference on Robotics and Automation (ICRA)*, 2022.

[14] N. Messikommer, D. Gehrig, M. Gehrig, and D. Scaramuzza, “Bridging the gap between events and frames through unsupervised domain adaptation,” *IEEE Robotics and Automation Letters (RAL)*, vol. 7, no. 2, pp. 3515–3522, 2022.

[15] A. Tomy, A. Paigwar, K. S. Mann, A. Renzaglia, and C. Laugier, “Fusing event-based and rgb camera for robust object detection in adverse conditions,” in *International Conference on Robotics and Automation (ICRA)*, 2022.

[16] S. Tulyakov, A. Bochicchio, D. Gehrig, S. Georgoulis, Y. Li, and D. Scaramuzza, “Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.

[17] M. Gehrig, M. Millhäusler, D. Gehrig, and D. Scaramuzza, “E-raft: Dense optical flow from event cameras,” in *International Conference on 3D Vision (3DV)*, 2021.

[18] L. Sun, C. Sakaridis, J. Liang, Q. Jiang, K. Yang, P. Sun, Y. Ye, K. Wang, and L. Van Gool, “Event-based fusion for motion deblurring with cross-modal attention,” in *European Conference on Computer Vision (ECCV)*, 2022.

[19] S. Tulyakov, D. Gehrig, S. Georgoulis, J. Erbach, M. Gehrig, Y. Li, and D. Scaramuzza, “Time lens: Event-based video frame interpolation,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021.

[20] S. Tulyakov, A. Bochicchio, D. Gehrig, S. Georgoulis, Y. Li, and D. Scaramuzza, “Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.

[21] M. Gehrig, W. Aarents, D. Gehrig, and D. Scaramuzza, “Dsec: A stereo event camera dataset for driving scenarios,” *IEEE Robotics and Automation Letters (RAL)*, vol. 6, no. 3, pp. 4947–4954, 2021.

[22] Y. Hu, S.-C. Liu, and T. Delbruck, “v2e: From video frames to realistic dvs events,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021.

[23] J. Hidalgo-Carrió, D. Gehrig, and D. Scaramuzza, “Learning monocular dense depth from events,” in *International Conference on 3D Vision (3DV)*, 2020.

[24] A. Z. Zhu, D. Thakur, T. Özeslan, B. Pfrommer, V. Kumar, and K. Daniilidis, “The multivehicle stereo event camera dataset: An event camera dataset for 3d perception,” *IEEE Robotics and Automation Letters (RAL)*, vol. 3, no. 3, pp. 2032–2039, 2018.

[25] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” *arXiv preprint arXiv:1804.02767*, 2018.

[26] Ultralytics, “Yolov5,” https://github.com/ultralytics/yolov5.

[27] W. Cheng, H. Luo, W. Yang, L. Yu, S. Chen, and W. Li, “Det: A high-resolution dvs dataset for lane extraction,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, 2019.

[28] S. Kundu, G. Datta, M. Pedram, and P. A. Beerel, “Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression,” in *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, 2021.

[29] S. Woźniak, A. Pantazi, T. Bohnstingl, and E. Eleftheriou, “Deep learning incorporating biologically inspired neural dynamics and in-memory computing,” *Nature Machine Intelligence (Nat. Mach. Intell)*, vol. 2, no. 6, pp. 325–336, 2020.

[30] S. B. Shrestha and G. Orchard, “Slayer: Spike layer error reassignment in time,” *Advances in Neural Information Processing Systems (NIPS)*, 2018.

[31] J. H. Lee, T. Delbruck, and M. Pfeiffer, “Training deep spiking neural networks using backpropagation,” *Frontiers in Neuroscience (Front. Neurosci.)*, vol. 10, p. 508, 2016.

[32] B. Rueckauer, I.-A. Lungu, Y. Hu, M. Pfeiffer, and S.-C. Liu, “Conversion of continuous-valued deep networks to efficient event-driven networks for image classification,” *Frontiers in Neuroscience (Front. Neurosci.)*, vol. 11, p. 682, 2017.

[33] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, et al., “Event-based vision: A survey,” *IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)*, vol. 44, no. 1, pp. 154–180, 2020.

[34] A. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “Ev-flownet: Self-supervised optical flow estimation for event-based cameras,” in *Robotics: Science and Systems (RSS)*, 2018.

[35] D. Gehrig, H. Rebecq, G. Gallego, and D. Scaramuzza, “Eklt: Asynchronous photometric feature tracking using events and frames,” *International Journal of Computer Vision (IJCV)*, vol. 128, no. 3, pp. 601–618, 2020.

[36] M. Liu and T. Delbruck, “Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors,” in *British Machine Vision Conference (BMVC)*, 2018.

[37] A. Mondal, S. R, J. H. Giraldo, T. Bouwmans, and A. S. Chowdhury, “Moving object detection for event-based vision using graph spectral clustering,” in *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)*, 2021.

[38] A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman, “Hats: Histograms of averaged time surfaces for robust event-based object classification,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018.

[39] J. Manderscheid, A. Sironi, N. Bourdis, D. Migliore, and V. Lepetit, “Speed invariant time surface for learning to detect corner points with event-based cameras,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019.

[40] Y. Zhou, G. Gallego, H. Rebecq, L. Kneip, H. Li, and D. Scaramuzza, “Semi-dense 3d reconstruction with a stereo event camera,” in *European Conference on Computer Vision (ECCV)*, 2018.

[41] H. Rebecq, R. Ranftl, V. Koltun, and D. Scaramuzza, “Event-to-video: Bringing modern computer vision to event cameras,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019.
[42] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, "Unsupervised event-based learning of optical flow, depth, and egomotion," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019.
[43] P. Bardow, A. J. Davison, and S. Leutenegger, "Simultaneous optical flow and intensity estimation from an event camera," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016.
[44] S. Zhaoning, M. Nico, G. Daniel, and S. Davide, "Ess: Learning event-based semantic segmentation from still images," *European Conference on Computer Vision (ECCV)*, 2022.
[45] N. Messikommer, S. Georgoulis, D. Gehrig, S. Tulyakov, J. Erbach, A. Bochicchio, Y. Li, and D. Scaramuzza, "Multi-bracket high dynamic range imaging with event cameras," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.
[46] Y. Li, Z. Wang, L. Wang, and G. Wu, "Actions as moving points," in *European Conference on Computer Vision (ECCV)*, 2020.
[47] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016.
[48] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "Cbam: Convolutional block attention module," in *European Conference on Computer Vision (ECCV)*, 2018.
[49] J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018.
[50] W. Qilong, W. Banggu, Z. Pengfei, L. Peihua, Z. Wangmeng, and H. Qinghua, "ECA-Net: Efficient channel attention for deep convolutional neural networks," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020.
[51] X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, and G. Zeng, "Bi-directional cross-modality feature propagation with separation-and-aggregation gate for rgb-d semantic segmentation," in *European Conference on Computer Vision (ECCV)*, 2020.
[52] W. Ji, J. Li, S. Yu, M. Zhang, Y. Piao, S. Yao, Q. Bi, K. Ma, Y. Zheng, H. Lu, et al., "Calibrated rgb-d salient object detection," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021.
[53] T. Zhou, H. Fu, G. Chen, Y. Zhou, D.-P. Fan, and L. Shao, "Specificity-preserving rgb-d saliency detection," in *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2021.
[54] P. Weinzaepfel, Z. Harchaoui, and C. Schmid, "Learning to track for spatio-temporal action localization," in *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, 2015.
[55] G. Gkioxari and J. Malik, "Finding action tubes," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015.
[56] V. Kalogeiton, P. Weinzaepfel, V. Ferrari, and C. Schmid, "Action tubelet detector for spatio-temporal action localization," in *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, 2017.