# Steering Prediction via a Multi-Sensor System for Autonomous Racing

Zhuyun Zhou, Zongwei Wu, Florian Bolli, Rémi Boutteau, Fan Yang, Radu

Timofte, Dominique Ginhac, Tobi Delbruck

► To cite this version:

Zhuyun Zhou, Zongwei Wu, Florian Bolli, Rémi Boutteau, Fan Yang, et al. Steering Prediction via a Multi-Sensor System for Autonomous Racing. IEEE International Conference on Robotics and Automation (ICRA), May 2025, Atlanta (USA), United States. hal-04915921

HAL Id: hal-04915921

https://hal.science/hal-04915921v1

Submitted on 9 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Steering Prediction via a Multi-Sensor System for Autonomous Racing

Zhuyun Zhou<sup>1,2</sup>, Zongwei Wu<sup>3,*</sup>, Florian Bolli<sup>2</sup>, Rémi Boutteau<sup>4</sup>, Fan Yang<sup>1</sup>,
Radu Timofte<sup>3</sup>, Dominique Ginhac<sup>1</sup>, and Tobi Delbruck<sup>2</sup>

**Abstract**— Autonomous racing has rapidly gained research attention. Traditionally, racing cars rely on 2D LiDAR as their primary visual system. In this work, we explore the integration of an event camera with the existing system to provide enhanced temporal information. Our goal is to fuse the 2D LiDAR data with event data in an end-to-end learning framework for steering prediction, which is crucial for autonomous racing. To the best of our knowledge, this is the first study addressing this challenging research topic. We start by creating a multisensor dataset specifically for steering prediction. Using this dataset, we establish a benchmark by evaluating various SOTA fusion methods. Our observations reveal that existing methods often incur substantial computational costs. To address this, we apply low-rank techniques to propose a novel, efficient, and effective fusion design. We introduce a new fusion learning policy to guide the fusion process, enhancing robustness against misalignment. Our fusion architecture provides better steering prediction than LiDAR alone, significantly reducing the RMSE from 7.72 to 1.28. Compared to the second-best fusion method, our work represents only 11% of the learnable parameters while achieving better accuracy. The source code, dataset, and benchmark will be released to promote future research.

## I. INTRODUCTION

Autonomous racing has recently attracted growing research interest as it provides a novel platform for testing and evaluating emerging technologies in autonomous driving. Researchers can significantly reduce costs by utilizing scaled-down prototypes like the F1tenth, while still effectively testing hardware systems and software algorithms.

In this work, we address the challenge of steering angle prediction, which is essential for keeping the vehicle within track boundaries. This challenge is particularly difficult in F1tenth prototypes, where most vehicles are equipped only with a 2D LiDAR for environment perception [1], [2], [3]. As a result, [4], [5], [6] have emerged among the most widely used approaches for steering control thanks to their advantages from known geometric properties of the vehicle. Nevertheless, these methods are based on the 2D LiDAR,

<figure><img src="image_2.png" /><figcaption>Fig. 1. (a) The F1tenth racing car used in our experiments is equipped with a DAVIS346 event camera [8] and a Hokuyo 2D LiDAR sensor. (b) Our network processes two consecutive LiDAR scans captured at times *t* and *t* + 1, along with an event-accumulated frame that includes all “on” and “off” brightness change events occurring between *t* and *t* + 1. The network’s objective is to predict the steering angle at time *t* + 1. The LiDAR depth maps are depicted with a blank background, indicating areas with no data, while black pixels correspond to scan points, with the intensity of darkness reflecting proximity. We show that it is possible to leverage the joint benefit within such a multisensor system to achieve accurate steering prediction.</figcaption></figure>

which is the primary sensor for most F1tenth prototypes. Since the 2D LiDAR is only sensitive to depth changes, existing methods are limited by the lack of spatial awareness along the Y- and Z-axis of the ego F1tenth coordinate basis, as well as the temporal clues. When it comes to a scenario where the vehicle operates at high speeds in dynamic environments, these methods may suffer instability from perception delay, which can hinder the vehicle's ability to make rapid decisions.

To overcome these limitations, we first enhance the conventional mono-sensor system by integrating an event camera to create a multisensor setup. As a bio-inspired device, the event camera [7] provides sparse output with high time resolution, high dynamic range, and energy efficiency. These features make it an ideal, though not yet fully explored, component for autonomous racing. By combining distance data from the 2D LiDAR with the event camera's dynamic visual output, we aim to develop a multisensor fusion system that leverages the strengths of both sensors to enhance steering predictions.

Secondly, for accurate and rapid steering prediction, we propose an end-to-end learning-based approach capable of fusing both sensory inputs. It is important to note that, unlike 3D dense point cloud data commonly used in existing

This research is financed in part by the French National Research Agency through ANR CERBERE (ANR-21-CE22-0006), by the Swiss-European Mobility Program, and by the Alexander von Humboldt Foundation.

<footnote n="1">¹Zhuyun Zhou, Fan Yang, and Dominique Ginhac are with University of Burgundy, Dijon, France. {zhuyun.Zhou@etu., fanyang@, dginhac}@u-bourgogne.fr</footnote>

<footnote n="2">²Zhuyun Zhou, Florian Bolli, and Tobi Delbruck are with Sensors Group, Institute of Neuroinformatics, UZH/ETH Zürich, Switzerland. firstname.lastname@uzh.ch</footnote>

<footnote n="3">³Zongwei Wu and Radu Timofte are with University of Wurzburg, Germany. firstname.lastname@uni-wuerzburg.de</footnote>

<footnote n="4">⁴Rémi Boutteau is with Université Rouen Normandie, INSA Rouen Normandie, Université Le Havre Normandie, Normandie Université, LITIS UR 4108, Rouen, France. remi.boutteau@univ-rouen.fr</footnote>

<footnote>* Corresponding author</footnote>
methods, no viable learning-based approach currently exists
for the primary 2D LiDAR data. In terms of event data,
[9] pioneered an event-based steering prediction method.
However, these existing methods have yet to address how
to adapt such techniques to a multisensor setting.

To fill this gap, we propose a novel multisensor fusion
framework specifically designed for F1tenth. To the best of
our knowledge, this is the first work to address the integration
of a 2D LiDAR and event camera for autonomous racing.
While our overall architecture follows a traditional learning-
based pipeline, featuring sensor data extraction, fusion, and
prediction, our primary contribution lies in devising an ef-
fective information fusion strategy. This strategy maximizes
the joint entropy between the two sensor inputs, while tak-
ing practical concerns into consideration such as efficiency,
heterogeneous data representation, and sensor misalignment.

## II. RELATED WORK

**Autonomous Racing and F1tenth Prototype:** Recent ad-
vancements in computer technology have significantly accel-
erated autonomous vehicle research, leading to the rise of au-
tonomous racing competitions. These events have drawn sub-
stantial attention from researchers [10], [11], [12], [13], [14],
focusing on the design and real-time onboard computation of
1/10 scale prototypes, which are commonly used as proof-of-
concept platforms in academic settings. Given the emphasis
on energy efficiency, strict demands are placed not only
on the onboard computational units but also on the choice
of visual and sensor systems. For example, UC Berkeley's
BARC project utilizes ODROID single-board computers and
custom wheel encoders to support its operations [15], [11].
MIT's RACECAR project [10] employs NVIDIA's Jetson
TX1 system-on-module, along with VESC and a sensor suite,
including a 2D LiDAR, stereo camera, and IMU.

**Event Camera and Fusion:** Event cameras have garnered
significant research interest due to their asynchronous pro-
cessing, high dynamic range, and energy efficiency. When
combined with conventional RGB inputs, these cameras
show great potential to revolutionize traditional vision and
robotic tasks, such as autonomous driving [16], [17], depth
estimation [18], and semantic segmentation [19]. Early ap-
proaches to sensor fusion focused on simple feature concate-
nation to produce joint outputs [20], [21]. Building on this,
many researchers have applied spatial and channel attention
mechanisms to refine sensor features either before or after
merging the information [22], [16]. More recent work [19],
[23], [24] leverages transformer mechanisms for deeper fea-
ture modeling, improving robustness against misalignment
but with increased computational demands.

Another research direction explores LiDAR-Event fusion,
focusing on enhancing 3D dense point clouds with 2D event
data. Despite the difference in target sensors, the underlying
motivation is similar to RGB-Event fusion, as demonstrated
by the effectiveness of transformer-based self and cross-
attention mechanisms for feature fusion [25], [26], [27]. In
practice, these fusion designs are quite similar to those used
in RGB-Event fusion [23], [19], with only minor variations,

hence can barely correspond to the real-time computational
exigence for autonomous racing.

**Steering Angle Prediction:** Steering angle prediction is a
crucial research topic for autonomous driving. There are
many surveys in the literature [28], [29]. Here we only review
some closely related learning-based works.

Learning-based approaches estimate steering angle pre-
dictions by directly mapping visual observations to control
actions. This is typically achieved by designing a deep neural
network that takes observations from two different time
stamps and predicts the steering angle, supervised by GT
targets [30], [31]. For example, mapping pixels from two
RGB frames or mapping vowels from two LiDAR scan can
produce a single steering prediction. However, both RGB and
LiDAR are limited by a lack of temporal awareness.

Recently, with advances in event camera technology and
the release of large-scale datasets like DDD17 [32] and
DDD20 [33], many studies [27], [9] have explored jointly
estimating the steering angle using event data or a combina-
tion of event data and other sensors. The latter, multisensor
approaches (event + X) typically demonstrate better accu-
racy. However, directly adapting these existing methods to
the F1tenth platform is not straightforward due to differences
in available sensor systems and the platform's cost-efficiency
requirements. In this work, we explore the feasibility of
fusing 2D LiDAR with event camera data to achieve both
high performance and efficiency in steering angle prediction
for the F1tenth platform. To the best of our knowledge, this
research topic is addressed for the first time in our setting.

## III. SENSOR SETUP AND DATASET

### A. *Our F1tenth Prototype*

Figure 1 shows the F1tenth car. This 1/10 scale vehicle,
based on the Traxxas Slash RC platform, is a variant of
the standard F1tenth car<fnref n="1" />. The car is equipped with an Intel
NUC12 as its primary onboard computer running ROS noetic
on Ubuntu20, a Hokuyo 2D LiDAR with a 10-meter range
and 40Hz scanning frequency, a DAVIS346 event camera
and a VESC MK6 electronic speed controller that manages
the brushless motor and includes an integrated IMU. The
car's state estimation is based on an extended Kalman Filter,
which integrates data from the motor encoder and IMU to
generate odometry information. The drifting odometry is
refined by the SLAM localization algorithm that determines
a more accurate position of the car, by aligning LiDAR
scans with the pre-mapped environment. Both mapping and
localization were performed, using the ROS-Cartographer<fnref n="2" />
SLAM algorithm. During the data collection, the car operated
autonomously using the MPA algorithm, and all sensor
data and system states were recorded and saved within the
Rosbags for further analysis.

### B. *ROS Messages and Synchronization*

Event cameras operate asynchronously, capturing per-pixel
brightness changes in real time. The output is a continuous

<footnote n="1">¹https://f1tenth.org/build.html</footnote>

<footnote n="2">²https://google-cartographer-ros.readthedocs.io/en/latest/</footnote>
stream of events, where each event represents a change in brightness at a specific pixel location and time [7]. Formally, the event stream is defined as:

$$ \epsilon = \{e_i|e_i = ((x_i, y_i), t_i, p_i), t_i \in [t_{start}, t_{end}]\} \quad (1) $$

where e denotes an individual event, (x, y) represents the pixel coordinates, t is the timestamp of the event, and p is the 1-bit polarity with $p \in \{+1, -1\}$, indicating whether the brightness at that pixel increases (+1) or decreases (-1).

Given that the 2D LiDAR operates at a frequency of 40Hz, and to facilitate multi-modal fusion, we accumulate the “on” and “off” events from the event camera according to the timestamps of the LiDAR scans. As a result, each pair of consecutive LiDAR scans is matched with a single accumulated event frame.

Steering angle extraction follows a similar approach. Given the imperfect synchronization of ROS messages, we associate each LiDAR scan with the steering angle whose timestamp is closest to that of the LiDAR scan, doing our best to ensure temporal alignment across all modalities.

### C. Cross-Sensor Calibration and Projection

As suggested in [26], when 3D LiDAR data is projected into 2D, or the event camera’s perspective, LiDAR and event data share a similar structural manifold. Building on this insight, we propose projecting our 2D LiDAR observations into the 2D event camera view. This approach not only brings the sensor representations closer together but also simplifies LiDAR processing, making it more efficient and lightweight in 2D compared to 3D.

To achieve such a transformation, we base on the hypothesis of the pinhole camera model. The projection model can be described as follows:

$$ \mathbf{P}_{\text{image}} = K [R|t] \mathbf{P}_{\text{LiDAR}} \quad (2) $$

where $P_{\text{LiDAR}}$ is the 3D point in the LiDAR’s coordinate system; $R \in \mathbb{R}^{3 \times 3}$ is the rotation matrix that accounts for the relative orientation between the LiDAR and the camera; $t \in \mathbb{R}^{3 \times 1}$ is the translation vector that represents the relative position of the LiDAR with respect to the camera; $K \in \mathbb{R}^{3 \times 3}$ is the intrinsic matrix of the event camera, which encodes the camera’s focal length and principal point.

It is worth noting that the rotation and translation ($[R|t]$) values are assigned using approximate estimates, which is a common practice and standard configuration on the 1stenth vehicle utilizing the ROS platform. These approximations inherently introduce a certain degree of error. Therefore, it can impact the performance and robustness during multi-sensor fusion. Therefore, we have designed a novel, effective, yet efficient method to take this misalignment issue into consideration during the learning phase.

### D. Cleansing and Final Version of the Dataset

To ensure data consistency, we remove the initial and final portions of each ROS bag that contain preparatory or irrelevant messages. After this cleansing process, we retain a total of 27452 valid event-LiDAR-steering angle pairs. For

the experiments, we utilize 21576 pairs from five distinct ROS bags for training, while 5876 pairs from two other separate ROS bags are reserved for testing. This ensures a robust evaluation of our model on diverse yet representative data subsets. Our dataset will be made publicly available to foster and support future research in this domain.

## IV. OUR METHOD

Figure 2 illustrates the overall architecture of our end-to-end learning approach. The network takes as input two consecutive 2D LiDAR scans, along with the event stream from the corresponding time period. We implement a dual-feature extraction pipeline, processing the LiDAR scans and event streams in parallel. To enhance processing speed, we fuse the two LiDAR scans at the input stage, allowing us to use a single encoder for LiDAR feature extraction. After extracting the features from both the LiDAR scans and the event stream, we apply a feature fusion method to generate a joint feature representation. This fused feature is then passed through a decoder to estimate the steering angle. The entire network is trained in a fully supervised manner, using GT steering angles. Additionally, we incorporate internal supervision during the feature fusion stage for further refinement. Technical details of each step are described as follows.

### A. Feature extraction

Without loss of generality and for the ease of review, we denote two consecutive LiDAR scans as $L_{t_1}$ and $L_{t_2}$, representing the scans acquired at timestamp $t_1$ and $t_2$, respectively. Following Eq. 2, we project the LiDAR scans and obtain the associated 2D representations or the 2D depth maps, denoted as $D_{t_1}$ and $D_{t_2}$. Note that each depth map only has 1 channel, referring to the geometric distance along the X-axis of the ego vehicle basis. These depth maps are then concatenated along the channel dimension to form the 2-channel depth input $D$.

For the event camera, we follow the process outlined in Section III-B to obtain the temporally-aligned event stream. “Temporally-aligned” refers to extracting the event stream based on the LiDAR timestamps. Following prior works [23], [17], the event data is accumulated to form a 2D image, denoted as $E$.

For feature extraction, we choose the EfficientNet [34] as the backbone thanks to its efficiency and effectiveness. We process LiDAR and event input in a parallel manner and obtain LiDAR/depth feature $f_D$ and event feature $f_E$, respectively.

### B. Fusion Learning Policy

Once we obtain the multimodal features $f_D$ and $f_E$, the next step is to create a shared output for joint decoding. Conventional fusion methods often rely on implicit fusion, which does not explicitly control the fusion process. These methods assume that both features are perfectly aligned and focus only on exploring their joint benefits. However, in our case, and also as indicated by previous research
<figure><img src="image_3.png" /><figcaption>Fig. 2. (a) Architecture Overview. **Please, zoom in** for better visualization of the BEV and 2D FOV view of the 2D LiDAR point. It is important to note that, unlike 3D LiDAR, the 2D LiDAR points form only a quasi-vertical line after projection. This characteristic makes our multisensor fusion particularly challenging, and, to the best of our knowledge, this specific issue is being addressed for the first time. The overall architecture follows a conventional feature extraction, fusion, and decoding pipeline. However, we introduce a novel fusion method with a new learning policy, as illustrated in (b), to fully exploit the mutual benefits between the 2D LiDAR and event camera, resulting in an efficient yet effective fusion strategy that maximizes joint entropy.</figcaption></figure>

[35], cross-sensor calibration is always imperfect, leading to misalignment between the two features.

To enhance robustness to misalignment, we propose a novel fusion learning policy that explicitly controls the fusion process. Given that $f_D$ and $f_E$ are inherently different and misaligned due to calibration errors, we model these differences as distances between the LiDAR and event domains. Thus, as shown in Figure 2(b), the fusion task is simplified to finding a point that minimizes the distance to both domains.

Specifically, let $f_S$ be the shared/fused output of $f_D$ and $f_E$. We use similarity loss to evaluate the pairings ($f_S$ - $f_D$) and ($f_S$ - $f_E$). In our implementation, we chose the Kullback-Leibler divergence ($KL$) to measure the feature similarity:

$$ \mathcal{L}_{\text{div}} = \mathcal{L}_{KL}(f_S, f_D) + \mathcal{L}_{KL}(f_S, f_E), \\ \mathcal{L}_{KL}(A, B) = KL(A||B) + KL(B||A). \tag{3} $$

Through iterative training, this approach encourages the fused feature $f_S$ to move in directions that reduce the domain distance, ideally, the middle point of these two domains, as shown in Figure 2(b). Such a method is expected to in turn to better align both domain and effectively bridge the gap between geometric knowledge and event clues.

### C. Efficient yet Effective Fusion

The LiDAR features capture geometric changes along the X-axis of the ego F1tenth coordinate basis during the target time period, while the event features represent pixel-wise 2D image changes, which refers to changes along the Y- and Z-axes. Since ego-motion will lead to changes in any of these 3D directions, steering angle prediction can be simplified as mapping the feature differences between observations at two consecutive timestamps from any direction. In other words, a LiDAR-only or event-only visual system can thus meet the basic requirements for estimating the steering angle using visual cues. However, such predictions are based on incomplete information, focusing solely on either the X-axis

or the Y- and Z-axes, without providing a comprehensive understanding of the ego motion.

A straightforward way to address this limitation is through late fusion, where the network generates two separate steering predictions, one based on LiDAR and one based on the event data, and computes their mean as the final output. The advantages and disadvantages of this approach are also straightforward. On the positive side, late fusion avoids the challenges of reconciling the heterogeneous representations of LiDAR and event data, as it operates in the shared space of steering angle prediction. Since the fusion happens at the output level, the predictions are naturally homogeneous or “aligned”. However, this approach also has drawbacks. As shown in Table I(Output fusion), first, it doubles the computational cost. Second, it fails to fully exploit the joint benefits of multimodal features during the deep feature modeling stage.

The fundamental principle of multimodal fusion is that different features provide heterogeneous yet complementary information for the target task. Various fusion methods have been proposed in the literature, as outlined in Section II. Figure 3(a) offers a detailed overview of these approaches. It is evident that recent methods have become increasingly sophisticated, with more complex feature modeling designs aimed at preserving the most informative and unique aspects of each feature for more effective fusion. While these advanced methods can achieve impressive performance, they often involve substantial computational demands. This can be problematic for embedded systems with limited computational resources, as shown in Table I.

To address this challenge, we draw inspiration from recent advancements in low-rank techniques. We propose an efficient yet effective fusion method that projects input features into a low-dimensional latent space, approximating the low-rank structure of the feature map. By performing all feature manipulations in this low-rank space, we significantly reduce the number of learning parameters, making our approach
<figure><img src="image_4.png" /><figcaption>Fig. 3. Comparison against existing fusion methods. Our approach begins by projecting the input features into a lower-dimensional latent space to reduce complexity. We then introduce a novel attention mechanism based on gated convolution. This combination makes our fusion technique both computationally efficient and highly effective, surpassing existing (transformer) attention methods</figcaption></figure>

<figure><img src="image_5.png" /></figure>

more suitable for resource-constrained embedded systems.

As illustrated in Figure 3(b), we first project the LiDAR feature $f_D$ and the event feature $f_E$ into a low-rank space $r$, resulting in $r_D$ and $r_E$, respectively. This projection is achieved using simple $1 \times 1$ convolutions:

$$r_D = Conv_{1 \times 1}(f_D); \quad r_E = Conv_{1 \times 1}(f_E). \quad (4)$$

Next, these features are combined to compute a joint attention map. Unlike traditional transformer-based attention mechanisms, our attention map is generated using gated convolution for improved efficiency:

$$attn = GeLU(Conv_{1 \times 1}(Concat(r_D, r_E))) \quad (5)$$

This joint attention map is then applied to both $r_D$ and $r_E$ to feature recalibration.

$$r'_D = attn \times r_D; \quad r'_E = attn \times r_E \quad (6)$$

Finally, the refined features are merged and projected back into the original latent space to form the shared output $f_S$.

## D. Objective Function

Our network is end-to-end learnable, with full supervision from the GT steering angle. Such a supervision is achieved through a simple L2 loss. Therefore, in addition to the Equation 3, our full loss function becomes:

$$\mathcal{L} = \lambda \cdot \mathcal{L}_{div} + \mathcal{L}_2 \quad (7)$$

where $\lambda$ is a hyperparameter set to be 0.25.

# V. EXPERIMENTS

## A. Implementation Details

We employ EfficientNet-B0 [34] as the encoder to extract features from two consecutive LiDAR frames and one event frame. Standard data augmentation techniques, such as random flipping, are applied during training. The AdamW optimizer is used with an initial learning rate of $1 \times 10^{-3}$, and a weight decay of $1 \times 10^{-2}$. For learning rate scheduling, we utilize CosineAnnealingWarmRestarts, with a restart interval of 30 epochs. The network is implemented in PyTorch and trained for 200 epochs. The total training time is approximately 4.5 hours on an NVIDIA 4090 GPU.

## B. Metrics

To quantitatively assess the performance of our model, we evaluate it using three standard regression metrics for steering angle prediction [33], [9], [27]: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Explained Variance (EVA). These metrics provide a comprehensive understanding of both the precision and variance of our predictions relative to the ground truth.

1) **Root Mean Squared Error (RMSE):** The RMSE is a widely used metric to measure the average magnitude of the prediction errors, providing insight into the accuracy of the model. It is computed as the square root of the mean squared differences between the predicted values $\hat{y}_i$ and the ground truth values $y_i$, as defined by the following equation:

$$\text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2} \quad (8)$$

where $N$ is the total number of samples. RMSE penalizes larger errors more severely than smaller ones, making it sensitive to outliers. A lower RMSE indicates better model performance.

2) **Mean Absolute Error (MAE):** MAE evaluates the average magnitude of errors in a set of predictions, without considering their direction. It is defined as:

$$\text{MAE} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i| \quad (9)$$

Unlike RMSE, MAE treats all errors equally by taking the absolute difference between the ground truth and predictions. A lower MAE reflects higher prediction accuracy and robustness against outliers compared to RMSE.

3) **Explained Variance (EVA):** The EVA measures the proportion of variance in the ground truth that is captured by the model. It is computed as:

$$\text{EVA} = 1 - \frac{\text{Var}(y - \hat{y})}{\text{Var}(y)} \quad (10)$$

where Var(y) is the variance of the ground truth values and Var(y - $\hat{y}$) is the variance of the residuals. An EVA of 1 signifies perfect prediction, while values close to 0 indicate
<table><caption>TABLE I
QUANTITATIVE COMPARISON WITH SOTA EVENT FUSION APPROACHES ON OUR DATASET. METRICS ARE ROOT MEAN SQUARED ERROR (RMSE), MEAN ABSOLUTE ERROR (MAE) SCORES, AND EVA (EXPLAINED VARIANCE). BEST VALUES IN BOLD.</caption><thead><tr><th>Fusion Module</th><th>Pub. & Year</th><th>Fusion Type</th><th>Params (M, ↓)</th><th>FLOPs (G, ↓)</th><th>RMSE (↓)</th><th>MAE (↓)</th><th>EVA (↑)</th></tr></thead><tbody><tr><td>Event only</td><td>/</td><td>/</td><td>4.828</td><td>1.582</td><td>7.164</td><td>4.737</td><td>0.190</td></tr><tr><td>LiDAR only</td><td>/</td><td>/</td><td>4.828</td><td>1.569</td><td>7.716</td><td>6.343</td><td>0.166</td></tr><tr><td>Output fusion</td><td>/</td><td>mean</td><td>2 × 4.828</td><td>3.151</td><td>5.980</td><td>4.588</td><td>0.462</td></tr><tr><td>RAMNet [18]</td><td>-</td><td>RAL<sup>2</sup>1</td><td>500.363</td><td>4.132</td><td>2.292</td><td>1.730</td><td>0.918</td></tr><tr><td>FPN-Fusion [21]</td><td></td><td>ICRA<sup>22</sup></td><td>38.328</td><td>3.208</td><td>1.744</td><td>1.290</td><td>0.952</td></tr><tr><td>EFNet [23]</td><td></td><td>ECCV<sup>22</sup></td><td>21.949</td><td>3.175</td><td>8.665</td><td>7.395</td><td>0.003</td></tr><tr><td>DRFuser [27]</td><td></td><td>EAAI<sup>23</sup></td><td>18.666</td><td>3.272</td><td>4.562</td><td>3.682</td><td>0.773</td></tr><tr><td>CMX [19]</td><td></td><td>TITS<sup>23</sup></td><td>79.331</td><td>3.290</td><td>1.590</td><td>1.156</td><td>0.962</td></tr><tr><td>RENet [16]</td><td></td><td>ICRA<sup>23</sup></td><td>71.521</td><td>3.275</td><td>2.971</td><td>2.377</td><td>0.897</td></tr><tr><td>SAFusion [36]</td><td></td><td>CVPR<sup>24</sup></td><td>155.616</td><td>3.442</td><td>2.504</td><td>1.950</td><td>0.902</td></tr><tr><td>Ours</td><td>-</td><td>conv.</td><td><b>8.919</b></td><td><b>3.149</b></td><td><b>1.282</b></td><td><b>1.007</b></td><td><b>0.975</b></td></tr></tbody></table>

that the model performs no better than predicting the mean of the ground truth. Negative values imply that the model is worse than a naive mean-based predictor.

### C. Benchmark and Comparison

To assess the effectiveness of our approach, we compare it against 7 state-of-the-art event fusion methods, as summarized in Table I. These methods all fall into the categories presented in Figure 3, ie., conv-based, spatial/channel attention-based, or transformer attention-based. Note that the plain methods of these counterparts are not officially designed for steering angle. Hence, we only take the fusion methods and replace our fusion block with it. For fair comparison, all the methods are evaluated with the same loss function with both divergence and L2 losses, as mentioned in Section IV-D.

It can be seen that (transformer) attention methods often surpass convolutional-based counterparts, but also come with additional learnable parameters and FLOPS. Differently, our convolution-based gated attention outperforms all the SOTA fusion alternatives, being more efficient while setting new SOTA records simultaneously. Our model can run at around 260 FPS. We plan to further test our model on the embedded onboard system in future work.

### D. Ablation Studies

We first conduct ablation studies on various choices of the latent space $r$. The latent space is crucial for achieving efficient and effective fusion by significantly reducing computational costs. Table II shows that extremely small latent spaces result in suboptimal performance, because the network cannot encode important features from the input data. We find that $r = 16$ delivers the best accuracy. Increasing $r$ to 32 leads to a deterioration in accuracy, which may be attributed to the added complexity of learning in a higher-dimensional space. Despite this, our network shows resilience to different choices of $r$, as most variants of $r$ outperform the state-of-the-art solutions listed in Table I.

We also evaluate the key components of our network. As shown in Table III, replacing our EfficientNet backbone with a ResNet backbone results in a slight deterioration in accuracy (see #1-#2). This decline may be due to the

<table><caption>TABLE II
ABLAÇÃO STUDY ON LATENT SPACE. OUR CHOICE IS IN BOLD.</caption><thead><tr><th>Latent Space</th><th>RMSE (↓)</th><th>MAE (↓)</th><th>EVA (↑)</th></tr></thead><tbody><tr><td>2</td><td>1.949</td><td>1.516</td><td>0.951</td></tr><tr><td>4</td><td>1.682</td><td>1.252</td><td>0.958</td></tr><tr><td>8</td><td>1.389</td><td>1.079</td><td>0.970</td></tr><tr><td><b>16</b></td><td><b>1.282</b></td><td><b>1.007</b></td><td><b>0.975</b></td></tr><tr><td>32</td><td>1.393</td><td>1.090</td><td>0.970</td></tr></tbody></table>

<table><caption>TABLE III
ABLAÇÃO STUDY ON KEY COMPONENTS. L-B STANDS FOR THE LIGHTER BACKBONE, I.E., EFFICIENTNET.</caption><thead><tr><th>#</th><th>L-B</th><th>KL-loss</th><th>fus.</th><th>RMSE (↓)</th><th>MAE (↓)</th><th>EVA (↑)</th></tr></thead><tbody><tr><td>1</td><td></td><td></td><td></td><td>6.796</td><td>4.874</td><td>0.319</td></tr><tr><td>2</td><td>✓</td><td></td><td></td><td>6.257</td><td>4.645</td><td>0.400</td></tr><tr><td>3</td><td>✓</td><td>✓</td><td></td><td>3.542</td><td>2.891</td><td>0.887</td></tr><tr><td>4</td><td>✓</td><td>✓</td><td>✓</td><td><b>1.282</b></td><td><b>1.007</b></td><td><b>0.975</b></td></tr></tbody></table>

higher number of learning parameters in ResNet compared to EfficientNet, which introduces additional complexity into the learning process, particularly for relatively simple LiDAR inputs. Additionally, we observe a significant accuracy improvement with the inclusion of our KL loss.

## VI. CONCLUSIONS

This work introduces a novel multisensor fusion framework for steering angle prediction in autonomous racing, combining 2D LiDAR and event camera data. Our approach enhances steering prediction by integrating the high temporal resolution of event cameras with the distance accuracy of 2D LiDAR. We propose an end-to-end learning-based system that effectively fuses these sensor inputs, addressing sensor misalignment and data heterogeneity. Our framework shows significant improvements in steering accuracy and efficiency compared to existing methods. By providing our source code, dataset, and benchmark, we aim to facilitate further advancements in autonomous racing technology.

**Acknowledgment:** The authors would like to sincerely thank Liam Boyle, Nicolas Baumann, and Niklas Bastuck from PBL, ETH Zürich, for helping recordings and expertise of the F1tenth car, which are essential for this work.
REFERENCES

[1] B. D. Evans, R. Trump, M. Caccamo, H. W. Jordaan, and H. A. Engelbrecht, "Unifying 1tenth autonomous racing: Survey, methods and benchmarks," *arXiv preprint arXiv:2402.18558*, 2024.

[2] V. S. Babu and M. Behl, "1tenth. dev-an open-source ros based fl/10 autonomous racing simulator," in *2020 IEEE 16th International Conference on Automation Science and Engineering (CASE)*. IEEE, 2020, pp. 1614–1620.

[3] B. D. Evans, J. Betz, H. Zheng, H. A. Engelbrecht, R. Mangharam, and H. W. Jordaan, "Bypassing the simulation-to-reality gap: Online reinforcement learning using a supervisor," in *2023 21st International Conference on Advanced Robotics (ICAR)*. IEEE, 2023, pp. 325–331.

[4] J. M. Snider et al., "Automatic steering methods for autonomous automobile path tracking," *Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RITR-09-08*, 2009.

[5] J. Becker, N. Imholz, L. Schwarzenbach, E. Ghignone, N. Baumann, and M. Magno, "Model-and acceleration-based pursuit controller for high-performance autonomous racing," in *2023 IEEE International Conference on Robotics and Automation (ICRA)*. IEEE, 2023, pp. 5276–5283.

[6] C. Lyu, D. Lu, C. Xiong, R. Hu, Y. Jin, J. Wang, Z. Zeng, and L. Lian, "Toward a gliding hybrid aerial underwater vehicle: Design, fabrication, and experiments," *Journal of Field Robotics*, vol. 39, no. 5, pp. 543–556, 2022.

[7] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, et al., "Event-based vision: A survey," *IEEE Transactions on pattern analysis and machine intelligence*, vol. 44, no. 1, pp. 154–180, 2020.

[8] C. Brandli, R. Berner, M. Yang, S. Liu, and T. Delbruck, "A 240 × 180 130 dB 3 µs latency global shutter spatiotemporal vision sensor," *IEEE J. Solid-State Circuits*, vol. 49, no. 10, pp. 2333–2341, Oct. 2014. [Online]. Available: http://dx.doi.org/10.1109/JSSC.2014.2342715

[9] A. I. Maqueda, A. Loquercio, G. Gallego, N. García, and D. Scaramuzza, "Event-based vision meets deep learning on steering prediction for self-driving cars," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 5419–5427.

[10] S. Karaman, A. Anders, M. Boulet, J. Connor, K. Gregson, W. Guerra, O. Guldner, M. Mohamoud, B. Plancher, R. Shin, et al., "Project-based, collaborative, algorithmic robotics for high school students: Programming self-driving race cars at mit," in *2017 IEEE integrated STEM education conference (ISEC)*. IEEE, 2017, pp. 195–203.

[11] U. Rosolia and F. Borrelli, "Learning how to autonomously race a car: a predictive control approach," *IEEE Transactions on Control Systems Technology*, vol. 28, no. 6, pp. 2713–2719, 2019.

[12] S. S. Srinivasa, P. Lancaster, J. Michalove, M. Schmittle, C. Summers, M. Rockett, R. Scalise, J. R. Smith, S. Choudhury, C. Mavrogiannis, et al., "Mushr: A low-cost, open-source robotic racecar for education and research," *arXiv preprint arXiv:1908.08031*, 2019.

[13] M. O'Kelly, H. Zheng, D. Karthik, and R. Mangharam, "1tenth: An open-source evaluation environment for continuous control and reinforcement learning," *Proceedings of Machine Learning Research*, vol. 123, 2020.

[14] X. Wu and A. Eskandarian, "An improved small-scale connected autonomous vehicle platform," in *Dynamic Systems and Control Conference*, vol. 59148. American Society of Mechanical Engineers, 2019, p. V001T04A003.

[15] J. Gonzales, F. Zhang, K. Li, and F. Borrelli, "Autonomous drifting with onboard sensors," in *Advanced Vehicle Control: Proceedings of the 13th International Symposium on Advanced Vehicle Control (AVEC16)*, 2016, p. 133.

[16] Z. Zhou, Z. Wu, R. Boutteau, F. Yang, C. Demonceaux, and D. Ginhac, "Rgb-event fusion for moving object detection in autonomous driving," in *2023 IEEE International Conference on Robotics and Automation (ICRA)*. IEEE, 2023, pp. 7808–7815.

[17] Z. Zhou, Z. Wu, D. P. Paudel, R. Boutteau, F. Yang, L. Van Gool, R. Timofte, and D. Ginhac, "Event-free moving object segmentation from moving ego vehicle," *arXiv preprint arXiv:2305.00126*, 2023.

[18] D. Gehrig, M. Rüegg, M. Gehrig, J. Hidalgo-Carrió, and D. Scaramuzza, "Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction," *IEEE Robotics and Automation Letters*, vol. 6, no. 2, pp. 2822–2829, 2021.

[19] J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, and R. Stiefelhagen, "Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers," *IEEE Transactions on intelligent transportation systems*, 2023.

[20] H. Cao, G. Chen, Z. Li, Y. Hu, and A. Knoll, "Neurograsp: multimodal neural network with euler region regression for neuromorphic vision-based grasp pose estimation," *IEEE Transactions on Instrumentation and Measurement*, vol. 71, pp. 1–11, 2022.

[21] A. Tomy, A. Paigwar, K. S. Mann, A. Renzaglia, and C. Laugier, "Fusing event-based and rgb camera for robust object detection in adverse conditions," in *2022 International Conference on Robotics and Automation (ICRA)*. IEEE, 2022, pp. 933–939.

[22] H. Cao, G. Chen, J. Xia, G. Zhuang, and A. Knoll, "Fusion-based feature attention gate component for vehicle detection based on event camera," *IEEE Sensors Journal*, vol. 21, no. 21, pp. 24540–24548, 2021.

[23] L. Sun, C. Sakaridis, J. Liang, Q. Jiang, K. Yang, P. Sun, Y. Ye, K. Wang, and L. V. Gool, "Event-based fusion for motion blurring with cross-modal attention," in *European conference on computer vision*. Springer, 2022, pp. 412–428.

[24] H. Cao, Z. Zhang, Y. Xia, X. Li, J. Xia, G. Chen, and A. Knoll, "Embracing events and frames with hierarchical feature refinement network for object detection," *arXiv preprint arXiv:2407.12582*, 2024.

[25] Z. Wan, Y. Mao, J. Zhang, and Y. Dai, "Rpeflow: Multimodal fusion of rgb-pointcloud-event for joint optical flow and scene flow estimation," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2023, pp. 10030–10040.

[26] H. Zhou, Y. Chang, and Z. Shi, "Bring event into rgb and lidar: Hierarchical visual-motion fusion for scene flow," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024, pp. 26477–26486.

[27] F. Munir, S. Azam, K.-C. Yow, B.-G. Lee, and M. Jeon, "Multimodal fusion for sensorimotor control in steering angle prediction," *Engineering Applications of Artificial Intelligence*, vol. 126, p. 107087, 2023.

[28] U. M. Gidado, H. Chiroma, N. Aljojo, S. Abubakar, S. I. Popoola, and M. A. Al-Garadi, "A survey on deep learning for steering angle prediction in autonomous vehicles," *IEEE Access*, vol. 8, pp. 163797–163817, 2020.

[29] H. Saleem, F. Riaz, L. Mostarda, M. A. Niazi, A. Rafiq, and S. Saeed, "Steering angle prediction techniques for autonomous ground vehicles: a review," *IEEE Access*, vol. 9, pp. 78567–78585, 2021.

[30] X. Hu, B. Tang, L. Chen, S. Song, and X. Tong, "Learning a deep cascaded neural network for multiple motion commands prediction in autonomous driving," *IEEE Transactions on Intelligent Transportation Systems*, vol. 22, no. 12, pp. 7585–7596, 2020.

[31] H. Wu, D. Lv, T. Cui, G. Hou, M. Watanabe, and W. Kong, "Sdlv: Verification of steering angle safety for self-driving cars," *Formal Aspects of Computing*, vol. 33, pp. 325–341, 2021.

[32] J. Binas, D. Neil, S.-C. Liu, and T. Delbruck, "Ddd17: End-to-end davis driving dataset," *arXiv preprint arXiv:1711.01458*, 2017.

[33] Y. Hu, J. Binas, D. Neil, S.-C. Liu, and T. Delbruck, "Ddd20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction," in *2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)*. IEEE, 2020, pp. 1–6.

[34] T. Mingxing and V. L. Quoc, "Efficientnet: Rethinking model scaling for convolutional neural networks," *arXiv preprint arXiv:1905.11946*, vol. 1, 2019.

[35] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L. Tai, "Transfusion: Robust lidar-camera fusion for 3d object detection with transformers," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2022, pp. 1090–1099.

[36] J. Jiang, X. Zhou, B. Wang, X. Deng, C. Xu, and B. Shi, "Completing event streams and rgb frames for hand mesh reconstruction," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024, pp. 24944–24954.