# Lightweight convolutional neural network for real-time 3D object detection in road and railway environments

A. Mauri, Redouane Khemmar, B. Decoux, M. Haddad, Rémi Boutteau

<figure><img src="image_1.png" /></figure>

## To cite this version:

A. Mauri, Redouane Khemmar, B. Decoux, M. Haddad, Rémi Boutteau. Lightweight convolutional neural network for real-time 3D object detection in road and railway environments. Journal of Real-Time Image Processing, 2022, 10.1007/s11554-022-01202-6. http://hal-03592337

HAL Id: hal-03592337

https://hal.science/hal-03592337v1

Submitted on 8 Nov 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Lightweight convolutional neural network for real-time 3D object detection in road and railway environments

A. Mauri<fnref n="1" /> · R. Khemmar¹ID · B. Decoux<fnref n="1" /> · M. Haddad<fnref n="2" /> · R. Boutteau<fnref n="3" />

Received: 29 July 2021 / Accepted: 10 January 2022 / Published online: 11 February 2022
© The Author(s) 2022

## Abstract

For smart mobility, and autonomous vehicles (AV), it is necessary to have a very precise perception of the environment to guarantee reliable decision-making, and to be able to extend the results obtained for the road sector to other areas such as rail. To this end, we introduce a new single-stage monocular real-time 3D object detection convolutional neural network (CNN) based on YOLOv5, dedicated to smart mobility applications for both road and rail environments. To perform the 3D parameter regression, we replace YOLOv5's anchor boxes with our hybrid anchor boxes. Our method is available in different model sizes such as YOLOv5: small, medium, and large. The new model that we propose is optimized for real-time embedded constraints (lightweight, speed, and accuracy) that takes advantage of the improvement brought by split attention (SA) convolutions called small split attention model (Small-SA). To validate our CNN model, we also introduce a new virtual dataset for both road and rail environments by leveraging the video game Grand Theft Auto V (GTAV). We provide extensive results of our different models on both KITTI and our own GTAV datasets. Through our results, we show that our method is the fastest available 3D object detection with accuracy results close to state-of-the-art methods on the KITTI road dataset. We further demonstrate that the pre-training process on our GTAV virtual dataset improves the accuracy on real datasets such as KITTI, thus allowing our method to obtain an even greater accuracy than state-of-the-art approaches with 16.16% 3D average precision on hard car detection with inference time of 11.1 ms/image on an RTX 3080 GPU.

**Keywords** 3D bounding box estimation · 3D multi-object detection · Multi-modal dataset · Deep learning · Smart mobility

## Abbreviations

<table><tr><td>ADAS</td><td>Advanced driver assistance system</td></tr><tr><td>GTAV</td><td>Grand theft auto V</td></tr><tr><td>CNN</td><td>Convolutional neural network</td></tr></table>

<figure><img src="image_3.png" /></figure>

R. Khemmar
redouane.khemmar@esigelec.fr

A. Mauri
antoine.mauri@esigelec.fr

B. Decoux
benoit.decoux@esigelec.fr

M. Haddad
madjid.haddad@segula.fr

R. Boutteau
remi.boutteau@univ-rouen.fr

<table><tr><td>YOLOv5</td><td>You only look once</td></tr><tr><td>KITTI</td><td>Karlsruhe Institute of Technology and Toyota technological institute</td></tr><tr><td>LiDAR</td><td>Light detection and ranging</td></tr><tr><td>L-CNN</td><td>Lightweight-CNN</td></tr><tr><td>CARLA</td><td>CAR learning to act</td></tr><tr><td>ROAD</td><td>ROAD event awareness dataset</td></tr><tr><td>SYNTHIA</td><td>SYNTHetic collection of Imagery and Annotations</td></tr><tr><td>R-CNN</td><td>Region-based CNN</td></tr><tr><td>LSTM</td><td>Long short term memory</td></tr><tr><td>GAM3D</td><td>Ground-aware monocular 3D object</td></tr><tr><td>KFPN</td><td>Keypoint feature pyramid network</td></tr><tr><td>M3D-RPN</td><td>Monocular 3D region proposal network</td></tr><tr><td>SA</td><td>Split attention</td></tr><tr><td>API</td><td>Application programming interface</td></tr><tr><td>IOU</td><td>Intersection over union</td></tr><tr><td>AP</td><td>Average precision</td></tr><tr><td>ToF</td><td>Time-of-flight</td></tr><tr><td>DS</td><td>Dimension score</td></tr><tr><td>CS</td><td>Center score</td></tr></table>

<footnote n="1">¹ Normandie Univ., UNIROUEN, ESIGELEC, IRSEEM, 76000 Rouen, France</footnote>

<footnote n="2">² SEGULA Technologies, 19 Rue d'Arras, 92000 Nanterre, France</footnote>

<footnote n="3">³ Normandie Univ., UNIROUEN, UNILEHAVRE, INSA Rouen, LITIS, 76000 Rouen, France</footnote>
<table><tr><td>OS</td><td>Orientation score</td></tr><tr><td>GT</td><td>Ground truth</td></tr><tr><td>RoIs</td><td>Region of interests</td></tr><tr><td>SRE</td><td>Squared relative error</td></tr><tr><td>RMSE</td><td>Root mean square error</td></tr><tr><td>MTL</td><td>Multi task learning</td></tr></table>

# 1 Introduction

Multiple object detection consists of detecting and identifying objects in scenes. In smart mobility applications, vehicles must have a better perception of their environment. This perception must guarantee not only a good detection of the objects, but also a good interpretation of the scenes. In multimodal road and railway smart mobility, the objects of interest to be detected are vehicles, pedestrians, bus, cyclists, trees, etc. If the road sector is widely covered in the scientific literature, this is less the case for the railway smart mobility. This is in part due to the lack of specific railway datasets with ground truth data dedicated to 3D object detection. This poses a real challenge in that it requires the 3D object detection algorithm to train under dataset with ground truth data.

In the past few decades, specifically since 2012, Deep Learning has become a very powerful tool because of its ability to handle large amounts of data. The interest using more and more hidden and intermediate fully connected layers has surpassed traditional techniques in image processing and computer vision, especially in pattern recognition, object detection, and classification [1, 2]. One of the most popular deep neural networks is CNN. With the rise of Deep Learning approaches for computer vision tasks and the emergence of CNNs, it has become possible for cameras to be used for object detection, depth estimation, tracking, instance, and semantic segmentation, etc. These kinds of methods have many applications, especially for smart mobility to enhance safety and increase vehicle autonomy. This need for environmental perception has led to the development of 3D detection methods, using either a LiDAR-type depth sensor or images from cameras, to improve safety. Although 3D object detection is a much sought-after field, few methods focus on the real-time aspect that is essential for a real-world application. Of course, in road and railway multimodal smart mobility, it is desired that 3D multi-object detection is performed in real time. This represents another challenge. In this paper, we propose to fill these gaps, lack of railway dataset with ground truth data and real-time 3D object detection, by proposing a new virtual dataset (GTAV) dedicated to both road and railway environments for 3D object detection, which includes images taken from the point of view of both cars and trains. With this dataset, we propose a new method dedicated to real-time 3D object detection based on the well-known 2D object detector YOLOv5

[3]. With this new method, we introduce 3D anchor boxes which makes our method the fastest method available for 3D object detection with accuracy comparable to state-of-the-art methods. Using a pre-trained model on our GTAV dataset, our approach outperforms state-of-the-art methods. Through our experiments, we also demonstrated the validity of our method for real-time embedded applications for autonomous vehicles (cars and trains). The main contributions put forward in this paper are the following:

1. A new Lightweight CNN (L-CNN) method for real-time 3D object detection based on the YOLOv5 2D object detector. Our L-CNN allows the following object predictions:

- 2D detection

- Distance from the camera

- 3D centers projected on the image plane

- 3D dimensions and orientation

2. A new virtual dataset (GTAV) dedicated to both road and railway environments for 3D object detection. The GTAV dataset includes images taken from the point of view of both cars and trains.

The remainder of this paper is organized as follows. Section 1 introduces the paper. In Sect. 2, we review the related work for autonomous navigation datasets, 3D object detection approaches in which we will present methods based on depth sensors as well as methods based on monocular images. In Sect. 3, we describe in more details our multimodal road/railway GTAV dataset. Our new L-CNN model about the 3D multi-class object detection is presented in Sect. 4. Analysis and experimental results are described in Sect. 5. Finally, the conclusion and future directions are outlined in Sect. 6.

# 2 Related work

## 2.1 Autonomous driving datasets

Any approach based on Deep Learning requires annotations (also called “ground-truth”) to perform a training. One of the main factors in the performance of any of these approaches in terms of precision is the quality and quantity of the data in the dataset. Among the datasets dedicated to autonomous vehicles, the vast majority are intended to autonomous driving in road environments and focus on object detection and scene segmentation. Some of them are mono-modal (in general, with images as a single modality), and others are multi-modal.
Even though the building of mono-modal datasets is less time-consuming and expensive than multimodal datasets. The latter represent a crucial advantage in the use of supervised artificial intelligence approaches because they offer a wider variety of ground truths from different sensors. Although these data are expensive because they require the use of expensive sensors such as LiDAR or radar, they are indispensable for environmental perception applications such as object detection/localization, distance estimation, tracking. We can also find many other road datasets for similar tasks such as CityScape [4], PascalVOC [5], etc. The best-known road dataset is KITTI [6] (Karlsruhe Institute of Technology and Toyota Technological Institute), which includes a rich ground truth from both distance sensors (LiDAR), a GPS/IMU unit, and stereoscopic cameras (color or grayscale). It comprises different information captured and synchronized at 10Hz. The vehicle is equipped with different sensors: cameras, 3D Velodyne LiDAR (100k points per frame), 3D GPS/IMU data (location, speed, acceleration). The camera is calibrated and synchronized with Velodyne LiDAR and with a GPS/IMU sensor [6]. KITTI includes 3D object track labels for different object classes (cars, trucks, trams, pedestrians, cyclists). It is a reference base for both training and evaluation of methods for stereo, optical flow, visual odometry, 2D object detection, 3D object detection, depth estimation, 2D tracking, etc. Among the road datasets, we can also find the NuScenes [7] dataset which is a public large-scale dataset for autonomous driving. NuScenes is more recent than the KITTI dataset which inspired it. It includes 1000 driving scenes (of 20 s in length each) in Boston and Singapore, cities known for their dense traffic. It includes 23 object classes annotated with accurate 3D bounding boxes at 2 Hz which is a good annotation for common autonomous driving tasks such as object detection, distance estimation, and tracking. NuScenes includes approximately 1.4 M camera images, 390 k LIDAR sweeps, 1.4 M RADAR sweeps, and 1.4 M object bounding boxes. As with KITTI, this dataset incorporates one LiDAR sensor, 5 radar sensors, 6 cameras to provide 360° coverage around the vehicle, and one GPS/IMU unit [7]. This dataset contains 100x more images than KITTI and allows training methods for 2D and 3D object tracking, 2D and 3D object detection, depth estimation.

ROad [8] event Awareness Dataset for Autonomous Driv-
ing (ROAD), includes 22 videos with 122 K annotated video
frames, and 560 K detection bounding boxes with 1.7 M
individual labels. ROAD is a dataset that was designed to
test a robot-car’s situation awareness capabilities. The anno-
tation process follows a multi-label approach in which road
agents (vehicles, pedestrians, etc.), their locations as well as
the action they [8].

We can find many other road datasets for similar tasks
which are also multimodal large-scale autonomous vehicles

datasets. Argoverse [9] dataset images are captured under
different weather conditions. It includes annotations for
RGB images, LiDAR/radar, and 3D boxes. It includes more
sensors than both KITTI [6] and NuScenes [7] datasets.
Argoverse includes 3D bounding boxes with tracking infor-
mation for 15 objects of interest [9]. Another multimodal
dataset, Lyft [10] uses cameras and LiDAR to provide 2D
annotation and 3D bounding boxes for 4 object classes. All
these datasets provide contextual knowledge through human-
annotation which is very important for scene understanding
applications. Waymo open dataset [11] is a new large-scale,
high-quality LiDAR and camera data that includes 1150
scenes (each with the 20s). Sensors produce annotated data
with 2D (camera image) and 3D (LiDAR) bounding boxes,
with consistent identifiers across frames.

To our knowledge, there are only a few image datasets
dedicated to the railway [12, 13]. However, these datasets are
not dedicated to 3D detection and do not have ground truth
information on depth estimation. Therefore, these datasets
cannot be used for the training or evaluation of 3D object
detection approaches.

Finally, there are databases of virtual images created
with simulators such as CARLA [14] or SYNTHIA [15].
The advantage of these databases is that they are simpler to
acquire, the ground truth data are acquired automatically by
the API simulator, which allows the acquisition of mass data.
It also allows acquiring a database adapted to specific needs
such as the understanding of scenes in the railway environ-
ment. However, one of the disadvantages of this kind of
dataset is that they have dated graphics, which makes it dif-
ficult to use trained dataset in real conditions. To overcome
this problem, recent work has been done on the acquisition
of a database via a video game [16]. Indeed, video games
make graphic quality their priority, which makes using of
video games particularly interesting. The video game GTAV
is used to acquire road databases similar to KITTI or City-
Scape. Although there are many virtual road datasets, to
our knowledge, there is no similar state-of-the-art work for
the railway.

## 2.2 3D object detection from monocular images

In the very recent years, many methods based on deep learn-
ing have been proposed for the detection of 3D objects from
monocular images. Most of them use single RGB images,
and a few use image sequences. Many of these methods are
extension of proven 2D detectors, thus leveraging their very
good performances. 3D parameters processing is then added
to the models. Some other methods infer 3D parameters
directly through end-to-end learning.

In [17], the problem is tackled in two steps. First can-
didate 3D boxes are generated and scored by exploiting
several features, namely class semantic, instance semantic,
contour, object shape, context, and location prior. Then the
candidates with the best scores are applied to a Fast R-CNN
[18] to predict class labels, bounding-box offsets and object
orientation.

In [19], 2D detection is done by the multi-scale region-
proposal MS-CNN [20], extended to regress the orientation
and the dimensions of the 3D bounding boxes. Estimation
of orientation uses a MultiBin principle, in which angle
is turned to a hybrid classification and regression task, in
which values are separated into bins representing the classes.

DeepMANTA [21] is a coarse-to-fine model for simulta-
neous vehicle detection, localization, visibility characteri-
zation, and 3D dimension estimation. It is based on a two-
step process, the first one of which outputs bounding boxes
associated with vehicle information, and the second one uses
these outputs and a 3D vehicle virtual dataset to recover 3D
orientations and locations. In [22], a depth map obtained
by means of a fully-convolutional network (FCN) is fused
with the input RGB image before feeding a region-proposal
network (RPN). It is also combined at different level in the
flow stream to finally get estimation of class labels, 2D box
position, and 3D box orientation, dimensions and location.
For the orientation, the same MultiBin principle as in [20]
is used.

In [23], a CNN model for joint 3D object detection and
tracking is proposed. It is based on a faster RCNN to predict
the 2D bounding boxes, which are then associated to 3D
information including position, orientation, dimensions, and
projected 3D box centers of each object. The tracking across
sequences of frames is obtained by two LSTM (Long Short-
Term Memory) layers, allowing a further refinement of the
3D bounding box positions.

In SMOKE [24], a CNN is trained end-to-end in a single
stage by means of a unified loss function. The parameters
of 3D bounding boxes are regressed directly without using
2D detection, thanks to a network made of two branches: a
classification branch where an object is encoded by its 3D
center (projected on the image plane), and a 3D parameter
regression branch to construct 3D bounding box around each
center. In MonoGRNet [25], 3D object detection is decom-
posed into four sub-tasks: 2D object detection, object center
depth estimation, projected 3D center estimation and local
corner regression. Those four pieces of information are
obtained in parallel through a single-pass process, and then
combined.

Anchor-based methods have also been proposed in M3D-
RPN [26] and GAM3D [27]. M3D-RPN leverages depth-
aware convolution for locating the specific features and
improve the 3D scene understanding. The approach con-
sists only of a single-stage using anchor boxes for predicting
both 2D and 3D object positions. GAM3D also uses 3D/2D
anchor boxes to predict the objects’ 3D bounding box and
introduces ground-aware convolution module to provide

additional 3D cues to the network, thus improving the accu-
racy of the network.

In [28], 3D detection is reformulated as a keypoint detec-
tion problem. A keypoint feature pyramid network (KFPN)
is defined, providing the center and the perspective points of
the 3D bounding-boxes. The fact that the network backbone
is based on a lightweight architecture (such as ResNet-18
[29] or DLA-34 [30]) and the anchor-free nature of the
detector, allow real-time processing on 1080Ti GPU.

Most of these models are dedicated to improve perfor-
mance of 3D object detection in terms of accuracy. A few
other focus more on processing time, aiming for real-time on
powerful GPUs. In our case, we want to focus on lightweight
CNN architectures making it possible to achieve real-time
on low-power GPUs like the Jetson-TX2. We believe that
it is important to go further on this point to facilitate the
large-scale development of deep-learning solutions for the
safety of autonomous vehicle. Another important aspect of
our needs is to be able to perform well in road environment
as well as railway environment. The lack of existing rail-
way dataset has led us to develop our own dataset, made of
virtual images from GTA-V game. We will use this custom
dataset in complement to KITTI. In addition to our virtual
railway dataset, we are developing a real railway dataset with
ground truth allowing us to go further in the development of
railway e-Advanced Driver Assistance Systems (e-ADAS).

3 Multimodal road/rail dataset

3.1 GTAV virtual multi-modal dataset presentation

To compensate for lack of real dataset in the railway environ-
ment, we turn to a virtual dataset. The main advantage of
a virtual dataset is the simplification of its acquisition and
annotation. However, this is done at the cost of the fidelity
to reality. Indeed, most of these datasets (such as CARLA)
do not have sufficient graphical fidelity to allow the methods
trained on them to offer good performances once applied in
real world conditions. To overcome this problem, we turn
to video games to acquire a railway database for 3D object
detection. Indeed, where simulators lack graphical fidelity,
video games try to offer the best graphical fidelity to offer
users a feeling of authenticity. This is why several works
have already been carried out in this field to extract data-
bases. One of the most interesting games is the video game
Grand Theft Auto V, which allows the user to drive road
vehicles and presents realistic road scenes with, for example,
level crossings, crosswalks, heavy traffic, etc. A lot of work
has been done in this direction to build databases for the
road environment [16, 23]. Our railway database has been
acquired with a data acquisition pipeline based on the work
of [31]. To do this we used a modified version of the game

<figure><img src="image_4.png" /></figure>
[32] code to allow the user to drive a subway or a train, allowing us to build a hybrid database with both road and rail scenes that can then be used for training and evaluation of our 3D detection approach.

## 3.2 GTAV virtual multi-modal dataset architecture

The ground truth is generated automatically by the game's API and includes among other things the 3D and 2D bounding boxes, the object class, the depth map and the semantic map. All these annotations are taken from the point of view of a stereoscopic camera with an acquisition frequency of 10 Hz. Some statistics for our dataset can be found in Figure 1.

During the acquisition of the dataset, several classes of objects were annotated. However, to combine our virtual dataset with the 3D detection KITTI dataset, we selected only the classes that are also present on KITTI (Car, Person, Cyclist) dataset. Our dataset is composed of 140 K samples from the stereoscopic camera. Each of these samples has a

depth map and a ground truth for 3D detection (position in the camera coordinates, dimensions, orientation, etc.). Image samples from our dataset can be found in Fig. 2. Our dataset also offers various situations and environmental conditions resulting in a challenging dataset for computer vision tasks. The dataset includes night, day, rainy and clear weather scenes, offering a wide variety of visibility conditions.

## 4 3D multi-class object detection

Our method is a single-stage 3D object detector based on the YOLOv5 2D detector neural network. Indeed, it is a light convolutional network that offers good accuracy, allowing real-time use even on embedded systems such as Nvidia Jetson TX2. YOLOv5 offers a significant performance gain compared to YOLOv5 thanks to an improved network architecture as well as innovations on data augmentation during training. The single-stage

<figure><img src="image_5.png" /><figcaption>**Fig. 1** Statistic details for our new dataset: Top Left is the image daytime distribution, Top Right is the weather condition distribution, Bottom Left is the Road/Railway distribution, Bottom Right is the class distribution</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Fig. 2** Images with 3D bounding box ground truths. Our dataset presents a variety of environments and conditions. From top left to bottom right: road with clear weather, Road at night, Road with rainy weather and Rail with clear weather. The ground truths shown are for the same classes as KITTI: Car, Person and Cycle</figcaption></figure>

design of our network allows for better computational times compared to multiple stage methods [19, 21, 23] and it can be trained end-to-end. Another problem posed by multiple-stage methods was the degradation in accuracy for the 3D parameters regression when the predicted RoIs used for feature alignment vary from the ones used in the training [33]. Our single-stage method uses hybrid

anchor boxes to directly regress the 2D bounding box as well as the 3D parameters. We thus avoid the problem related to the fluctuation of the accuracy when the ROIs are not fixed. This new approach is the fastest method for 3D prediction from images without sacrificing accuracy. Figure 3 shows an overview of our 3D multi-object detection method.

<figure><img src="image_7.png" /><figcaption>**Fig. 3** Overview of our method for 3D multi-object detection. A single RGB image is used as input. We leverage our Hybrid anchor boxes to predict both the 2D and the 3D bounding boxes. Non-Maximum-Suppression is used to filter the predictions. Among the 3D</figcaption></figure>

parameters that we predict, we have the projected 3D center on the image plane, the distance of the object, its dimensions, and finally its orientation

<figure><img src="image_8.png" /></figure>
## 4.1 3D bounding box estimation

The 3D bounding box of an object can be described by its center position from the camera $T = [x y z]^T$, its dimensions $D = [w, h, l]$ (with $w, h, l$ respectively width, height and length) and its orientation $R(\phi, \theta, \psi)$ (with $\phi, \theta, \psi$ respectively the elevation, azimuth, and roll angles). Given $K$ the matrix of intrinsic parameters of the camera and $X_0 = [x_0 y_0 z_0 1]^T$ a 3D point in the object coordinate system, the re-projection of this point into the image plane $x_{im} = [u v 1]^T$ is given by:

$$x_{im} = K \cdot [R \ T] \cdot X_0, \qquad (1)$$

where $[RT]$ represents the extrinsic matrix with $R$ ($3 \times 3$ matrix) the rotation and $T$ ($3 \times 1$) the translation. It is an extrinsic matrix integrating Translation ($T$) and Rotation ($R$). By considering that the center of the 3D bounding box is the origin of the object coordinates, the coordinates of the 3D bounding box are: $X_1 = [w/2 \ h/2 \ l/2]$, $X_2 = [w/2 \ -h/2 \ l/2]$, ..., $X_8 = [-w/2 \ -h/2 \ -l/2]$. The 3D bounding box coordinates in the image can then be obtained using Equation (1).

## 4.2 Parameters to regress

To better compare the performances of multi-stage approaches with single ones for 3D object detection, we predict similar parameters as it was presented in our previous multi-stage approach [33].

### 4.2.1 2D object detection

In our work, we use YOLOv5 to perform 2D object detection. YOLOv5 performs the object class classification `cls` as well as the bounding box position regression `b`. The bounding box `b` is then used as prior information for predicting the object center and the object class `cls` is used in the object dimension prediction.

### 4.2.2 Object center prediction

In this work, we assume that the 3D center of the object is the center of the 3D bounding box. Predicting the center of the 3D bounding boxes of the objects is, therefore, equivalent to predicting their position $X_0 = [x_0 y_0 z_0 1]^T$. But instead of directly predicting the coordinates of the 3D object center on the image plane, we use cues from the 2D bounding box estimation. We predict the offset position in pixels of the projected object center from the 2D bounding box center $\tilde{c} = [\tilde{c}x \ \tilde{c}y]$. The variance of the prediction is thus reduced, making it easier to be learned by the algorithm. The

object center $X_0$ is then computed using the object distance estimation on the Z-axis and the inverted calibration matrix $K^{-1}$.

### 4.2.3 Object distance estimation

To obtain the center of the 3D bounding box, it is necessary to get its position on the Z-axis of the camera coordinates. For each ROIs from the object detector, we predict the object’s center distance $\tilde{z}$ in meters.

### 4.2.4 Object dimensions

Instead of directly predicting the object’s dimensions, we exploit the fact that those dimensions have a very low variance within the same class (car, truck, etc). Therefore, we choose to use the average dimensions of each object class as a strong prior for the estimation of the dimensions.

### 4.2.5 Object orientation

In our work, we assume that only the azimuth (noted $\theta$) matters for application on the road environment, and thus we do not predict the orientation characterized by the elevation and roll and we set $\phi = 0$ and $\psi = 0$. Since the same azimuth can lead to multiple observed orientations from the camera point of view (see Fig. 4), we cannot directly predict the angle $\theta$. Instead, we predict the observed angle $\alpha$ and retrieve the global orientation $\theta$ using Eq. (2):

$$\theta = \alpha + \arctan(\frac{x}{z}). \qquad (2)$$

Following the work described in [19], instead of considering the angle prediction as a regression task, we formulate it as a hybrid classification/regression problem in which the possible angles are separated into two bins of which the values are centered on 90 for the first one and -90 for the second one. We then perform a classification to predict in which bin the object angle is located. The parameter which is regressed is the difference between the bin center and $\alpha$.

## 4.3 Hybrid anchor boxes for 3D detection

To take advantage of the innovations brought by single-stage approaches, we have modified the YOLOv5 anchor boxes to create hybrid anchor boxes to perform the regression of the 2D bounding boxes and the 3D parameters. We predict three grids of features at different sizes. In each grid, three anchor boxes are predicted. These boxes contain the predictions of the classes of the objects, the relative position of the bounding box from the center of the grid, the probability that an object is present, and finally the 3D bounding box parameters (projected center, distance, dimensions, orientation).
<figure><img src="image_9.png" /><figcaption>Fig. 4 Illustration of the object azimuth $\theta$ and its observed orientation $\alpha$. The local orientation is retrieved by computing the angle between the normal to the ray between the camera and the object center and the X-axis of the camera. Given that we use left-hand coordinates, the rotation is clockwise. Our method estimates the observed orientation and $\theta$ can be obtained using Eq. (2)</figcaption></figure>

The output of our hybrid anchor boxes for our 3D object detector is detailed in Eq. 3. Let $y$ be the output value of one of our new anchor boxes:

$$y = [\mathbf{b}_{\text{off}} \ obj \ cls \ c_{\text{off}} \ Z \ D \ O.] \quad (3)$$

With $\mathbf{b}_{\text{off}} = [x \ y \ w \ h]$ the 2D bounding box offset prediction, $obj$ is the object prediction, $cls = [cls_1 \ ... \ cls_N]$ the score for the N classes, $\mathbf{c}_{\text{off}} = [x_c \ y_c]$ the prediction of the offset between the projected 3D center and the center of the bounding box in pixels, $Z$ is the position on the Z-axis of the 3D center of the object in meters, $\mathbf{D} = [W_1 \ H_1 \ L_1 \ ... \ W_N \ H_N \ L_N]$ is the predicted dimensions offset in meters for each N class. The predicted offset is between the object's dimension and its class mean dimensions. Finally we have the orientation prediction $\mathbf{O} = [\text{bin}_1 \ \overline{\text{bin}_1} \ \sin_1 \cos_1 \ \text{bin}_2 \ \overline{\text{bin}_2} \ \sin_2 \cos_2]$, where $\text{bin}_1$ and $\text{bin}_2$ represent the predicted probability that $\alpha \in [-195 \ 15]$ and $\alpha \in [-15 \ 105]$, respectively with $\alpha$ the observed orientation of the object. And $\sin_{i \in \{1,2\}} \cos_{i \in \{1,2\}}$

are the sinus and cosine of the observed angle offset regarding the center of the ith bin. Our bin centers are 90° and -90°, respectively. By adding the prediction of the 3D parameters inside the anchor boxes, our method learns to predict the distance of the objects, the position of their centroid, their dimensions, and their azimuth. With these parameters, we are then able to draw the 3D bounding boxes of the objects. All the anchor boxes of the different grids are then merged and re-projected onto the image and a Non-Maximum-Suppression function is used to filter the anchor boxes. This new approach taking advantage of anchor boxes allows our method to do 3D bounding box prediction in a single stage, thus reducing the size of our network without sacrificing accuracy. Another notable advantage over other multiple stage 3D detection methods is that it avoids the problem of deteriorating accuracy of 3D detection when the predicted ROIs for objects vary, especially when aligning features for 3D regression.

## 4.4 Losses

In this subsection, we present the loss calculation of our method. The loss is specified in Eq. (4):

$$L = L_{yolo} + k_1 L_{center} + k_2 L_{distance} + k_3 L_{dim} + k_4 L_{orient}, \quad (4)$$

where $L_{yolo}$ is the loss for 2D detection and class prediction, and $L_{center}$, $L_{distance}$, $L_{dim}$ and $L_{orient}$ are the losses for the 3D object center, distance, dimensions and orientation, respectively. $k_{i \in \{1,...,4\}}$ are the weights of the different losses. For $L_{yolo}$ loss, we use the same as YOLOv5. With $\mathbf{c}_k = [cx^k \ cy^k]$ the projection on image plane of 3D object k ground-truth center in pixels, $\mathbf{Rc}_k = [Rcx_k \ Rcy_k]$ the center of the corresponding ROI, N the number of objects and $\tilde{\mathbf{c}}_k$ the prediction from our network, the center loss is defined by Equation (5):

$$L_{center} = \text{Mean}(\frac{1}{N} \sum_{k=1}^{N} |\mathbf{c}_k - \mathbf{Rc}_k - \tilde{\mathbf{c}}_k|). \quad (5)$$

Assuming $z_k$ the ground-truth distance of an object k, N the total number of objects and $\tilde{z}_k$ the distance prediction from our method, the distance loss is then calculated using the L1 loss and is detailed in Eq. (6):

$$L_{distance} = \frac{1}{N} \sum_{k=1}^{N} |z_k - \tilde{z}_k|. \quad (6)$$

With $\mathbf{d}_k = [dx \ dy \ dz]$ the ground-truth dimensions of an object k in meters, $dd_k$ the mean dimension for the class of object k, N the number of objects and $\tilde{d}_k$ the prediction from our method, the distance loss is detailed in Eq. (7):
$$L_{dim} = Mean(\frac{1}{N} \sum_{k=1}^{N} |\mathbf{d}_k - \tilde{\mathbf{d}}_k - \mathbf{d}\mathbf{d}_k|). \qquad (7)$$

Finally, assuming $\alpha_k$ the ground-truth observed angle of object k, N the total number of objects and $\tilde{\alpha}_k$ the prediction, we use the smooth L1 loss as the orientation loss. The loss is written in Eq. (8) with $\beta = 1$:

$$L_{orient} = \frac{1}{N} \sum_{k=1}^{N} e_k, \qquad (8)$$

where

$$e_k = \begin{cases} 0.5(\alpha_k - \tilde{\alpha}_k)^2 / \beta, & \text{if } |\alpha_k - \tilde{\alpha}_k| < \beta \\ |\alpha_k - \tilde{\alpha}_k| - 0.5 \times \beta, & \text{otherwise} \end{cases}.$$

# 5 Experimental results

## 5.1 Training details

### 5.1.1 Model size

Our method can be presented in different versions with a different size for each, such as in YOLOv5. The depth and width of the network vary with the versions which result in several different learnable parameters. Thus our model L-CNN comes in 3 versions: Large, Medium, and Small. To analyze the performance in terms of 3D detection accuracy and computation time of each version, we have trained and evaluated each of them on the KITTI and GTA datasets.

### 5.1.2 Split attention model

Among the convolutional blocks composing the network, the bottleneck introduced in [34] is extensively used in YOLOv5. Improvements to these convolutional blocks have recently been introduced in [35] along with Split Attention convolutions. These new Split Attention Bottlenecks increase the performance of networks at the cost of a slight increase in computation time. To create a model with a good compromise between accuracy and computation time, we have modified the small model by replacing the bottlenecks with Split Attention Bottlenecks. This new model, called Small SA (Split Attention), is dedicated to applications on systems with limited computational resources, such as the Jetson TX2 embedded boards, without sacrificing prediction accuracy.

### 5.1.3 KITTI

We trained our method on the KITTI dataset dedicated to 3D detection on the same training and validation split defined in [36]. The training split contains 3712 images and

the validation split 3769 images. To speed up the training process and improve accuracy, we trained our models with either pre-trained weights from YOLOv5 or pre-trained weights from our model after training for 15 epochs on our GTAV dataset. We demonstrate that pre-training our model on our dataset significantly improves the accuracy of our method on the KITTI dataset. Furthermore, as in [27] we perform the training with both the left and right images of the training split of KITTI thus doubling the number of images available for the training resulting in improved performances of our method during evaluation. To make our method compatible with embedded boards such as the Nvidia Jetson TX2, we have done the training with a lowered resolution of 672 × 224. We also trained with a resolution of 1312 × 416, closer to the original resolution, to compare the accuracy of our method with the state-of-the-art method on the KITTI dataset.

### 5.1.4 GTA

Inspired by previous work on dataset creation [16] using images from the video game Grand Theft Auto V, we created our hybrid dataset of road and rail images. This new dataset allows us to overcome the problem of having a railway dataset with a ground truth rich enough to allow 3D bounding box learning for cars, pedestrians, and motorcycles. The training of our method was conducted on a split containing road and railway images (107 K images). The evaluation was then conducted on the validation split of the dataset containing 11,630 images. The training on this dataset was performed on 50 epochs.

### 5.1.5 Data augmentation

Data augmentation is very useful to improve performance of CNN models through the construction of new and different examples for training. Even though KITTI and our GTA-V dataset presents a variety of environments and conditions (in GTA-V dataset for example, we have different situations like road with clear weather, road at night, road with rainy weather, and rail with clear weather as it shown in Fig. 1), the risk of overfitting our models is still present. Data augmentation gives multiple advantages: 1. Reduce overfitting, 2. Increases the generalisation capabilities of the trained models thus yielding better results in real-world environment. This is a good way to introduce “non-linearity” into the dataset model making it more similar to the real world.

We use Mosaic data augmentation as defined in YOLOv4 [37] which consists in mixing 4 training images to improve the understanding of the spatial context of objects for our method. We also applied translation and scaling transformations to the images as further augmentation.
### 5.1.6 Hyperparameters

The hyperparameters of our models determined after many iterations of short training sessions. We use the Adam optimizer with a cyclic learning rate scheduler as described in [38]. The maximum learning rate was set to 9.4e-4 with a final learning rate of 1.8e-5. Finally, we use gradient accumulation to perform training with an effective batch size of 64.

### 5.1.7 Loss weights

Since our method is trained end-to-end, our loss function combines the loss of each task. To obtain optimal results it is necessary to find the weight ($k1, ..., k4$) of each loss. Through multiple trials we found $k1 = 0.005$, $k2 = 0.2$, $k3 = 0.0176$, $k4 = 0.01$. To avoid overfitting for 2D detection, we imposed the condition $L_{yolo} < 0.1$ which when satisfied $L_{yolo}$ is ignored in the optimization loop.

## 5.2 Evaluation

In this subsection, we present the metrics that will be used in the evaluation of our models.

### 5.2.1 3D average precision

For evaluation on the KITTI dataset, we used the official benchmark metrics such as the 3D Average Precision (AP) with 40 recall positions as presented by [39]. The official KITTI benchmark uses an IOU threshold of 0.7 for the car class and a 0.5 threshold for the person and cycle classes. We also added evaluation results for the car class with a 0.5 threshold.

### 5.2.2 2D detection

To evaluate the performance of the 2D detection, we used the metrics described by the authors of YOLOv5 on each class of the dataset. Given that the regression of the 3D bounding box parameters relies on accurate RoI and classes for the objects, evaluating the precision of the 2D detector is a necessity. The error metrics are the Average Precision (AP), the Recall (R), and the mean Average Precision ($mAP_{50}$).

### 5.2.3 Distance estimation

For our distance estimator evaluation, we used the same evaluation as the ones used for image-level depth estimation methods. The metrics used include Absolute Relative Error (Abs Rel), the Squared Relative Error (SRE), the Root Mean Square Error (RMSE), the logarithmic RMSE (log RMSE),

and the percentage of Bad Matching Pixels. Let $z_{gt}$ and $z_{pd}$ be, respectively, the ground truth and predicted distance of the object $i$, it is calculated using Eq. (9), where $\delta = 1.25^k$:

$$ \alpha_{k=[1..3]} = \max(\frac{z_{gt}}{z_{pd}}, \frac{z_{pd}}{z_{gt}}) < \delta^k. \qquad (9) $$

### 5.2.4 Dimensions

The dimension prediction evaluation is performed using the Dimension Score (DS) described by the authors of [23]. With $V_{pd}$ and $V_{gt}$ the predicted and ground truth volume of the object, the DS is computed using Eq. (10) averaged across all examples:

$$ DS = \min(\frac{V_{pd}}{V_{gt}}, \frac{V_{gt}}{V_{pd}}). \qquad (10) $$

### 5.2.5 Object center

The object center predictions were evaluated with the Center Score (CS) as described in [23]. Assuming $x$ and $y$ the projected center coordinates in pixels and $w$ and $h$ the width and the height of the 2D bounding box, CS is computed with Eq. (11):

$$ CS = (2 + \cos(\frac{x_{gt} - x_{pd}}{w_{pd}}) + \cos(\frac{y_{gt} - y_{pd}}{h_{pd}}))/4. \qquad (11) $$

### 5.2.6 Orientation

To evaluate the orientation predictions, we use the Orientation Score (OS) averaged across all examples. OS is calculated using Equation (12):

$$ OS = (1 + \cos(\alpha_{gt} - \alpha_{pd}))/2. \qquad (12) $$

### 5.2.7 Computation time

To evaluate the computation time of our models, we compute the average time necessary for a forward pass (including Non-Max-Suppression) for one single image during inference. We tested all the methods presented on an Nvidia RTX 3080 GPU. We also included computation time on an embedded Nvidia Jetson TX2 board as shown in Table 4.

## 5.3 Result analysis

We present the quantitative results of our models on the KITTI dataset using the official 3D AP metric in Table 1 for the Car class, Table 2 for the Person class, and Table 3 for the Squared Relative Error (SRE).
<table><caption>**Table 1** Quantitative results for the Car class from our evaluation on the KITTI dataset. We can see that our method benefits from being pre-trained on our GTA dataset and manages to outperform other</caption><thead><tr><th rowspan="3">Method</th><th rowspan="3">Pretrained on GTA</th><th rowspan="3">Image size</th><th colspan="3">[val]</th><th colspan="3">[val]</th><th rowspan="3">Time/img (ms)</th><th rowspan="3">Memory consumption</th></tr><tr><th colspan="3">IOU 0.7</th><th colspan="3">IOU 0.5</th></tr><tr><th>Easy</th><th>Mod</th><th>Hard</th><th>Easy</th><th>Mod</th><th>Hard</th></tr></thead><tbody><tr><td>Mono3D [17]</td><td></td><td>–</td><td>2.53</td><td>2.31</td><td>2.31</td><td>25.19</td><td>18.20</td><td>15.52</td><td>–</td><td>–</td></tr><tr><td>Multi-fusion [22]</td><td></td><td>1696x512</td><td>10.53</td><td>5.69</td><td>5.39</td><td>47.88</td><td>29.48</td><td>26.44</td><td>–</td><td>–</td></tr><tr><td>M3D-RPN [26]</td><td></td><td>1696x512</td><td>20.27</td><td>17.06</td><td>15.21</td><td>48.96</td><td>39.57</td><td>33.01</td><td>78.3</td><td>5 GB</td></tr><tr><td>GAM3D [27]</td><td></td><td>1280x288</td><td><b>23.63</b></td><td>16.16</td><td>12.06</td><td>60.92</td><td>42.18</td><td>32.02</td><td>34.2</td><td>2.1 GB</td></tr><tr><td rowspan="2">Ours (small)</td><td></td><td>672x224</td><td>7.29</td><td>5.48</td><td>5.34</td><td>44.35</td><td>31.80</td><td>29.91</td><td>1.5</td><td>1.6 GB</td></tr><tr><td>×</td><td>1312x416</td><td>15.52</td><td>13.27</td><td>13.19</td><td>48.24</td><td>36.14</td><td>31.78</td><td>4.4</td><td>1.7 GB</td></tr><tr><td rowspan="2">Ours (small SA)</td><td></td><td>1312x416</td><td>15.59</td><td>13.50</td><td>13.14</td><td>49.71</td><td>38.02</td><td>32.70</td><td>4.4</td><td>1.7 GB</td></tr><tr><td>×</td><td>672x224</td><td>13.79</td><td>11.72</td><td>11.39</td><td>44.51</td><td>32.47</td><td>27.26</td><td>3.1</td><td><b>1.5 GB</b></td></tr><tr><td rowspan="2">Ours (medium)</td><td></td><td>1312x416</td><td>10.34</td><td>10.07</td><td>9.85</td><td>23.22</td><td>20.22</td><td>19.56</td><td>6.1</td><td>1.6 GB</td></tr><tr><td>×</td><td>1312x416</td><td>15.57</td><td>13.32</td><td>13.39</td><td>51.33</td><td>38.09</td><td>33.31</td><td>6.1</td><td>1.6 GB</td></tr><tr><td rowspan="2">Ours (large)</td><td></td><td>672x224</td><td>9.42</td><td>7.96</td><td>6.52</td><td>46.22</td><td>34.66</td><td>33.01</td><td>2.7</td><td>1.7 GB</td></tr><tr><td>×</td><td>1312x416</td><td>17.96</td><td>14.27</td><td>13.73</td><td>51.36</td><td>38.28</td><td>37.41</td><td>8.2</td><td>1.9 GB</td></tr><tr><td rowspan="2">Ours (large)</td><td></td><td>1312x416</td><td>17.63</td><td>15.04</td><td>14.68</td><td>53.95</td><td>44.73</td><td>39.86</td><td>8.2</td><td>1.9 GB</td></tr><tr><td>×</td><td>672x224</td><td>12.52</td><td>10.52</td><td>9.75</td><td>53.27</td><td>40.58</td><td>35.34</td><td>4</td><td>2 GB</td></tr><tr><td rowspan="2">Ours (large)</td><td></td><td>1312x416</td><td>21.07</td><td>16.07</td><td>15.68</td><td>55.69</td><td>41.89</td><td>40.46</td><td>11.2</td><td>2.15 GB</td></tr><tr><td>×</td><td>1312x416</td><td>23.26</td><td><b>18.46</b></td><td><b>16.16</b></td><td><b>60.93</b></td><td><b>48.45</b></td><td><b>42.72</b></td><td>11.2</td><td>2.15 GB</td></tr></tbody></table>

state-of-the-art methods, especially for moderate and hard targets. The Car class on the KITTI dataset represents ~ 82% of all the dataset objects

<table><caption>**Table 2** Quantitative results for the Person class from our evaluation on the KITTI dataset. We can see that our method benefits from being pre-trained on our GTA dataset. The Person class on the KITTI dataset represents ~ 13% of all the dataset objects.</caption><thead><tr><th rowspan="3">Method</th><th rowspan="3">Pretrained on GTA</th><th rowspan="3">Image size</th><th colspan="3">[val]</th><th rowspan="3">Time/img (ms)</th><th rowspan="3">Memory consumption</th><th rowspan="3"># Parameters</th></tr><tr><th colspan="3">IOU 0.5</th></tr><tr><th>Easy</th><th>Mod</th><th>Hard</th></tr></thead><tbody><tr><td rowspan="3">Ours (small)</td><td></td><td>672x224</td><td>5.30</td><td>4.53</td><td>4.59</td><td><b>1.5</b></td><td>1.6 GB</td><td rowspan="3">7.3 M</td></tr><tr><td>×</td><td>1312x416</td><td>4.66</td><td>4.23</td><td>3.95</td><td>4.4</td><td>1.7 GB</td></tr><tr><td></td><td>1312x416</td><td>6.91</td><td>5.45</td><td>5.03</td><td>4.4</td><td>1.7 GB</td></tr><tr><td rowspan="3">Ours (small SA)</td><td></td><td>672x224</td><td>9.96</td><td>9.09</td><td>9.09</td><td>3.1</td><td><b>1.5 GB</b></td><td rowspan="3">9.7M</td></tr><tr><td>×</td><td>1312x416</td><td>4.55</td><td>4.55</td><td>4.55</td><td>6.1</td><td>1.6 GB</td></tr><tr><td></td><td>1312x416</td><td>11.75</td><td>10.85</td><td>10.56</td><td>6.1</td><td>1.6 GB</td></tr><tr><td rowspan="3">Ours (medium)</td><td></td><td>672x224</td><td>6.51</td><td>6.18</td><td>5.90</td><td>2.7</td><td>1.7 GB</td><td rowspan="3">21.6M</td></tr><tr><td>×</td><td>1312x416</td><td>7.94</td><td>5.93</td><td>5.44</td><td>8.2</td><td>1.9 GB</td></tr><tr><td></td><td>1312x416</td><td>8.66</td><td>6.19</td><td>6.01</td><td>8.2</td><td>1.9 GB</td></tr><tr><td rowspan="3">Ours (large)</td><td></td><td>672x224</td><td><b>11.76</b></td><td><b>11.36</b></td><td><b>10.62</b></td><td>4</td><td>2 GB</td><td rowspan="3">47.5M</td></tr><tr><td>×</td><td>1312x416</td><td>8.55</td><td>7.00</td><td>6.77</td><td>11.2</td><td>2.15 GB</td></tr><tr><td></td><td>1312x416</td><td>11.58</td><td>8.59</td><td>7.99</td><td>11.2</td><td>2.15 GB</td></tr></tbody></table>

for the Cycle class. We present our results along with state-of-the-art methods for comparison. These results show that our approach, although less complex than state-of-the-art methods, has accuracy comparable or even superior to them. Our model is also significantly faster than all other tested approaches, with a computation time of 11.2ms per image on our Large model when inferring on an RTX 3080 GPU. Our method is even able to achieve higher accuracy using

pre-trained weights on our GTAV dataset, which shows the potential of video game datasets to improve 3D object detection methods on real datasets. Thanks to this, our “Large” model can outperform state-of-the-art methods in moderate and difficult scenarios. Finally, we also conducted experiments on a Jetson TX2 embedded board to confirm the applicability of our smallest model for real-time tasks for embedded applications. We present the results in Table 4.
<table><caption>**Table 3** Quantitative results for the Cycle class from our evaluation on the KITTI dataset. We can see that our method benefits from being pre-trained on our GTA dataset. The Cycle class on the KITTI dataset represents ~ 5% of all the dataset objects</caption><thead><tr><th rowspan="3">Method</th><th rowspan="3">Pretrained on GTA</th><th rowspan="3">Image size</th><th colspan="3">[val]</th><th rowspan="3">Time/img (ms)</th><th rowspan="3">Memory consumption</th><th rowspan="3"># Parameters</th></tr><tr><th colspan="3">IOU 0.5</th></tr><tr><th>Easy</th><th>Mod</th><th>Hard</th></tr></thead><tbody><tr><td rowspan="3">Ours (small)</td><td rowspan="3">×</td><td>672 × 224</td><td>3.68</td><td>3.12</td><td>2.27</td><td><b>1.5</b></td><td>1.6 GB</td><td rowspan="3">7.3M</td></tr><tr><td>1312 × 416</td><td>4.66</td><td>4.23</td><td>3.95</td><td>4.4</td><td>1.7 GB</td></tr><tr><td>1312 × 416</td><td>14.56</td><td>11.83</td><td>11.64</td><td>4.4</td><td>1.7 GB</td></tr><tr><td rowspan="3">Ours (small SA)</td><td rowspan="3">×</td><td>672 × 224</td><td>4.39</td><td>2.65</td><td>2.20</td><td>3.1</td><td><b>1.5 GB</b></td><td rowspan="3">9.7M</td></tr><tr><td>1312 × 416</td><td>0.98</td><td>0.88</td><td>0.82</td><td>6.1</td><td>1.6 GB</td></tr><tr><td>1312 × 416</td><td>9.39</td><td>6.39</td><td>6.53</td><td>6.1</td><td>1.6 GB</td></tr><tr><td rowspan="3">Ours (medium)</td><td rowspan="3">×</td><td>672 × 224</td><td>13.34</td><td>10.37</td><td>10.39</td><td>2.7</td><td>1.7 GB</td><td rowspan="3">21.6 M</td></tr><tr><td>1312 × 416</td><td>10.75</td><td>7.65</td><td>7.53</td><td>8.2</td><td>1.9 GB</td></tr><tr><td>1312 × 416</td><td>18.12</td><td><b>13.58</b></td><td><b>12.02</b></td><td>8.2</td><td>1.9 GB</td></tr><tr><td rowspan="3">Ours (large)</td><td rowspan="3">×</td><td>672 × 224</td><td>12.88</td><td>10.26</td><td>10.31</td><td>4</td><td>2 GB</td><td rowspan="3">47.5 M</td></tr><tr><td>1312 × 416</td><td><b>14.88</b></td><td>10.33</td><td>8.21</td><td>11.2</td><td>2.15 GB</td></tr><tr><td>1312 × 416</td><td>11.08</td><td>5.94</td><td>5.71</td><td>11.2</td><td>2.15 GB</td></tr></tbody></table>

<table><caption>**Table 4** Computing time on the Nvidia Jetson TX2 embedded board. Our results show that our new Small SA model is the most appropriate for real-time embedded applications with 60ms/img (17 fps)</caption><thead><tr><th>Method</th><th>Image size</th><th>Time/img (ms)</th><th># Parameters</th></tr></thead><tbody><tr><td rowspan="2">Ours (small)</td><td>672 × 224</td><td>70</td><td rowspan="2">7.3 M</td></tr><tr><td>1312 × 416</td><td>130</td></tr><tr><td rowspan="2">Ours (small SA)</td><td>672 × 224</td><td><b>60</b></td><td rowspan="2">9.7M</td></tr><tr><td>1312 × 416</td><td>200</td></tr><tr><td rowspan="2">Ours (medium)</td><td>672 × 224</td><td>143</td><td rowspan="2">21.6M</td></tr><tr><td>1312 × 416</td><td>359</td></tr><tr><td rowspan="2">Ours (large)</td><td>672 × 224</td><td>199</td><td rowspan="2">47.5M</td></tr><tr><td>1312 × 416</td><td>613</td></tr></tbody></table>

Our new SA Small model results show that it performs the same, or worse when not using pre-trained weights, than the Small model when training at full resolution (1312 × 416). However, when using a smaller resolution (672 × 224) it significantly outperforms the Small model. It is a particularly interesting model for applications on systems where the computational capabilities are limited since its memory requirements are lower and it runs at 3.1ms/img while having greater accuracy than the Small and Medium models trained at the same resolution. Our tests with the Jetson TX2 also show that it is the fastest of our methods on this platform. The quantitative results of our models on both datasets for all classes along with the results of our baseline based on our previous multi-stage approach based on YOLOv5 are presented in Table 5 and in Fig. 5. We can see that we achieved significant accuracy gains by switching from a multi-stage approach based on YOLOv3 [40] from our previous approach [33] to an anchor-based approach

using YOLOv5. We conducted the experiments using either the ROIs from YOLOv3 predictions or the ground truth for the feature alignment necessary for the regression of the 3D bounding box parameters. This to highlight the problem posed by ROIs variation in the 3D parameter regression in multi-stage approaches. We show that our new single-stage approach significantly outperforms our previous method in both cases, especially for depth estimation and orientation.

We also showcase the various models from our current approach (small, SA small, medium, large) and we can see that the precision between the different models for 2D detection, Dimensions, Center, and Orientation does not significantly benefit from the heavier models while Distance estimation improves when using larger models. Qualitative results on both KITTI and GTA datasets are presented in Fig. 6.

# 6 Conclusion and outlook

In this paper, we presented our new anchor-based 3D object detection method from single-image input using a modified YOLOv5 network. Along with this method, we also presented our new virtual dataset based on the video game GTAV for both road and rail environments, filling the gap in railway datasets for 3D object detection and localization. Through our work, we showed that state-of-the-art precision on the KITTI benchmark could be reached by adopting transfer learning from models trained on our new dataset while being the fastest method available for 3D object detection from images. Our approach is particularly well suited for embedded use, as demonstrated by our tests on the Nvidia Jetson TX2 embedded board. We also provided
<figure><img src="image_10.png" /><figcaption>**Fig. 5** Depth RMSE error obtained by our different models along the baseline multi-stage [33] method for each class on the KITTI and GTA datasets. Our new method improves accuracy on the KITTI dataset significantly when we use weights pre-trained on our new GTA dataset.</figcaption></figure>

With these weights, our new model with Split Attention (SA) convolutions is able to reach the same performances as our medium-sized model
<table><caption>**Table 5** Quantitative results from our evaluation on both the KITTI and our GTA datasets. Results from our previous method are displayed along with our new method. Our new method, significantly outperforms our previous approach, especially in terms of depth estimation accuracy</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Classes</th><th rowspan="2">Method</th><th colspan="3">2D detection</th><th colspan="3">Distance</th><th colspan="3">Dimensions</th><th colspan="3">Orientation</th></tr><tr><th>AP</th><th>R</th><th>mAP@0.5</th><th>Abs Rel</th><th>SRE</th><th>RMSE (m)</th><th>log RMSE</th><th>α<sub>1</sub></th><th>α<sub>2</sub></th><th>α<sub>3</sub></th><th>DS</th><th>Center</th><th>CS</th><th>OS</th></tr></thead><tbody><tr><td rowspan="12">KITTI</td><td rowspan="12">Car</td><td>Baseline (w GT RoIs)</td><td>-</td><td>-</td><td>-</td><td>0.096</td><td>3.012</td><td>3.05</td><td>0.1800</td><td>0.941</td><td>0.980</td><td>0.988</td><td>0.872</td><td><strong>0.981</strong></td><td>0.781</td><td></td></tr><tr><td>Baseline (w/o GT RoIs)</td><td>0.558</td><td><strong>0.879</strong></td><td>0.783</td><td>0.163</td><td>1.290</td><td>5.28</td><td>0.2710</td><td>0.839</td><td>0.948</td><td>0.971</td><td>0.867</td><td>0.962</td><td>0.783</td><td></td></tr><tr><td>Ours (small)</td><td>0.886</td><td>0.814</td><td>0.790</td><td>0.051</td><td>0.109</td><td>2.00</td><td>0.0782</td><td>0.987</td><td>0.997</td><td><strong>0.999</strong></td><td>0.879</td><td>0.960</td><td>0.907</td><td></td></tr><tr><td>Ours (small + GTA)</td><td>0.882</td><td>0.818</td><td><strong>0.796</strong></td><td>0.050</td><td>0.099</td><td>1.83</td><td>0.0757</td><td>0.988</td><td>0.997</td><td><strong>0.999</strong></td><td>0.882</td><td>0.961</td><td>0.935</td><td></td></tr><tr><td>Ours (small SA)</td><td>0.880</td><td>0.764</td><td>0.739</td><td>0.062</td><td>0.133</td><td>1.97</td><td>0.0901</td><td>0.981</td><td>0.997</td><td><strong>0.999</strong></td><td>0.874</td><td>0.956</td><td>0.859</td><td></td></tr><tr><td>Ours (small SA + GTA)</td><td>0.901</td><td>0.777</td><td>0.755</td><td>0.047</td><td>0.086</td><td>1.65</td><td>0.0733</td><td>0.991</td><td><strong>0.998</strong></td><td><strong>0.999</strong></td><td>0.883</td><td>0.960</td><td>0.908</td><td></td></tr><tr><td>Ours (medium)</td><td>0.892</td><td>0.800</td><td>0.777</td><td>0.047</td><td>0.085</td><td>1.66</td><td>0.0755</td><td>0.988</td><td>0.997</td><td><strong>0.999</strong></td><td>0.883</td><td>0.963</td><td>0.929</td><td></td></tr><tr><td>Ours (medium + GTA)</td><td>0.887</td><td>0.822</td><td>0.798</td><td>0.045</td><td>0.083</td><td>1.69</td><td>0.0704</td><td>0.989</td><td><strong>0.998</strong></td><td><strong>0.999</strong></td><td>0.883</td><td>0.963</td><td>0.938</td><td></td></tr><tr><td>Ours (large)</td><td><strong>0.908</strong></td><td>0.787</td><td>0.787</td><td>0.042</td><td>0.072</td><td><strong>1.53</strong></td><td>0.0670</td><td><strong>0.992</strong></td><td><strong>0.998</strong></td><td><strong>0.999</strong></td><td><strong>0.885</strong></td><td>0.964</td><td><strong>0.947</strong></td><td></td></tr><tr><td>Ours (large + GTA)</td><td>0.903</td><td>0.798</td><td>0.777</td><td><strong>0.041</strong></td><td><strong>0.070</strong></td><td>1.56</td><td><strong>0.0657</strong></td><td><strong>0.992</strong></td><td><strong>0.998</strong></td><td><strong>0.999</strong></td><td><strong>0.885</strong></td><td>0.964</td><td>0.946</td><td></td></tr><tr><td rowspan="12">Person</td><td>Baseline (w GT RoIs)</td><td>-</td><td>-</td><td>-</td><td>0.098</td><td>2.01</td><td>0.1540</td><td>0.935</td><td>0.983</td><td>0.992</td><td>0.705</td><td><strong>0.993</strong></td><td>0.584</td><td></td></tr><tr><td>Baseline (w/o GT RoIs)</td><td>0.508</td><td>0.518</td><td>0.464</td><td>0.424</td><td>7.920</td><td>7.70</td><td>0.4850</td><td>0.729</td><td>0.844</td><td>0.892</td><td>0.719</td><td>0.885</td><td>0.616</td><td></td></tr><tr><td>Ours (small)</td><td>0.817</td><td>0.528</td><td>0.514</td><td>0.057</td><td>0.068</td><td>0.94</td><td>0.0800</td><td>0.979</td><td>0.998</td><td><strong>1</strong></td><td>0.729</td><td>0.949</td><td>0.750</td><td></td></tr><tr><td>Ours (small + GTA)</td><td>0.740</td><td>0.554</td><td>0.525</td><td>0.054</td><td>0.062</td><td>0.94</td><td>0.0762</td><td>0.980</td><td>0.999</td><td><strong>1</strong></td><td>0.736</td><td>0.948</td><td>0.745</td><td></td></tr><tr><td>Ours (small SA)</td><td>0.773</td><td>0.435</td><td>0.419</td><td>0.086</td><td>0.136</td><td>1.33</td><td>0.1050</td><td>0.964</td><td>0.999</td><td><strong>1</strong></td><td>0.682</td><td>0.945</td><td>0.581</td><td></td></tr><tr><td>Ours (small SA + GTA)</td><td>0.821</td><td>0.436</td><td>0.427</td><td>0.059</td><td>0.067</td><td>0.92</td><td>0.0803</td><td>0.981</td><td>0.999</td><td><strong>1</strong></td><td>0.709</td><td>0.944</td><td>0.716</td><td></td></tr><tr><td>Ours (medium)</td><td>0.828</td><td>0.527</td><td><strong>0.527</strong></td><td>0.049</td><td>0.065</td><td>0.95</td><td>0.0745</td><td>0.981</td><td>0.996</td><td><strong>1</strong></td><td>0.733</td><td>0.950</td><td><strong>0.818</strong></td><td></td></tr><tr><td>Ours (medium + GTA)</td><td>0.825</td><td>0.532</td><td>0.517</td><td>0.051</td><td>0.055</td><td>0.86</td><td>0.0731</td><td>0.982</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.733</td><td>0.950</td><td>0.812</td><td></td></tr><tr><td>Ours (large)</td><td><strong>0.851</strong></td><td>0.514</td><td>0.501</td><td>0.048</td><td>0.050</td><td>0.83</td><td>0.0689</td><td>0.987</td><td>0.999</td><td><strong>1</strong></td><td>0.722</td><td>0.951</td><td>0.772</td><td></td></tr><tr><td>Ours (large + GTA)</td><td>0.817</td><td><strong>0.539</strong></td><td>0.519</td><td><strong>0.046</strong></td><td><strong>0.049</strong></td><td><strong>0.82</strong></td><td><strong>0.0667</strong></td><td><strong>0.993</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td><strong>0.736</strong></td><td>0.950</td><td>0.738</td><td></td></tr><tr><td rowspan="12">Cyclist</td><td>Baseline (w GT RoIs)</td><td>-</td><td>-</td><td>-</td><td>0.098</td><td>3.57</td><td>0.1500</td><td>0.940</td><td>0.979</td><td>0.988</td><td>0.809</td><td><strong>0.997</strong></td><td>0.738</td><td></td></tr><tr><td>Baseline (w/o GT RoIs)</td><td>0.342</td><td>0.371</td><td>0.253</td><td>1.040</td><td>30.900</td><td>17.40</td><td>0.7970</td><td>0.415</td><td>0.622</td><td>0.719</td><td>0.812</td><td>0.678</td><td>0.542</td><td></td></tr><tr><td>Ours (small)</td><td>0.568</td><td>0.400</td><td>0.342</td><td>0.058</td><td>0.075</td><td>1.25</td><td>0.0755</td><td>0.989</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.787</td><td>0.960</td><td>0.851</td><td></td></tr><tr><td>Ours (small + GTA)</td><td>0.589</td><td>0.403</td><td>0.364</td><td>0.056</td><td>0.099</td><td>1.57</td><td>0.0780</td><td>0.986</td><td>0.997</td><td><strong>1</strong></td><td><strong>0.827</strong></td><td>0.964</td><td>0.857</td><td></td></tr><tr><td>Ours (small SA)</td><td>0.490</td><td>0.274</td><td>0.215</td><td>0.099</td><td>0.205</td><td>1.88</td><td>0.1140</td><td>0.959</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.800</td><td>0.952</td><td>0.868</td><td></td></tr><tr><td>Ours (small SA + GTA)</td><td>0.655</td><td>0.348</td><td>0.321</td><td>0.053</td><td>0.072</td><td>1.07</td><td>0.0746</td><td>0.987</td><td>0.997</td><td><strong>1</strong></td><td>0.819</td><td>0.961</td><td>0.919</td><td></td></tr><tr><td>Ours (medium)</td><td>0.692</td><td>0.476</td><td>0.436</td><td>0.051</td><td>0.083</td><td>1.49</td><td>0.0664</td><td>0.995</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.824</td><td>0.962</td><td>0.865</td><td></td></tr><tr><td>Ours (medium + GTA)</td><td>0.654</td><td><strong>0.492</strong></td><td><strong>0.460</strong></td><td>0.049</td><td>0.083</td><td>1.54</td><td>0.0665</td><td>0.995</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.822</td><td>0.961</td><td>0.888</td><td></td></tr><tr><td>Ours (large)</td><td><strong>0.780</strong></td><td>0.429</td><td>0.407</td><td><strong>0.042</strong></td><td>0.059</td><td>1.25</td><td><strong>0.0571</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.825</td><td>0.963</td><td>0.878</td><td></td></tr><tr><td>Ours (large + GTA)</td><td>0.629</td><td>0.448</td><td>0.404</td><td>0.046</td><td><strong>0.058</strong></td><td><strong>1.18</strong></td><td>0.0622</td><td>0.998</td><td>0.998</td><td><strong>1</strong></td><td>0.808</td><td>0.965</td><td><strong>0.913</strong></td><td></td></tr></tbody></table>

<figure><img src="image_11.png" /></figure>
<table><caption>Table 5 (continued)</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Classes</th><th rowspan="2">Method</th><th colspan="2">2D detection</th><th colspan="2">Distance</th><th colspan="2">Dimensions</th><th colspan="2">Orientation</th></tr><tr><th>AP</th><th>R</th><th>Abs Rel</th><th>SRE</th><th>RMSE (m)</th><th>log RMSE</th><th>α<sub>1</sub></th><th>α<sub>2</sub></th><th>α<sub>3</sub></th><th>DS</th><th>Center</th><th>CS</th><th>OS</th></tr></thead><tbody><tr><td rowspan="10">GTA</td><td rowspan="5">Car</td><td>Baseline (w GT RoIs)</td><td>—</td><td>—</td><td>—</td><td>0.055</td><td>0.385</td><td>4.74</td><td>0.0761</td><td>0.990</td><td>0.999</td><td>0.999</td><td>0.860</td><td><strong>0.999</strong></td><td>0.993</td></tr><tr><td>Baseline (w/o GT RoIs)</td><td>0.477</td><td>0.915</td><td>0.778</td><td>0.129</td><td>2.280</td><td>9.59</td><td>0.2170</td><td>0.873</td><td>0.938</td><td>0.965</td><td>0.812</td><td>0.894</td><td>0.905</td></tr><tr><td>Ours (small)</td><td>0.921</td><td>0.942</td><td>0.957</td><td>0.062</td><td>0.222</td><td>3.40</td><td>0.0749</td><td>0.998</td><td><strong>1</strong></td><td><strong>1</strong></td><td><strong>0.876</strong></td><td>0.982</td><td><strong>0.997</strong></td></tr><tr><td>Ours (small SA)</td><td>0.906</td><td>0.937</td><td>0.947</td><td>0.041</td><td>0.134</td><td>2.75</td><td>0.0542</td><td>0.998</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.873</td><td>0.980</td><td>0.995</td></tr><tr><td>Ours (medium)</td><td><strong>0.930</strong></td><td>0.951</td><td><strong>0.964</strong></td><td>0.033</td><td>0.086</td><td>2.18</td><td>0.0440</td><td><strong>0.999</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.865</td><td>0.984</td><td><strong>0.997</strong></td></tr><tr><td rowspan="5">Motorcycle</td><td>Ours (large)</td><td><strong>0.930</strong></td><td>0.905</td><td>0.919</td><td><strong>0.029</strong></td><td><strong>0.075</strong></td><td><strong>2.05</strong></td><td><strong>0.0402</strong></td><td><strong>0.999</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.867</td><td>0.984</td><td><strong>0.997</strong></td></tr><tr><td>Baseline (w GT RoIs)</td><td>—</td><td>—</td><td>—</td><td>0.062</td><td>0.266</td><td>3.84</td><td>0.0753</td><td><strong>1</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td><strong>0.918</strong></td><td><strong>1.00</strong></td><td>0.967</td></tr><tr><td>Baseline (w/o GT RoIs)</td><td>0.614</td><td>0.764</td><td>0.725</td><td>0.186</td><td>3.580</td><td>13.70</td><td>0.2810</td><td>0.691</td><td>0.878</td><td>0.967</td><td>0.901</td><td>0.689</td><td>0.909</td></tr><tr><td>Ours (small)</td><td>0.937</td><td>0.925</td><td>0.935</td><td>0.061</td><td>0.190</td><td>2.99</td><td>0.0755</td><td>0.998</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.877</td><td>0.976</td><td>0.993</td></tr><tr><td>Ours (small SA)</td><td>0.935</td><td>0.911</td><td>0.918</td><td>0.041</td><td>0.116</td><td>2.42</td><td>0.0548</td><td>0.998</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.871</td><td>0.975</td><td>0.993</td></tr><tr><td rowspan="5">Person</td><td>Ours (medium)</td><td>0.934</td><td><strong>0.938</strong></td><td><strong>0.944</strong></td><td>0.034</td><td>0.086</td><td>2.14</td><td>0.0463</td><td>0.998</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.876</td><td>0.980</td><td><strong>0.995</strong></td></tr><tr><td>Ours (large)</td><td><strong>0.944</strong></td><td>0.933</td><td>0.943</td><td><strong>0.029</strong></td><td><strong>0.070</strong></td><td><strong>1.95</strong></td><td><strong>0.0414</strong></td><td><strong>0.999</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.876</td><td>0.981</td><td>0.994</td></tr><tr><td>Baseline (w GT RoIs)</td><td>—</td><td>—</td><td>—</td><td>0.116</td><td>0.612</td><td>4.56</td><td>0.1590</td><td>0.877</td><td>0.966</td><td>1.00</td><td><strong>0.963</strong></td><td><strong>0.997</strong></td><td>0.856</td></tr><tr><td>Baseline (w/o GT RoIs)</td><td>0.263</td><td>0.659</td><td>0.488</td><td>0.372</td><td>10.500</td><td>15.10</td><td>0.4530</td><td>0.620</td><td>0.834</td><td>0.903</td><td>0.961</td><td>0.812</td><td>0.735</td></tr><tr><td>Ours (small)</td><td><strong>0.932</strong></td><td>0.814</td><td>0.834</td><td>0.049</td><td>0.170</td><td>2.92</td><td>0.0674</td><td>0.996</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.872</td><td>0.970</td><td><strong>0.957</strong></td></tr><tr><td rowspan="5">Ours</td><td>Ours (small SA)</td><td>0.913</td><td>0.790</td><td>0.798</td><td>0.038</td><td>0.127</td><td>2.60</td><td>0.0538</td><td><strong>0.998</strong></td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.868</td><td>0.970</td><td>0.920</td></tr><tr><td>Ours (medium)</td><td>0.931</td><td>0.832</td><td>0.844</td><td>0.030</td><td>0.088</td><td><strong>2.16</strong></td><td>0.0452</td><td>0.997</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.878</td><td>0.974</td><td>0.955</td></tr><tr><td>Ours (large)</td><td>0.929</td><td><strong>0.833</strong></td><td><strong>0.847</strong></td><td><strong>0.028</strong></td><td><strong>0.087</strong></td><td>2.17</td><td><strong>0.0435</strong></td><td>0.997</td><td><strong>1</strong></td><td><strong>1</strong></td><td>0.872</td><td>0.975</td><td>0.953</td></tr></tbody></table>
<figure><img src="image_12.png" /><figcaption>**Fig. 6** Qualitative results of our methods on both KITTI and GTA datasets. These images are extracted from the validation split of each dataset. We display the results of both our best model (large pre-trained on GTAV) and our new lightweight model (L-CNN) for embedded platforms (Small SA pre-trained on GTAV). 4 top lines:</figcaption></figure>

results obtained under GTAV dataset (left column: ground truth, middle column: Small SA prediction, right column: large model pre-diction), 4 bottom lines: results obtained under KITTI dataset (left column: ground truth, middle column: Small SA prediction, right column: large model prediction)

an extensive evaluation of our approach, along with our previous 2 stage network on both the KITTI and our own GTAV road/railway virtual dataset. These results showed that our method is suitable for railway applications in a virtual environment, yet we still need to prove the applicability of our light-weight network under real conditions. Our fastest model (Small-SA) has also an important gap in precision with our largest model (Large). To address these

issues, we first plan to acquire our railway dataset with real-world images (as a multimodal dataset provided by 2 cameras, 1 LiDAR, 1 Radar, 1 GPS/IMU) to further validate our approach. To improve the precision of our smallest model (Small-SA) we aim to use knowledge distillation to improve its precision and thus reduce the gap in precision without augmenting the computation time. We also aim to develop a lightweight 3D tracking algorithm and integrate it into our

<figure><img src="image_13.png" /></figure>
method to provide a complete method for road and rail safety and further improve detection accuracy using Multi-Task Learning (MTL).

**Acknowledgements** This research is supported by SEGULA Technologies and the M2SINUM project (This project is co-financed by the European Union with the European Regional Development Fund (ERDF, 18P03390/18E01750/18P02733) and by the Haute-Normandie Regional Council via the M2SINUM project). We would like to thank SEGULA Technologies for their collaboration and the research engineers and technicians of the IRSEEM LNA (Autonomous Navigation Laboratory) for their support. Finally, this work was carried out in part on computing resources provided by CRIANN (Centre Régional Informatique et d'Applications Numériques de Normandie, Normandy, France).

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

## References

1. Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., Zhang, B.: More diverse means better: multimodal deep learning meets remote-sensing imagery classification. IEEE Trans. Geosci. Remote Sens. 59(5), 4340–4354 (2020)

2. Hong, D., Han, Z., Yao, J., Gao, L., Zhang, B., Plaza, A., Chanussot, J.: Spectralformer: Rethinking hyperspectral image classification with transformers. IEEE Trans. Geosci. Remote Sens. (2021)

3. Jocher, G., Stoken, A., J. B. et al., ultralytics/yolov5: v4.0 - nn.SiLU() activations, Weights & Biases logging, PyTorch Hub integration (2021). [Online]. Available: https://doi.org/10.5281/zenodo.4418161

4. Cordts, M., Omran, M., Ramos, S., Scharwächter, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset. In: CVPR Workshop on the Future of Datasets in Vision, vol. 2 (2015)

5. Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015)

6. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)

7. Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: “nuscenes: A multimodal dataset for autonomous driving,” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11 621–11 631, (2020)

8. Singh, G., Akrigg, S., Di Maio, M., Fontana, V., Alitappeh, R. J., Saha, S., Jeddisaravi, K., Yousefi, F., Culley, J., Nicholson, T., et al.: Road: The road event awareness dataset for autonomous driving. arXiv preprint arXiv:2102.11585, (2021)

9. Chang, M.-F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., et al.:

Argoverse: 3d tracking and forecasting with rich maps. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8748–8757 (2019)

10. Kesten, R., Usman, M., Houston, J., Pandya, T., Nadhamuni, K., Ferreira, A., Yuan, M., Low, B., Jain, A., Ondruska, P., et al.: Lyft level 5 av dataset 2019. https://level-5.global/data/ (2019)

11. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446–2454 (2020)

12. Zendel, O., Murschitz, M., Zeilinger, M., Steininger, D., Abbasi, S., Beleznai, C.: Railsem19: A dataset for semantic rail scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0 (2019)

13. Harb, J., Rébéna, N., Chosidow, R., Roblin, G., Potarusov, R., Hajri, H.: “Frsign: A large-scale traffic light dataset for autonomous trains,” CoRR, vol. abs/2002.05665, (2020). [Online]. Available: arXiv:2002.05665

14. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning, pp. 1–16, (2017)

15. Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A. M.: The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234–3243 (2016)

16. Richter, S. R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: European Conference on Computer Vision (ECCV), ser. LNCS, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds., vol. 9906. Springer International Publishing, pp. 102–118, (2016)

17. Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., Urtasun, R.: Monocular 3d object detection for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2147–2156 (2016)

18. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp. 1440–1448 (2015)

19. Mousavian, A., Anguelov, D., Flynn, J., Kosecka, J.: 3d bounding box estimation using deep learning and geometry. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7074–7082 (2017)

20. Cai, Z., Fan, Q., Feris, R. S., Vasconcelos, N.: A unified multi-scale deep convolutional neural network for fast object detection (2016)

21. Chabot, F., Chaouch, M., Rabarisoa, J., Teuliere, C., Chateau, T.: Deep manta: a coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2040–2049 (2017)

22. Xu, B., Chen, Z.: Multi-level fusion based 3d object detection from monocular images. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2345–2353 (2018)

23. Hu, H.-N., Cai, Q.-Z., Wang, D., Lin, J., Sun, M., Krahenbuhl, P., Darrell, T., Yu, F.: Joint monocular 3d vehicle detection and tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5390–5399 (2019)

24. Liu, Z., Wu, Z., Tóth, R.: Smoke: single-stage monocular 3d object detection via keypoint estimation (2020)

25. Qin, Z., Wang, J., Lu, Y.: Monogrnet: a general framework for monocular 3d object detection (2021)

26. Brazil, G., Liu, X.: M3d-rpn: Monocular 3d region proposal network for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9287–9296 (2019)
27. Liu, Y., Yixuan, Y., Liu, M.: Ground-aware monocular 3d object detection for autonomous driving. IEEE Robot Autom. Lett. 6(2), 919–926 (2021)

28. Li, P., Zhao, H., Liu, P., Cao, F.: Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving (2020)

29. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)

30. Yu, F., Wang, D., Shelhamer, E., Darrell, T.: Deep layer aggregation (2019)

31. UM and F. C. for Autonomous Vehicles (2021) umautobots/gta-visionexport. https://github.com/umautobots/GTAVisionExport, Online; Accessed July (2021)

32. Jotrius. (2015) Railroad engineer. https://www.gta5-mods.com/scripts/railroad-engineer, Online; Accessed July (2021)

33. Mauri, A., Khemmar, R., Decoux, B., Haddad, M., Boutteau, R.: Real-time 3d multi-object detection and localization based on deep learning for road and railway smart mobility. J Imaging 7(8), 145 (2021)

34. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)

35. Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Zhang, Z., Lin, H., Sun, Y., He, T., Mueller, J., Manmatha, R. Li, M., Smola, A. J.: Resnest: Split-attention networks. arXiv:2004.08955

36. Chen, X., Kundu, K., Zhu, Y., Berneshawi, A. G., Ma, H., Fidler, S., Urtasun, R.: 3d object proposals for accurate object class detection. In: Advances in Neural Information Processing Systems. Citeseer, pp. 424–432 (2015)

37. Bochkovskiy, A., Wang, C., Liao, H. M.: Yolov4: Optimal speed and accuracy of object detection. arXiv:2004.10934

38. Smith, L. N., Topin, N.: Super-convergence: Very fast training of neural networks using large learning rates. In: Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006. International Society for Optics and Photonics, p. 1100612 (2019)

39. Simonelli, A., Bulo, S. R., Porzi, L., López-Antequera, M., Kontschieder, P.: Disentangling monocular 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1991–1999 (2019)

40. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv:1804.02767 (2018)

**Publisher's Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Antoine Mauri is currently a CIFRE (Convention Industrielle de Formation par la REcherche) PhD Student with the IRSEEM, Normandy University of Rouen, France and SEGULA Technologies. He obtained an Engineer degree in Robotics and Computer Vision, from Telecom

Physique Strasbourg in 2019 along with a Master IRIV (Imagerie, Robotique, Ingenierie pour le Vivant). His research interest includes Artificial Intelligence, deep learning, and computer vision for Road and Rail Smart Mobility.

Redouane Khemmar is currently associate professor with the ESIGELEC, Normandy University of Rouen, France. He received his European Master of Science in Computer Vision from University of Poitiers (France) and his PhD in Computer Vision from the University of Strasbourg (France). For two years, he was with the University of Strasbourg as an associate teacher researcher, before moving to the industry, where he worked as engineer and project manager in Jouve, Alten, and Thales companies. After, he joined IRSEEM/ESIGELEC High Engineer School (Normandy University) as an associate professor in embedded systems and computer vision, and the “Instrumentation, Computer Sciences and Systems” research team in the IRSEEM laboratory. His research interests include mobile robotics, environment perception, and computer vision dedicated to autonomous vehicles and smart mobility.

Benoît Decoux is currently with the ESIGELEC, Normandy University of Rouen, France. He obtained his PhD from the University of Rouen in 1995. From 2008 to 2012, he has worked as a research engineer in the Kelenn Technology Company, on industrial applications of optical character recognition. Since 2012, he works as a teacher and researcher, and his research interests include embedded systems, computer vision and artificial intelligence.

Madjid Haddad is a Research and Innovation Manager at SEGULA Technologies. After completing a Master's degree in Mechanical Engineering at the University of Toulouse III, he has carried out his PhD in Mechanical Engineering at the University of Toulouse III. Afterwards, he did a post doctorate in Mechanical Engineering at the University of Technology of Compiègne. He then joined the SEGULA Technologies as Research and Innovation Manager.

Rémi Boutteau received his engineering degree from the IMT Lille Douai and his MSc degree in Computer Science from the University of Lille in 2006. In 2010, he received his PhD degree from the University of Rouen Normandy for works related to Computer Vision (catadioptric sensors, 3D reconstruction, Structure-from-Motion). From 2009 to 2020, he was an Associate Professor at the ESIGELEC engineering school and a researcher in the IRSEEM research institute. In 2018, he obtained the HDR (French Habilitation to supervise research) from the University of Rouen Normandy for his research on autonomous vehicles localization. Since 2020, he is a Full Professor at University of Rouen Normandy within the STI team (Intelligent Transportation System) at the LITIS Lab (IT Laboratory, Information Processing and Systems). His research interests are perception, localization and computer vision dedicated to autonomous vehicles.