# Exploring augmentation strategies in mixed reality for autonomous driving with depth cameras

Imane Argui<fnref n="1" />, Maxime Gueriau<fnref n="1" />, and Samia Ainouz<fnref n="1" />

**Acknowledgments:** This work was supported by the ANR RAIMo under grant reference ANR-20-CHIA-0021.

Transactions of the Institute of Measurements and Control

04(7):??–22

©The Author(s) 2016

Reprints and permission:

sagepub.co.uk/journalsPermissions.nav

DOI: 10.1177/01423312241296

www.sagepub.com/

SAGE

<footnote n="1">¹Normandie Univ., INSA Rouen, UNIROUEN, UNIHAVRE, LITIS, 76000 Rouen, France</footnote>

<footnote>**Corresponding author:**</footnote>

<footnote>Imane Argui, Normandie Univ., INSA Rouen, UNIROUEN, UNIHAVRE, LITIS, 76000 Rouen, France.</footnote>

<footnote>Email: imane.argui@insa-rouen.fr</footnote>
Abstract

One significant challenge in improving autonomous driving algorithms is the lack of diverse real-world data. Moreover, transferring models from simulation to reality faces the reality gap problem. This study addresses this issue by developing an augmentation technique for mixed-reality environments, aimed at improving the testing and training of autonomous vehicles. Tested offline, it lays the groundwork for future online applications. The methodology focuses on creating virtual depth images using a virtual camera and applying an augmentation strategy to the KITTI dataset. This creates a mixed-reality representation by combining virtual and real depth maps, leveraging depth information in the fusion process. The outcomes of this process are notably effective, achieving a balance between virtual and real-world aspects. This fusion method adeptly combines elements from both environments, maintaining the quality of the images. The novel contributions of this work include a detailed augmentation strategy that seamlessly integrates virtual objects into real depth maps, accounting for occlusions and ensuring realistic depth representations. Additionally, this work demonstrates the feasibility of generating a large dataset using the proposed method, significantly expanding the available data for training autonomous driving models. The use of metrics such as SSIM, PSNR, and MAE, alongside object detection models such as Faster RCNN, provides a complete evaluation of both quantitative and qualitative aspects. The results demonstrate the quality of the augmented images, setting a foundation for potential online applications. The proposed strategy enables the generation of larger dataset and facilitates safe, effective training in scenarios considered too risky or challenging to simulate accurately.

Introduction

The automotive industry is witnessing a major change with the rise of autonomous
driving. This is being driven by the constant pursuit of advanced technology, which
will eventually lead to fully autonomous vehicles [Ying T. (2024)]. The evolution of
autonomous driving systems has progressed remarkably, with significant improvements
in several important technology areas. Researchers have contributed to the field of
computer vision by improving object detection, semantic segmentation, and image
recognition techniques [Huang and Chen (2020)]. The application of machine
learning, especially deep learning techniques, has been essential in enabling vehicles
to recognize complicated patterns in data and make wise decisions in challenging
traffic situations [Muhammad et al. (2021)]. Moreover, major developments in
sensor technology have been made possible by the integration of high-resolution LiDAR and radar systems, which have improved the abilities of autonomous cars perception [Chang et al. (2019)]. The incorporation of RGBD (Red, Green, Blue, and Depth) cameras has become an important step in parallel with those developments. These cameras provide not only visual information but also depth data, giving a more comprehensive understanding of the vehicle's surroundings [Nabati (2021)]. Despite the advancements in the field of autonomous driving systems, there are still numerous challenges that need to be addressed. For instance, the lack of diversity in datasets is considered one of the major concerns [Yao et al. (2020)]. Existing datasets, which are essential for testing and training algorithms for autonomous driving, frequently do not include a wide variety of real-world circumstances. This insufficiency presents a problem because the models may not have experienced the full range of situations found on roadways. As a result, the could lack the resilience and flexibility for real-world conditions [Mao et al. (2021)]. Moreover, effective real-world deployment strategies are crucial to ensure the seamless transition of models from simulated environments to the unpredictable reality of road conditions. This task can be challenging as it requires meticulous planning and strategic implementation. Testing in simulation can be an imperative step in the development of models for autonomous driving, but it only provides a controlled environment. It does not fully capture the complexity and unpredictability of real-world driving scenarios. [Schöner (2018)]. A significant approach to bridge the gap between simulated testing and the unpredictability of the real world is mixed-reality testing. Through the integration of virtual elements in physical environments, mixed reality generates situations that are more representative of the complexities of real-world conditions [Zofka et al. (2018)]. This testing method enables to integrate vehicles, pedestrians, and other elements into a physical environment. By simulating unpredictable real-world variables within a controlled configuration, mixed-reality testing provides a more robust and flexible system for identifying and rectifying potential issues, training and testing models to face dangerous scenarios, and repeating experiments as much as necessary [Mitchell et al. (2020)].

The main focus of this work lies in vision-based autonomous systems, which depend on the identification of complex roadside elements. The use of a mixed-reality setup can enable models to learn to handle dangerous situations in safer environments. The paper presents an initial effort to construct the perception component of this framework. A fusion method is proposed to augment the perception of the vehicle using depth-sensing cameras. This method is tested offline on the Kitti dataset, which is augmented with virtual images from the Gazebo simulator. The resulting images are evaluated
using metrics such as the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSI) to assess image fidelity, noise levels, and structural integrity after the augmentation. To further evaluate the practical utility of these augmented images in real-world scenarios, state-of-the-art object detection models were applied, including Faster RCNN. This is an important way to determine whether the models can detect both real and virtual elements within a mixed-reality context. This work presented in this paper is an evolution of our previous work on augmentation strategies. The initial work aimed to explain the proposed augmentation strategy and was applied to different scenarios using various simulators [Argui et al. (2023)]. However, the testing in the previous work was conducted on a small number of images, and the evaluation was solely qualitative. In contrast, the current study focuses on a larger dataset by testing on the KITTI dataset. Furthermore, this work provides both qualitative and quantitative evaluations of the augmentation, demonstrating the scalability and effectiveness of our approach in more complex and diverse scenarios.

## Related work

In the development of autonomous vehicles, simulation testing can be considered an essential step for various reasons. First of all, researchers can test a wide range of driving scenarios, including rare or dangerous situations that would be unsafe or hard to recreate in the real world [O'Kelly et al. (2018)]. Additionally, simulation facilitates the rapid development and iteration of autonomous driving algorithms, enabling developers to quickly test and refine their algorithms without the need for extensive physical testing [Hu et al. (2023)]. While simulation offers a controlled and safe environment for testing, it is important to note that simulation environments lack the unpredictability and complexity of the real world [Tiedemann et al. (2022)]. This gap between simulated environments and real-world conditions is called the reality gap and it is due to the discrepancies between conditions in both environments [James et al. (2019)].

Another key purpose of using simulation tools is the generation of a vast amount of labeled training data. This data is required for training machine learning methods, including deep neural networks, which are the core of autonomous driving technology. Simulated environments enable the creation of diverse and extensive datasets [Carruth (2018)]. This approach ensures that the machine learning models are exposed to a wide range of driving conditions, thus allowing them to learn and adjust to different scenarios [Li et al. (2019)]. However, generating training data from simulation also presents several challenges. For instance, it is difficult to ensure that
the simulated data includes a wide range of real-world scenarios, such as traffic behaviors and rare, unexpected events. [Vargas et al. (2021)]. Furthermore, the process of generating, processing, and storing large volumes of high-fidelity simulated data requires substantial computational resources [Bai et al. (2023)].

<figure><img src="image_1.png" /><figcaption>**Figure 1.** Representation of mixed-reality [Milgram et al. (1994)]</figcaption></figure>

One approach to address these issues is the application of mixed-reality testing for autonomous driving systems. Mixed-reality, in this context, consists of combining virtual and physical elements to create a single environment that allows vehicles to learn and adapt [Milgram et al. (1994)]. Figure 1 is a representation of mixed-reality. The fusion of the two worlds (real and virtual) aims to bridge the reality gap, as it offers a more realistic representation of real-world conditions. By doing so, autonomous vehicles learn to adapt to unpredictable scenarios and handle the difficulties of real-world road conditions [Xu et al. (2023)]. Moreover, the use of mixed-reality testing can more effectively capture the complexities of traffic behavior and uncommon events that are difficult to simulate. It also has the potential to be more effective than high-fidelity simulations, which need substantial processing resources [Peintner et al. (2021)]. The use of real-world data minimizes the requirement for substantial computer resources to generate all elements of the virtual environment [Herath et al. (2017)]. By incorporating virtual features into pre-existing surroundings, mixed-reality can reduce the amount of computing power needed to complete scene generation.

Many research projects worked on developing a mixed-reality framework for training and testing autonomous driving systems. For instance, researchers developed a framework for training an agent to drive in a multi-vehicle mixed-reality framework [Mitchell et al. (2020)]. The agent learns to drive more effectively in a multi-lane environment by interacting with virtual agents. While the learning agent is a real 4-wheel robot, the other agents are simulated. The authors integrated live updates of the position and the orientation of the real robot. While the learning outcomes from this work were promising, it is important to recognize that the use of telemetry data for live updates of the robot's position and orientation may not be entirely realistic
for real-world autonomous driving. In real environments, autonomous vehicles rely
on their onboard sensors to navigate and make decisions. Therefore, it would be
more practical and relevant to focus on augmenting their sensory capabilities. In the
literature, a mixed-reality framework was proposed based on GPS data, where the
authors created a digital twin of the test vehicle using real-time GPS measurements
to position and move the vehicle accurately within the virtual environment [Ying
T. (2024)]. Additionally, other researches focused on LiDAR data to augment the
perception. The authors employed an augmentation module that uses inputs from both
virtual and real laser scans to compute an augmented laser scan. They tested the
planning and obstacle avoidance components of their technique on a real vehicle in
a parking lot [Zofka et al. (2018)]. Another work suggests augmenting the vehicle's
perception through LiDAR, enabling it to interact seamlessly with both virtual and
real elements. This advancement allows for a more integrated and realistic interaction
within mixed-reality environments, demonstrating the potential of sensor-based mixed
reality in automotive technology [Genevois et al. (2022)]. While LiDAR offers high
accuracy in 3D perception, depth cameras have the advantage of high resolution, faster
speed, rich information transmission, and low cost [Wang P. (2021)]. Although LiDAR
is known to be less susceptible to varying light conditions in the environment, its
generated point clouds tend to be sparser and less detailed compared to the rich, feature-
filled data provided by RGB images. [Liu et al. (2023)]. Prior studies have explored
the possibility of improving an agent's perception by augmenting depth information
captured by depth cameras [Argui et al. (2023)]. Building upon this foundation, the
current paper extends this research, aiming to refine and improve these results. This
is achieved by investigating a larger dataset and employing additional metrics for
evaluation, thus providing more thorough proof of the strategy's effectiveness. These
advancements are intended to ensure that the strategy is well-validated and ready for
implementation in real-world robotic applications.

Methodology

In this study, two steps were employed for the augmentation procedure. Virtual image
processing is the first step when virtual information is generated, processed, and
prepared for further actions. The second part of the technique is the augmentation
strategy that makes use of depth information for comparing and combining real images
with virtual ones. Figure 2 is a representation of the proposed methodology.
<figure><img src="image_2.png" /><figcaption>**Figure 2.** Overview of the methodology</figcaption></figure>

## Virtual images processing

The initial step before any operation is the pre-processing of virtual images obtained from the Gazebo simulator [Koenig et al. (2004)]. The aim is indeed to obtain an RGB image and its corresponding depth map. Instead of employing a complex algorithm to generate the virtual depth images, a straightforward approach is taken by using the Universal Robot Description Format (URDF) *. The URDF specifies a robot's physical and sensor characteristics in detail. The depth camera is defined in the URDF file, describing the robot's components. This camera is configured to capture both RGB and depth images. A plugin in Gazebo enables the depth camera to generate the required images by simulating the depth sensor's functionality within the virtual environment †. It is essential to note that the goal is to augment real images with virtual elements, which implies that the important aspect of these operations is the comparison of the depth of virtual images. Therefore, it is necessary to eliminate the virtual background, otherwise, it might interfere during the comparison of depth maps and potentially lead to calculation errors. To remove the background, several techniques are available, ranging from computer vision operations [Zeng et al. (2018)] to the use of deep learning models [Kang et al. (2021)]. In this paper, the decision was made to use a visual plugin in the simulator to hide the ground from the camera. As a result, it becomes possible to compare the depth of only the virtual elements with the real scene (see Figure 3). Once this is done, it becomes important to handle the vast range of intensity values of the generated virtual depth map that is [0, 2000] due to simulation constraints. To reduce the dynamic range of values in the depth map, log transformation was applied. This technique enables the compression of the values while preserving

<footnote>* https://wiki.ros.org/fr/urdf/Tutorials</footnote>

<footnote>† https://gazebosim.org/api/plugin/2/</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 3. Overview of the images with and without the introduced plugin</figcaption></figure>

the relative differences [Feng et al. (2014)]. Equation 1 computes the logarithm of the virtual depth map, represented as I. $I_{log}$ is the log-transformed image.

$$I_{log} = \log(I - I.min + 1) \qquad (1)$$

After log transformation, normalization is applied to scale the image data to the real depth map range, in this case, it is [0, 255]. This step ensures that the depth map uses the full range of the display intensity and is critical for consistent visualization and processing. In Equation 2, $I_{norm}$ is the normalized image.

$$I_{norm} = \frac{I_{log} - I_{log.min}}{I_{log.max} - I_{log.min}} * 255 \qquad (2)$$

## Augmentation strategy

Following the pre-processing of the virtual depth map, the next phase involves implementing the augmentation technique. This approach entails assessing and comparing the depth values from both virtual and real depth maps. In instances where the virtual depth values exceed those of the real map, the latter's values are substituted accordingly. In this representation, objects closer to the camera appear with darker pixel shades, indicative of elevated values. Through this methodology, virtual entities are seamlessly integrated into the real depth map, with occlusion factors accounted for. Denoting the real depth image as $I_1$ and the virtual depth image as $I_2$, the formula for this augmentation process is presented in Equation 3, where the operation is conducted
on a pixel-by-pixel basis:

$$I_1 = (I_1 \cap (I_1 > I_2)) \cup (I_2 \cap (I_2 > I_1)) \quad (3)$$

In this equation, the augmentation process is achieved through two primary operations: intersection and union.

* **Intersection with condition** ($I_1 \cap (I_1 > I_2)$) This part of the equation selects pixels from $I_1$ where the depth value in $I_1$ is greater than that in $I_2$. It effectively retains the depth values from the real depth map $I_1$ in regions where they are closer (or more significant) than those in the virtual depth map $I_2$

* **Intersection with condition** ($I_2 \cap (I_2 > I_1)$) Conversely, this part selects pixels from $I_2$ where the depth value in $I_2$ is greater than that in $I_1$. It retains the depth values from the virtual depth map $I_2$ in regions where they are closer to the camera than those in the real depth map $I_1$.

* **Union of results** The union operation combines the results of the two intersections. This means that for each pixel, the final depth value is taken from either $I_1$ or $I_2$, depending on which one has the greater depth value (indicating a closer object).

This approach effectively balances features from both the virtual and real worlds for several reasons:

* **Accurate Object Placement:** By comparing depth values from both virtual and real images, the algorithm ensures that objects are placed accurately within the mixed-reality environment. This preserves the spatial integrity and relative positioning of objects, which is crucial for realistic representation.

* **Occlusion Handling:** The method accounts for occlusion by replacing real depth values with virtual ones only when the virtual objects are closer to the camera. This ensures that virtual objects seamlessly blend into the real environment without any visually incorrect overlaps.

* **Dynamic Range Alignment:** By using techniques like histogram equalization before applying the augmentation, the depth maps from both sources are aligned in terms of intensity distribution. This uniformity is essential for a smooth transition and integration between the virtual and real elements.

Figure 4 is an illustration of the outcome of the described augmentation strategy. It displays an initial virtual image featuring a virtual pedestrian and an initial real image of a road scene. The resulting image effectively integrates the virtual pedestrian into
the road scene, illustrating the application of the augmentation technique. This visual
outcome demonstrates the fusion of virtual elements into real-world contexts.

<figure><img src="image_4.png" /><figcaption>**Figure 4.** Result of the augmentation strategy</figcaption></figure>

Experimentation and results

The main aim of these experiments was to evaluate the fusion of real and virtual
images. This objective determined the approach of this study. The experiments were
not designed to evaluate the generation of depth maps or to ameliorate object detection
algorithms but rather to analyze how effectively real and virtual images could be
integrated. To make this evaluation, open-source tools were employed. The foundation
of the experimental approach is to make sure that object detection locations are
consistent before and after the image augmentation process. While it is understood
that maintaining identical detection scores post-fusion might not be feasible, the goal
is to minimize the deviation in these scores.

Experimentation

In the evaluation of the proposed algorithm, the incorporation of two different data
types is essential: real-world images and simulation-based virtual images. This dual
strategy facilitates the application and evaluation of the algorithm. The KITTI dataset
was selected for real-world images [Geiger et al. (2013)].
<figure><img src="image_5.png" /><figcaption>**Figure 5.** RGB image and its depth map</figcaption></figure>

This dataset includes 7481 RGB images, each alongside ground truth data for 3D object detection. To augment these images, the MIDAS algorithm, known for its monocular depth estimation capabilities, was used [Ranftl et al. (2020)]. Trained on up to 12 datasets, MIDAS effectively generates depth maps, thereby enriching the dataset's dimensionality and utility in real-world scenarios. Figure 5 illustrates a sample image from the KITTI dataset with its corresponding depth map generated by the MIDAS algorithm. Virtual images were generated through the Gazebo simulator, a tool that can create realistic, simulated environments. This process involved capturing RGB images and their corresponding depth maps, using a virtual depth camera within the simulator. A total of 80 distinct images were generated, encompassing four diverse classes of objects: pedestrians, cars, trucks, and motorcycles. The careful selection and variety in these images ensure a diverse dataset, intended to test the performance of the algorithm in simulated settings under real-world conditions. Figure 6 represents a sample of two initial RGB images used for the fusion strategy, and the resulting augmented image.

The quality of image fusion necessitates a rigorous evaluation methodology that includes both qualitative and quantitative analyses. Qualitative evaluation offers a visual inspection of the fusion results, while quantitative measures provide objective, numerical assessments of fusion reliability. Three main measures were used to evaluate
<figure><img src="image_6.png" /><figcaption>Figure 6. Sample of the fusion</figcaption></figure>

the fusion of virtual and real images quantitatively: the Structural Similarity Index (SSIM), the Peak Signal-to-Noise Ratio (PSNR), and the correlation. Equation 4 describes the calculation for the PSNR that is considered a standard measure for assessing the quality of reconstructed images. PSNR provides a clear indication of the level of distortion or noise introduced during the augmentation process compared to the original image.

$$PSNR = 20 \cdot \log_{10} \left( \frac{MAX_{I}}{\sqrt{MSE}} \right) \quad (4)$$
where MAXI is the maximum possible pixel value of the image and MSE represents the mean squared error. It serves as a standard benchmark for measuring image quality. SSIM, on the other hand, evaluates the visual impact of three characteristics of an image: luminance, contrast, and structure, using the Equation 5.

$$SSIM(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)} \quad (5)$$

Where μ and σ represent the average and variance of xx and yy, respectively. The correlation between two images can be calculated by the Pearson correlation coefficient. This coefficient measures the linear correlation between two variables (in this case, the pixel intensities of the two images) as shown in Equation 6.

$$r = \frac{\sum_{i=1}^{n} (A_i - \bar{A})(B_i - \bar{B})}{\sqrt{\sum_{i=1}^{n} (A_i - \bar{A})^2 \sum_{i=1}^{n} (B_i - \bar{B})^2}} \quad (6)$$

$A_i$ and $B_i$ are the pixel intensities of the images A and B at position i respectively. $\bar{A}$ and $\bar{B}$ are the mean pixel intensities of images A and B, respectively, and n is the total number of pixels in each image.

Measuring the correlation between the original and augmented images helps in understanding the linear relationship between their pixel intensities. A high correlation indicates that the augmented image closely follows the pixel intensity patterns of the original image. The values range from -1 to 1, where 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship.

These metrics were chosen for their ability to objectively evaluate the perceptual quality of the augmented images, ensuring a reliable assessment of the fusion process, by proving that the fusion doesn't negatively impact the quality of the initial images (the real images). In addition to the initial quantitative evaluation, a further evaluation was conducted with the Faster RCNN algorithm [Chen et al. (2019)], which was pre-trained on the MS-COCO dataset beforehand [Lin et al. (2014)]. The primary objective is to verify whether the fusion process preserves the detectability of objects from both real and virtual environments. By applying this model, the study aims to ensure that all relevant objects are accurately identified in the augmented images. Furthermore, the evaluation extends to comparing the mean Average Precision (mAP) across three sets of images: real, virtual, and augmented. This comparison intends to investigate if the fusion negatively impacts the model's performance. This is important for ensuring the practicality of the fusion process in real-world applications. The adopted model (Faster
RCNN) was pre-trained on the extensive MS-COCO dataset. Therefore, there was no need for additional training specific to this study. Instead, the focus was on the use of the existing performance of the model. For the real images, the ground truth data from the KITTI dataset served as a benchmark for comparison with the model's predictions. The goal was not to improve the object detection model's performance but to use it as a metric to validate the augmentation strategy by achieving results that closely match those of the real images. In the case of virtual images, a manual annotation process was employed to create the necessary ground truth data. Figure 7 illustrates the ground truth visualization, which includes the bounding box coordinates and the associated classes for each object.

<figure><img src="image_7.png" /><figcaption>**Figure 7.** Virtual ground truth</figcaption></figure>

It is important to emphasize that the objective of this evaluation was not to surpass the mean Average Precision (mAP) of real images. Rather, the aim was to achieve a level of mAP for the augmented images that are comparable to that of the real images, thereby demonstrating the practicality of the fusion process without compromising the model's detection capabilities.

## Results and discussion

To properly evaluate the image quality and the precision of the augmentation process, we decided to compute the PSNR, SSIM, and Correlation between an original image and its augmented version. The analysis was carried out on a large set of 7481 images to
ensure accuracy and reliability. The mean and median values of the calculated metrics provide information about how well the image augmentation procedure performed overall (see Table 1):

<table><caption>**Table 1.** Mean and Median Values of Image Metrics</caption><thead><tr><td>Metric</td><td>Mean</td><td>Median</td></tr></thead><tbody><tr><td>PSNR</td><td>23.88</td><td>24.44</td></tr><tr><td>SSIM</td><td>0.889</td><td>0.899</td></tr><tr><td>Correlation</td><td>0.965</td><td>0.981</td></tr></tbody></table>

* **Mean and median of PSNR:** The mean PSNR of 23.88 and median of 24.44 indicate a moderate level of distortion in the augmented images compared to the originals. Better quality is typically indicated by a larger PSNR value; for most applications, values above 30 dB are regarded as excellent quality. Values in the range of 20-30 dB indicate moderate quality, which is what we can see here. These values suggest that while there are some images with significant distortion, the majority are of acceptable quality. A PSNR in the mid-20s can be quite acceptable for many applications, such as those involving real-time processing [Sara et al. (2019)]. This is particularly true in cases like autonomous driving. Here, the augmented images are used to make models stronger rather than being used for direct visual consumption.

* **Mean and median of SSIM:** The mean value of 0.889 and the median value of 0.899 show a high level of structural similarity between the augmented and original images. This implies that the augmentation process preserves the structural integrity of the images quite well. This is significant because it implies that the augmented images preserve the fundamental features and textures of the original images, which are critical for the accurate detection and recognition of objects in autonomous driving scenarios.

* **Mean and median of correlation:** The high mean correlation of 0.965 and median of 0.981 indicate a strong linear relationship between the pixel intensities of the original and augmented images. This suggests that the augmentation process maintains the overall intensity patterns effectively. A high correlation coefficient indicates that the augmented images closely follow the intensity patterns of the original images. This is essential because it shows that there are no noticeable artifacts or distortion introduced by the augmentation procedure that would alter the visual perception of the images.
Figure 8 represents the results before and after augmentation, highlighting the virtual and real RGB and depth images and their fusion. Initially, the figure displays the virtual RGB and depth images alongside the real RGB and depth images. Following this, the figure illustrates the outcome of this fusion, beginning with the result of the depth fusion, which the algorithm relies on. Subsequently, it showcases the results when the depth output is switched to the RGB output.

<figure><img src="image_8.png" /><figcaption>Figure 8. Results before and after the augmentation strategy</figcaption></figure>

Figure 9 displays qualitative results from the object detection process in the augmented images. This figure provides a clear visual demonstration of both virtual (pedestrian) and real (cars) objects being successfully detected. It serves as a qualitative overview, highlighting the effectiveness of the fusion technique in maintaining the detectability of objects within a mixed-reality context. The result also reveals certain areas for improvement. Notably, virtual objects occasionally appear disproportionately large compared to real objects, impacting the overall realism. This observation suggests the need to adjust the sizes of virtual objects for a more realistic integration.
Additionally, discrepancies in the surface textures between real and virtual images are noticeable, attributable to differences in camera orientation. Addressing this by aligning the orientation of the virtual camera with the real images could significantly improve the qualitative overview, ensuring a smooth integration between virtual and real elements.

<figure><img src="image_9.png" /><figcaption>**Figure 9.** Result of object detection on the augmented image</figcaption></figure>

To explain the results obtained from the Faster RCNN testing on three different datasets, an evaluation was conducted. The model was tested on virtual images with their respective ground truth, real images from the KITTI dataset with ground truth, and augmented images under three scenarios: with virtual ground truth only, with real ground truth only, and with augmented ground truth. This approach provided a comprehensive understanding of the model's performance across varying data types. The mean Average Precision (mAP) for each of these tests was calculated and presented in Table 2, offering detailed insights into the effectiveness of the model in different contexts.

<table><caption>**Table 2.** mAP values for different datasets and different ground truth</caption><thead><tr><td>mAP</td><td>Real images</td><td>Virtual images</td><td>Augmented images</td></tr></thead><tbody><tr><td>Real ground truth</td><td>0.830</td><td>-</td><td>0.8134</td></tr><tr><td>Virtual ground truth</td><td>-</td><td>0.75</td><td>0.869</td></tr><tr><td>Combined ground truth</td><td>-</td><td>-</td><td>0.8267</td></tr></tbody></table>

From the results obtained, the model displayed high mean Average Precision values when tested on real images and virtual images separately, with mAPs of 0.830 and 0.75, respectively. The mAP variation depends on the ground truth used on augmented images. With real ground truth, the mAP was slightly lower at 0.8134 while with virtual ground truth, it increased to 0.869. The mAP for augmented images using augmented ground truth was 0.8267, indicating a balanced performance. These results highlight the model's adaptability across different image types and the effectiveness
of the fusion process in maintaining object detection accuracy. The observed mAP
on virtual images can be attributed to the fact that the model was initially trained
on real images from the MS-COCO dataset, and later tested on virtual images. To
potentially improve the mAP for virtual images, it is possible to train the model first on
a larger set of virtual images. This approach could fine-tune the model's capabilities for
virtual environments, potentially improving its performance in detecting objects in such
scenarios. In general, the results indicate that despite minor variations in mAP, there
is no significant fluctuation with the augmented images. This suggests that the mixed
reality method could be a suitable alternative. Moreover, it has been observed that in
low-light or poor visibility conditions, the integration of virtual elements significantly
improved the perception capabilities of the model. The addition of a clear virtual
depth information might have compensated for the lack of visual data in dark of foggy
scenarios, by providing additional data that were not affected by lighting conditions,
thereby leading to a better object detection and scene understanding.

Conclusion

This paper introduces an augmentation strategy for mixed-reality environments,
addressing the essential issue of the reality gap in autonomous driving. The approach
combines virtual depth images with the KITTI dataset, improving the realism of
autonomous vehicle training scenarios. The results, evaluated using different metrics
and the Faster RCNN model, show how well the augmented images replicate real-
world circumstances. This research contributes to the field by providing a method for
training and testing autonomous vehicles in diverse and complex scenarios, thereby,
going towards safer and more efficient autonomous driving systems. Nevertheless, a
wide range of opportunities still exist.

* For offline testing, the proposed approach has only evaluated a pre-trained model with augmented images. It would be insightful to train the model on this augmented dataset to fully assess its training capabilities.

* Improving the virtual data generation process, such as working on object size accuracy and camera orientation, could increase realism.

* Future work could investigate alternative methods for generating virtual data to further improve the diversity and quality of training datasets. Techniques such as generative adversarial networks (GANs) could be explored to create more realistic and varied virtual elements.

* Integrating other sensors, such as LiDAR and radar data, can provide a more complete perception system. Combining these sensors might improve the
accuracy of the augmented data. This suggestion is supported by the work of Genevois et al. (2022), who demonstrated the potential of integrating LiDAR data for improved perception in mixed-reality frameworks.

* The transition to online applications has a lot of potential in the future. This includes exploring online learning for navigating in high-risk situations and testing various models in real time. For instance, adding a similar real-time approach as Mitchell et al. (2020) did, where agents interact with virtual elements might provide a dynamic and responsive training environment.

* Using more complex assessment metrics may offer a better understanding of the augmentation strategy's impact. Further insights could be obtained by investigating metrics that quantify motion smoothness and temporal consistency in dynamic environments.

## References

Argui, I, Gueriau, M, and Ainouz, S (2023) Building a vision-based mixed-reality framework for autonomous driving navigation. In: *9th International Conference on Control, Decision and Information Technologies (CoDIT)*, Jul 03, pp.1960-1965. IEEE.

Bai, X, Jiang, L, Luo, Y, Gupta, A, Kaveti, P, Singh, H, and Ostadabbas, S (2023). An evaluation platform to scope performance of synthetic environments in autonomous ground vehicles simulation. In *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 1-5. IEEE.

Carruth, D W, (2018). Simulation for training and testing intelligent systems. In *2018 World Symposium on Digital Intelligence for Systems and Machines (DISA)*, pp. 101-106.

Chang, Y P, Chen, H A, Chang, S H, Pei, Z, Liu, C N, Han, P, and Cheng, W H (2019). Integration of LiDAR with laser headlight for autonomous driving. In: *ODS: Industrial Optical Devices and Systems*, Sept 12, Vol. 11125, p. 111250H, SPIE.

Chen, K, Wang, J, Pang, J, Cao, Y, Xiong, Y, and al. (2019). MMDetection: Open mmlab detection toolbox and benchmark. ArXiv abs/1906.07155

Feng, C, Wang, H, Lu, N, Chen, T, He, H, Lu, Y, and Tu, X M, (2014). Log-transformation and its implications for data analysis. In *Shanghai Archives of Psychiatry*, vol. 26, pp. 105 - 109.

Geiger, A, Lenz, P, Stiller, C, and Urtasun, R (2013). Vision meets robotics: The kitti dataset. *The International Journal of Robotics Research*. vol. 32 no. 11, pp.1231-1237.

Genevois, T, Horel, J B, Renzaglia, A, and Laugier, C (2022). Augmented Reality on LiDAR data: Going beyond Vehicle-in-the-Loop for Automotive Software Validation. In: *33rd IEEE Intelligent Vehicles Symposium IV*, Jun 04, pp 971-976. IEEE
Herath, S, Dissanayake, V, Rasnayaka, S, Seneviratne, S, Vidanaarachchi, R, Gamage, C. (2017). Generation of intermediate viewpoints for scalable adaptation of real world environments for virtual reality. In *International Conference on Industrial and Information Systems (ICIIS)*, pp. 1-6.

Huang, Y, and Chen, Y (2020). Survey of State-of-Art Autonomous Driving Technologies with Deep Learning. In: *20th International Conference on Software Quality, Reliability and Security Companion (QRS-C)*, Dec 11, pp 221-228. IEEE.

Hu, X, Li, S, Huang, T, Tang, B, Huai, R, and Chen, L (2023). How Simulation Helps Autonomous Driving: A Survey of Sim2real, Digital Twins, and Parallel Intelligence. *IEEE Transactions on Intelligent Vehicles*, vol. 9, no. 1, pp. 593-612.

James, S, Wohlhart, P, Kalakrishnan, M, Kalashnikov, D, Irpan, A, Ibarz, J, Levine, S, Hadsell, R, and Bousmalis, K (2019). Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In: *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. June 15, pp. 12619-12629. IEEE.

Kang, M S, and An, Y, (2021). Deep Learning-Based Automated Background Removal for Structural Exterior Image Stitching. In *Applied Sciences*, vo. 11, pp. 3339.

Koenig, N, and Howard, A (2004). Design and use paradigms for gazebo, an open-source multi-robot simulator. In: *RSJ international conference on intelligent robots and systems (IROS)*. Sep 28, vo. 3, pp. 2149-2154. IEEE.

Lin, T Y, Maire, M, Belongie, S, Hays, J, Perona, P, et al. (2014). Microsoft coco: Common objects in context. In: *Computer Vision-ECCV 2014: 13th European Conference*, September 6-12, vo. 13, pp. 740-755. Springer International Publishing.

Li, W, Pan, C W, Zhang, R, Ren, J P, Ma, Y X, Fang, J, Yan, F L, Geng, Q C, Huang, X Y, Gong, H J, Xu, W W, Wang, G P, Manocha, D, and Yang, R G (2019). AADS: Augmented autonomous driving simulation using data-driven algorithms. *Science robotics*, vo 4, no 28, pp. eaaw0863.

Liu, H, Wu, C, and Wang, H (2023). Real time object detection using LiDAR and camera fusion for autonomous driving. *Scientific Reports*, vo. 13, pp. 8056.

Mao, J, Niu, M, Jiang, C, Liang, H, Chen, J, Liang, X, Li, Y, Ye, C, Zhang, W, Li, Z, and Yu, J (2021). One Million Scenes for Autonomous Driving: ONCE Dataset. 10.48550/arXiv.2106.11037.

Milgram, P, and Kishino, F (1994). A taxonomy of mixed reality visual displays. *IEICE TRANSACTIONS on Information and Systems*, vo. 77, no. 12, pp. 1321-1329.

Mitchell, R, Fletcher, J, Panerati, J, and Prorok, A (2020). Multi-Vehicle Mixed Reality Reinforcement Learning for Autonomous Multi-Lane Driving. In: *Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)*, pp
1928-1930.

Muhammad, K, Ullah, A, Lloret, J, Ser, J D, and de Albuquerque, V H C (2021) Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions. *IEEE Transactions on Intelligent Transportation Systems*, vo. 22, no. 7, pp. 4316-4336.

O'Kelly, M, Sinha, A, Namkoong, H, Tedrake, R, and Duchi, J C (2018). Scalable end-to-end autonomous vehicle testing via rare-event simulation. *Advances in neural information processing systems*, vo. 31.

Nabati, M R (2021). Sensor Fusion for Object Detection and Tracking in Autonomous Vehicles. PhD Thesis, University of Tennessee.

Peintner, J, Funk Drechsler, M, Reway, F, Seifert, G, Huber, W, and Riener, A (2021). Mixed reality environment for complex scenario testing. In: *Proceedings of Mensch und Computer*, no. 4, pp. 605-608.

Ranftl, R, Lasinger, K, Hafner, D, Schindler, K, and Koltun, V (2020). Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. *IEEE transactions on pattern analysis and machine intelligence*, vo. 44, no. 3, pp. 1623-1637.

Sara U, Akter M, Uddin MS. (2019). Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study. *Journal of Computer and Communications*, vo.7, no. 3, pp. 8-18.

Schöner, H P (2018). Simulation in development and testing of autonomous vehicles. In: *18th. Internationales Stuttgarter Symposium: Automobil-und Motorentechnik*, pp. 1083-1095. Springer Fachmedien Wiesbaden.

Szalai, Mátyás, et al. (2020). Mixed reality test environment for autonomous cars using Unity 3D and SUMO. In: *18th World Symposium on Applied Machine Intelligence and Informatics (SAMI)*, pp. 73-78, IEEE.

Tiedemann, T, Schwalb, L, Kasten, M, Grotkasten, R, and Pareigis, S (2022). Miniature autonomy as means to find new approaches in reliable autonomous driving AI method design. *Frontiers in neurorobotics*, vo. 16, pp. 846355.

Vargas J, Alsweiss S, Toker O, Razdan R, Santos J. (2021). An Overview of Autonomous Vehicles Sensors and Their Vulnerability to Weather Conditions. *Sensors.*, vo. 21, no. 16, pp. 5397.

Wang, P (2021). Research on comparison of LiDAR and camera in autonomous driving. In *Journal of Physics: Conference Series*, vo. 2093, no. 1, pp. 012032.

Xu, M, Niyato, D, Chen, J, Zhang, H, Kang, J, Xiong, Z, Mao, S, Han, Z (2023). Generative AI-empowered simulation for autonomous driving in vehicular mixed reality metaverses. in *IEEE Journal of Selected Topics in Signal Processing.*, vo. 17, pp. 1064-1079.
Yao, S, Zhang, J, and Wang, Y (2020). VR-based dataset for autonomous-driving system. *The Journal of Engineering*, no. 13, pp. 411-415.

Ying, Tianyu. (2024). Analysis on key technologies of autonomous vehicle sensing system. *Applied and Computational Engineering*. vo. 70. pp. 67-72.

Zeng, D, and Zhu, M (2018). Background Subtraction Using Multiscale Fully Convolutional Network. In *IEEE Access*, vo. 6, pp. 16010-16021.

Zofka, M R, Essinger, M, Fleck, T, Kohlhaas, R, and Zöllner, J M (2018). The sleepwalker framework: Verification and validation of autonomous vehicles by mixed reality LiDAR stimulation. In: *IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR)*, pp. 151-157. IEEE.