# A Generic Framework for Enhancing Autonomous Driving Accuracy through Multimodal Data Fusion
## 4 Proposed Study
### Proposed Feature Extraction Models
#### 4.2.3 Proposed Hybrid Fusion Model
The initial layer is utilized to carry out the process of extracting features, while the second layer is responsible for combining the extracted features using a proposed hybrid fusion model. The third layer converts the resulting 2D fused image into 3D point cloud data using the equation referred to in (21). Following that, the fourth layer is dedicated to performing the tasks of the discriminator. Finally, the last layer is responsible for executing the final fusion. Table 8 depicts the design implementation of the proposed GAN model. The 3D point cloud environment setup was carried out using the following steps. Initially, a vehicle perception system purchased from a third-party vendor was used, which helped to create an advanced 3D point cloud perception system. This cloud-based system was installed in the Ubuntu operating system using advanced Docker commands and Python scripting. Applications were developed using Python scripts and later integrated with the perception system to perform advanced analysis. Both real-time data collected from Velodyne sensors and 3D point cloud data obtained from the GeoTiles dataset were used to evaluate the performance of the proposed GAN and other popular fusion models such as FusionGAN, curvelet transform (CVT), dual-tree complex wavelength transform (DTCWT), Laplacian pyramid transform (LP), lowpass pyramid (LP) and multi-resolution singular value decomposition (MSVD). A sample of 20 images was selected from the GeoTiles dataset to train the GAN model, and a maximum of four images were used to test the GAN model initially. Figure 16 illustrates the performance of various fusion models for the metrics (MI, EN, SD, and PSNR). A sample of 30 point cloud scenes was taken, which analyzed the performance of various popular fusion models along with our extended GAN model for four metrics, namely MI, EN, SD, and PSNR. From the results, the proposed GAN outperformed other fusion models. Table 9 depicts the CPU time consumed by different fusion models to fuse 2D image data and 3D point cloud data. A sample of 10 images was collected from the nuScenes and GeoTiles dataset to evaluate the efficiency of the fusion models. Once again, the proposed fusion models performed better than the other fusion models. Again, a sample of three 3D point cloud data from the GeoTiles dataset along with their blurred images (since it was difficult to obtain blurred images from the dataset, online applications to blur the selected images were used) was taken to analyze the performance of different fusion models. From the analysis, it was observed that the fusion accuracy of the proposed GAN model was better than other fusion models. Figure 17 depicts the \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Models** & **Proposed** & **PCA** & **WMFCS** & **GS** \\ \hline True positives (TP) & 15 & 130 & 140 & 120 \\ True negatives (TN) & 130 & 120 & 120 & 110 \\ False positives (FP) & 12 & 28 & 25 & 40 \\ False negatives (FN) & 8 & 22 & 15 & 30 \\ Accuracy (\%) & 93 & 83 & 86 & 77 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Accuracy of different image fusion models. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Layer** & **Discriminator Network** & & & & \\ \hline  & **1** & **O** & **KS** & **SS** & **PS** & \\ \hline Layer1 Conv & 1 & 64 & 4 & 1 & 0 & LeakyReLU \\ Layer2 Conv & 64 & 128 & 4 & 2 & 0 & LeakyReLU \\ Layer3 Conv & 128 & 128 & 4 & 1 & 0 & LeakyReLU \\ Layer4 Conv & 128 & 256 & 4 & 2 & 0 & LeakyReLU \\ Layer5 Conv & 256 & 256 & 4 & 1 & 0 & LeakyReLU \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Implementation details of proposed GAN model. ### Limitations of the Proposed Study

The study introduced two fusion models: one for merging 2D and 3D images and another for combining various image types, with a specific focus on 3D point cloud data. The initial hybrid image fusion model encountered significant challenges, including ineffective fusion of 3D point cloud data and difficulties in handling large datasets. These issues were addressed by the extended GAN model.