# A Generic Framework for Enhancing Autonomous Driving Accuracy through Multimodal Data Fusion
## 4 Proposed Study
### Proposed Feature Extraction Models
#### 4.2.3 Proposed Hybrid Fusion Model
The proposed GAN model organizes and performs many activities such as organizing the incoming image frames, preprocessing them, and finally fusing them to improve the efficiency and accuracy of the perceived data, which in turn improves the accuracy of decision-making in the AVs. Additionally, this study proposes optimized kernel functions to select the best-fused image for better decision-making. The new GAN model consists of two discriminators: one is the spatial discriminator and the other the spectral discriminator. After acquiring the images from the sensors, the proposed GAN separates the spatial and spectral information through the discriminators. Advanced Fourier transformation is used as a kernel function to extract the spatial and temporal image information. After separating them, the GAN model uses two-fold image fusion. First, it uses the previously proposed image fusion model to perform the initial fusion. Then, using the proposed GAN model, additional projection functions are used to convert the fused images obtained from the hybrid image fusion Model to 3D point cloud data. The proposed GAN model uses attention-gate principles and compares the fused images with the source image information. If the difference is minimum then the 3D point cloud data generated in the first-fold fusion are further fused with the source images to obtain the final fused 3D point cloud information. Figure 10 illustrates the overall functionality of the proposed GAN model to convert 2D data to 3D data and fuse them accordingly. The fusion image obtained from Equation (17) is represented in Cartesian coordinates, which have to be converted into 3D projection coordinates using Equation (18) listed below. An additional column for height is introduced and Equation (17) is multiplied with an identity matrix, which converts the 2D image format to pointed 3D projection data. In Equation (18), \(h\), \(w\), and \(f\) correspond to the width, height, and focal length of the sensing device, respectively. \[ \begin{pmatrix}x^{\prime}\\ y^{\prime}\\ z^{\prime}\end{pmatrix} = \begin{bmatrix}1&0&-w/2\\ 0&1&-h/2\\ 0&0&1\\ 0&0&1\end{bmatrix}\begin{bmatrix}x_{00}&x_{01}&x_{02}&x_{03}\\ y_{10}&y_{11}&y_{12}&y_{13}\\ z_{20}&z_{21}&z_{22}&z_{23}\\ x_{00}&x_{01}&x_{02}&x_{05}\end{bmatrix}\begin{bmatrix}f&0&w/2&0\\ 0&f&h/2&0\\ 0&0&1&0\\ 0&0&1&0\end{bmatrix}\begin{bmatrix}x\\ y\\ z\end{bmatrix}  \tag{18}\]

The proposed GAN model has five layers. The first layer is used for the feature extraction process, the second to fuse the extracted features using the proposed hybrid fusion model, the third to convert the 2D fused image to 3D point cloud data using Equation (21), the fourth to perform the discriminator tasks, and the last layer to perform the final fusion. Spatial features are extracted using the kernel function (19), where \(F(i,j)\) is the image information existing in the spatial domain and the exponential term is the basis function corresponding to each pixel value of the image, and temporal features are extracted using the kernel function (20), where the exponential function is the basis of every temporal pixel value of the image information, \(F(i,j)\) and \(\psi\) is an optimization factor. Following multiple iterations of the training phase, we fine-tuned the factor to an optimized value of 0.65. \[F(i,j)=\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}f(i,j)e^{-t2\pi(i/N+j/N)}+\psi \tag{19}\]

\[F(i,j)=I/N\sum_{b=0}^{N-1}f(i,j)e^{-t2\pi(b/N)}+\psi \tag{20}\]

Figure 10: Functioning of proposed GAN model to fuse 3D point cloud data. The proposed GAN model is trained using Equation (21), where \(W_{s}^{k}\) is the weight for the \(k^{th}\) feature of \(f_{s}^{k}\) of the \(s\)-th pooling scale. \[W_{s}^{k}=\sigma(w_{i}\sum_{i,j}f_{s}^{k}(i,j)) \tag{21}\]

## 5 Experimental Analysis

Table 3 highlights important software and tools used to implement and evaluate the proposed models.