# A Generic Framework for Enhancing Autonomous Driving Accuracy through Multimodal Data Fusion
## 4 Proposed Study
### Proposed Feature Extraction Models
#### 4.2.3 Proposed Hybrid Fusion Model
The operations are listed below in Equation (10)

\[\hat{E}=\left[\frac{B_{k}^{T}.B}{B_{k}^{T}B_{k}}B_{k}\right]_{k}=BB^{T}E\sum_{ D}^{-1} \tag{10}\]

where \(\sum_{D}\) is a diagonal \(FXF\) matrix whose diagonal components are \(B_{T}^{k}\) and \(B_{k}\). Now, we denote \(\cap W\) as the line vector of size \(B\), which is equal to \(B^{T}B\sum_{D}^{-1}\). Now, the 3D multispectral image can be defined as

\[\hat{E}=BW^{T} \tag{11}\]

By replacing the values of the first wavelength, \(\hat{E}\), a modified 3D image is obtained, which is listed in the below equation. When compared with the previous operations, this image matrix is a three-dimensional matrix representing pixel wavelength values, intensity values, and the dimension of the image, respectively. \[B^{\prime}=B\hat{W}^{T}=B \begin{array}{|cccc|}\hat{W}&&&&\\ 010&...&0&\\ 001&...&0&\\ &&.&\\ &&.&\\ &&.&\\ &&000&...&1\end{array}  \tag{12}\]

The next important stage is acquiring the original 3D image. To obtain the original 3D image, the inverse transformation of the new 3D image obtained using Equation (12) is decomposed using the QR() decomposition method, which postulates two important rules [46]. * Q is an orthogonal matrix, \(Q^{T}Q=I_{F}\) where \(I_{F}\) is the identity matrix. * R is an upper triangle matrix, having an upper triangle matrix inverse and \(QQ^{-1}=I_{F}\)

Now, the \(R\) inverse matrix obtained from the QR() decomposition is multiplied with the modified 3D image obtained using Equation (12) to obtain the original 3D fused image. The equations used to obtain the actual 3D image are discussed below. \[S=B^{\prime}R^{-1} \tag{13}\]

An important property of QR() decomposition is that the first column of the matrix \(S\) is proportional to that of \(B^{\prime}\), and hence it can still be interpreted as a multispectral image. Now, the first column of the multispectral image obtained from Equation (13) is replacedwith the modified 3D image matrix obtained using Equation (12), while all other columns are unchanged. The transformations are illustrated in the below final equations. \[S^{\prime}=SW_{s}^{T}+AW_{A}^{T}andW= \begin{vmatrix}00&...&0\\ 10&...&0\\ \cdot&\cdot&\cdot\\ \cdot&\cdot&\cdot\\ \cdot&\cdot&\cdot\\ 00&\cdot&1\end{vmatrix} W_{A}= \begin{vmatrix}W\\ 0\\ 0\end{vmatrix} \quad \tag{14}\]

\[= \begin{vmatrix}W_{0}W_{1}&...&W_{A-1}\\ \cdot&\cdot&\cdot\\ \cdot&\cdot&\cdot\\ \cdot&\cdot&\cdot\\ 00&\cdot&0\end{vmatrix}  \tag{15}\]

Since the first column of Equation (15) represents the pixel values of the 2D sentinel image and the other two columns represent the pixel values of 3D images, the final fusion equation can be represented as

\[\hat{B}=S^{\prime}R \tag{16}\]

#### 4.2.4 Enhanced Brovey Image Fusion Method

Though the multispectral fused image obtained from the proposed fusion method helps to improve the accuracy of the acquired 2D images, it lacks in producing the actual color intensity. In order to improve the RGB color intensities of the fused 3D image, the pixel values of the fused images are integrated with the Brovey fusion model, which computes the mean of individual red, green, and blue intensity values. The obtained mean value is multiplied by the PAN (panchromatic matrix value). The results obtained are finally multiplied with Equation (9) to obtain the full-fledged 3D fused image with the actual RGB color intensities. \[\hat{F}=R_{new} =\frac{R}{R+G+B}\times PAN(1)\times P_{B_{k}}(E) \tag{17}\] \[G_{new} =\frac{G}{R+G+B}\times PAN(2)\times P_{B_{k}}(E)\] \[B_{new} =\frac{B}{R+G+B}\times PAN(3)\times P_{B_{k}}(E)\]

### Proposed GAN Model

The proposed hybrid image fusion model did not achieve desirable results when integrating 3D point cloud data, and its performance deteriorated when evaluating extensive datasets. Hence, the study has extended its contribution by proposing a versatile GAN model that integrates the proposed feature selection and image fusion models.