# Exploring a Multimodal Mixture-Of-YOLOs Framework for Advanced Real-Time Object Detection
## 3 Multimodal Mixture-Of-YOLOs Framework for Object Detection
### Data Preprocessing  
LiDAR provides the reflected laser signal, \(\lambda=(\text{x},\text{ y},\text{ z},\text{ r})\), as a three-dimensional coordinate value (x, y, z) and reflectance information (r), which refers to the intensity of the reflected signal according to roughness, color, and materials of the ground and reflective surface of the object. Object detection using LiDAR is classified based on use the 3D coordinate value as is or by projecting it into 2D space of the top or front view [22]. Object detection using the top view allows easy extraction of the direction and speed of movement of the vehicle, although the computational process of this type of object detection is complicated. On the other hand, object detection using the same front view as the RGB camera and driver view is simpler than object detection using the top view. In this study, the PCD is converted into a two-dimensional pixel coordinate system such as image data through the conversion process of projecting the PCD to the same front view as the field of view (FOV) of the RGB camera. The pixel coordinate system refers to a 2D reference coordinate system of pixels included in the image data [23]. Since the PCD represents data obtained from all directions of x, as shown in Figure 4a, the PCD expressed in the FOV of the RGB camera is separated as shown in Figure 4b. In Figure 4a, the position of the LiDAR is the origin, the position of the RGB camera is the point shifted by 5 axes from the LiDAR, and the central axis of the FOV is the direction parallel to the y axis, satisfying the condition x \(>\) 5. The 3D LiDAR coordinate system, which represents spatial information, is converted to a 2D pixel coordinate system through a rotation/translation transformation matrix provided by the KITTI dataset so that it is projected onto the camera's image plane. As shown in Figure 5a, the coordinate \((\eta=x,\ y,\ z)\), which is a separated PCD, is extracted and converted into the homogeneous coordinate of \((x,\ y,\ z,\ 1)\), and \(\eta^{\prime}\) projected onto the image plane is obtained through the rotation/translation transformation matrix. Here, the homogeneous coordinate extends \(N\)-dimensional coordinate of \((a_{1},\ a_{2},\ \cdots,\ a_{N})\) to \(N+1\) dimension with respect to nonzero w, and is represented by \((wa_{1},\ wa_{2},\ \cdots,\ wa_{N})\). A PCD, \(\eta^{\prime}\), on the projected image plane is represented by \((w\text{x}^{\prime},\ \text{w}^{\prime},\ \text{w})\), where w represents the distance from the camera and the lidar to the projected PCD, \(\eta^{\prime}\), depending on the nature of the homogeneous coordinates. Dividing w by two-dimensional coordinates of \(\eta^{\prime}\) converts it to the pixel coordinate system as \(\eta^{\prime}=(x,y)\) and defines it as \(\text{H}=(\text{X},\ \text{Y})\). Figure 4: Example of point cloud data (PCD) from a LiDAR (**a**) PCD (Top View); (**b**) Extracted PCD (Top View). Through this process, the PCD projected by the 2D pixel coordinate system is illustrated in Figure 6a, and Figure 6b shows that the PCD is projected with a view the same as the FOV of the RGB camera. However, since PCD has much lower resolution than the original image, up-sampling for high resolution is performed using spatial interpolation based on bilateral filter [24]. The scaled high-resolution depth map and the reflectance map are generated by applying the weighted pixel values obtained based on the distance and reflectance information of the pixels adjacent to the PCD projected on the image and the adjacent pixels, as shown in Figure 7. The coordinates of the pixels of the depth map and the reflectance map are defined as \(H_{d}=(X_{d},\ Y_{d})\) and \(H_{r}=(X_{r},\ Y_{r})\), respectively, and the coordinates of the pixels of the image data are defined as \(H_{c}=(X_{c},\ Y_{c})\). Figure 5: Projection of LiDAR data onto RGB image plane. (**a**) homogeneous coordinate; (**b**) pixel coordinate.