# Exploring a Multimodal Mixture-Of-YOLOs Framework for Advanced Real-Time Object Detection
## 2 Object Detection Schemes with RGB Images
### Yolo  
YOLO is a state-of-the-art, real-time detection system that predicts and classifies bounding boxes for objects within a given image simultaneously. The image data input to YOLO is divided into \(\mathrm{S}\times\mathrm{S}\) grid cells according to resolution, features are extracted through CNN, and the predicted tensor is obtained through a fully connected layer, as shown in Figure 2. The predicted tensor has a size of (\(\mathrm{S}\times\mathrm{S}\)) and a length of (\(\mathrm{B}\times\mathrm{5}+\mathrm{C}\)). Here, \(\mathrm{S}\times\mathrm{S}\) is the number of grid zones, \(\mathrm{B}\) is the number of candidate boundary boxes whose center point is included in the grid zone, and \(\mathrm{C}\) is the number of objects that can be classified. Each lattice zone is represented by a vector having a length of (\(\mathrm{B}\times\mathrm{5}+\mathrm{C}\)), and the set of \(\mathrm{S}\times\mathrm{S}\) lattice zones constitutes a predicted tensor of \(\mathrm{S}\times\mathrm{S}\times(\mathrm{B}\times\mathrm{5}+\mathrm{C})\).  
The grid region predicts \(\mathrm{B}\) bounding boxes, which contain five pieces of information (\(\mathrm{x},\mathrm{y},\mathrm{w},\mathrm{h},\mathrm{S}_{\mathrm{conf}}\)); (\(\mathrm{x},\mathrm{y}\)) is the center coordinate of the bounding box, (\(\mathrm{w},\mathrm{h}\)) is the width and height, \(\mathrm{S}_{\mathrm{conf}}\) as in Equation (1) is the product of \(\mathrm{Pr}(\mathrm{Object})\) and the probability that an object is included in the bounding box, and \(IOU_{pred}^{truth}\) is the area relative to the ground truth and intersection of union (IOU), indicating how accurately the bounding box predicted the geometric information of the object.  
\[\mathrm{S}_{\mathrm{conf}}=\mathrm{Pr}(Object)\times IOU_{pred}^{truth} \tag{1}\]  
If the actual coordinates and the center coordinates of the predicted bounding box are included in the same grid area, the bounding box is considered to contain the object, and \(\mathrm{Pr}(\mathrm{Object})\) is calculated as 1; it is treated as 0 if it is included in a different grid area. The IOU is calculated as in Equation (2) by  
Figure 1: Block diagram of a faster regional convolutional neural network (R-CNN).  
Figure 2: Structure of a You Only Look Once (YOLO) network.  
dividing the intersection of two regions by the union of the two regions and is used to evaluate the accuracy of the predicted bounding box (\(b.b_{pred}\)) with respect to the ground truth (\(b.b_{truth}\)).  
\[IOU_{pred}^{truth}=\frac{area\big{(}b.b_{pred}\cap\ b.b_{truth}\big{)}}{area \big{(}b.b_{pred}\cup\ b.b_{truth}\big{)}} \tag{2}\]  
In addition, grid area is expressed as \(\Pr(Object)\) by calculating the conditional probability indicating which among C objects can be classified as the types included in the bounding box as follows:  
\[P_{class}=\Pr(Class\big{|}Object) \tag{3}\]  
In this way, after the tensor of length (B x 5 + C) predicts all the grid regions (S x S), it is extended to obtain the confidence score, \(CS_{conf}\), for classifying objects representing both \(S_{\text{conf}}\), which is the probability that the object is included in the predicted bounding box, and \(P_{class}\), indicating the probability that the classified object matches the ground truth, through the following:  
\[\begin{split} CS=\ S_{\text{conf}}\times P_{class}\\ =\Pr(Object)\times IOU_{pred}^{truth}\times\Pr(Class[Object) \\ =\Pr(Class)\times IOU_{pred}^{truth}\end{split} \tag{4}\]  
Finally, for the classified object, the bounding box with the highest CS among the predicted B bounding boxes of the input tensor is selected as the bounding box of the object [21].