## 4 Evaluation of Object Detectors  
### Evaluation Criteria  
In the objection detection task, the trained detector was evaluated according to average precision (AP), for a single class, and mean average precision (mAP), which is the average AP of each class, for multiple class detectors. The AP is calculated from two basic indexes: recall and precision. Recall and precision can be defined as follows:  
\[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{1}\]  
Figure 8: Example of labeled fire-causing items. Figure 7: Collected sample of supplementary small-sized electronic products: (**a**) heated tobacco, (**b**) smartphones. \[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{2}\]  
The values of the true positive (TP), false positive (FP), and false negative (FN) in the equations can be defined by the confusion matrix shown in Table 1. In the model's prediction results for the samples, TP shows that the model correctly predicted the positive class, FP shows that the model incorrectly predicted the positive class, and FN is refers to incorrect predictions of the negative class. For the detection task, the prediction bounding box for each detection was also evaluated using Intersection over Union (IoU), which calculates the overlap ratio between the prediction bounding boxes \(\text{B}_{\text{P}}\) and the ground truth bounding boxes \(\text{B}_{\text{gt}}\). IoU can be defined as follows:  
\[\text{IoU}=\frac{\text{Area}\big{(}\text{B}_{\text{P}}\cap\text{B}_{\text{gt} }\big{)}}{\text{Area}\big{(}\text{B}_{\text{P}}\cup\text{B}_{\text{gt}}\big{)}} \tag{3}\]  
Here, the threshold of IoU was set as 0.5. In the prediction results for the test data, for the predicted bounding boxes in the detection of a certain category, only the number of bounding boxes with an IoU greater than 0.5 will be counted as a TP. Conversely, the number of bounding boxes with an IoU less than or equal to 0.5 will be counted as an FP. Finally, the number of ground truths that were not detected will be counted as an FN. The confidence score reflects the accuracy of the model in predicting that an object is within a certain category, as shown in the equation below:  
\[\text{Confidence}=\text{P}_{\text{r}}(\text{Class}_{\text{i}}|\text{Object}) \times\text{P}_{\text{r}}(\text{Object})\times\text{IoU} \tag{4}\]  
In Equation (4), \(\text{P}_{\text{r}}(\text{Class}_{\text{i}}|\text{Object})\) is the probability of an object being a specific kind of object when it is known to be an object, and \(\text{P}_{\text{r}}(\text{Object})\) is the probability of whether the predicted bounding box contains an object. If it contains, \(\text{P}_{\text{r}}(\text{Object})=1\), otherwise \(\text{P}_{\text{r}}(\text{Object})=0\). To evaluate the trade-off between false positives and false negatives without defining arbitrary costs for each type of classification error [59], different confidence score thresholds were set, and some predictions will be excluded from the calculation of TP, FP, and FN when their confidence score is below this threshold, thereby generating multiple recall-precision pairs to construct the precision-recall curve. The final AP and mAP values were calculated from a precision-recall curve using the following equation:  
\[\text{AP}=\int_{0}^{1}\text{P}(\text{R})\text{dR} \tag{5}\]  
\[\text{mAP}=\frac{\sum_{i=1}^{N}\text{AP}_{\text{i}}}{\text{N}} \tag{6}\] \begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{2}{c}{**Predicted Value**} \\ \hline  & & Positive & Negative \\  & & (Mobile battery) & (Not mobile battery) \\ \hline \multirow{4}{*}{True value} & Positive & & \\  & (Mobile battery) & True positive (TP) & False negative (FN) \\ \cline{1-1}  & Negative & & \\  & (Not mobile battery) & False positive (FP) & True negative (TN) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Confusion matrix (using mobile batteries as an example). P, R, and N are precision, recall, and the total number of target object categories, respectively.