## 4 AdViSED Thresholds Analysis  
To evaluate the performance of the algorithm we run it over a selection of videos available in two datasets, one from [33; 34] and the other developed in the FP7 EU project Firesense and largely used in literature [35]. The final dataset is the union of test videos in [33; 34; 35], since some scenes are present both in [33; 34] and in [35] plus other videos, which are not public available, provided by the Engineering of _Trenitalia_, the Italian national railway company. Such videos have been acquired during specific fire/smoke tests on railway wagons, done at the Trenitalia testing facility in Osmanoro, Italy. The final dataset is composed of both videos with smoke presence and videos with no smoke presence. In both cases, we have outdoor and indoor videos of different quality (i.e., different levels of noise) and different frame rates, from 10 to 60 fps. Most of the used videos are adopted often in literature to compare the smoke/fire measurements and detection performance of state-of-art techniques. For the metrics computation, we need to define the true/false and positive/negative terms. We define a positive video from the dataset as a video with smoke presence, while a negative video is without it. In Table 1, if the algorithm identifies the smoke in a positive video, then we mark the result as true positive (TP). If the algorithm does not identify the smoke in a positive video, then we mark the result as false positive (FP). If the algorithm identifies the smoke in a negative video, then we mark the result as true negative (FN). If the algorithm does not identify the no smoke video in a negative video, then we mark the result as a true positive (TN). The metrics chosen to evaluate the goodness of the algorithm are recall, precision, accuracy, F1 score, and MCC; see formulas in Equations (7)-(11). The recall means how many relevant items are selected. The precision indicates how many selected items are relevant. The accuracy shows how close you are to the true value. The F1 score is the harmonic mean between precision and recall. The MCC parameter has a range that starts at \(-1\), when there is a complete misalignment between predicted and true value, and ends at \(+1\), when there is a complete alignment between predicted and true value. When the MCC is equal to 0 it means that the prediction is random compared to the true values. \[Recall=TP/(TP+FN) \tag{7}\]  
\[Accuracy=(TP+TN)/(TP+FN+TN+FP) \tag{8}\]  
\[F1\;score=2\cdot TP/(2\cdot TP+FP+FN) \tag{9}\]  
\[Precision=TP/(TP+FP) \tag{10}\]  
\[M\text{CC}=\frac{(TP\cdot TN-FP\cdot FN)}{\sqrt{(TP+FP)\cdot(TP+FN)\cdot(TN+ FP)\cdot(TN+FN)}} \tag{11}\]  
In this section, the analysis of the thresholds is reported. For each threshold type, many tests are performed until discovering the best values. As an example of the tests that have been done, we report in Figure 4 the metrics of some not optimized threshold sets comparing the achieved results to the set n. 6, which is the final selected one optimizing all metrics. The results in Figures 5 and 6 are obtained as an average over the whole testing videos. Each set of thresholds is detailed in Table 2. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Smoke Presence** & **No Smoke Presence** \\ \hline Alarm & TP & FP \\ No Alarm & FP & TN \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Confusion matrix. The \(THR_{sat}\) value is set using the real smoke saturation color. The \(THR_{mean}\), \(THR_{turb}\), \(THR_{recentr}\), \(THR_{ext\_min}\), \(THR_{ext\_max}\), and \(THR_{convArea}\) values are set using the real features of the blob smoke in the frame. The \(kill_{time}\) is set to delete the bounding box within a reasonable time. The \(THR_{overlap}\) is set considering a reasonable number of overlapping bounding boxes in a real smoke scene. The \(THR_{foreg}\) is set considering the motion detection operation result \(|I-\hat{B}G_{k-1}|\) and filtering this value with the threshold according to the best background subtraction.