# Cascaded Vehicle Matching and Short-Term Spatial-Temporal Network for Smoky Vehicle Detection
## 5 Experiments
### Comparison to the State of the Art on LaSSoV-Video  
As mentioned above, our LaSSoV-video contains diverse scenes which could be challenging for existing methods. We evaluated the pure YOLOv5s, our YOLOv5 tiny, and our cascaded framework on the test set (four scenes) of LaSSoV-video and reported the average performance. The detection threshold was set to 0.2 for our YOLOv5 tiny. For the pure YOLOv5s and YOLOv5 tiny, we reported the results with detection thresholds of both 0.2 and 0.5. We re-implemented the R-VLBC [7] on our LaSSoV-video and also compared it to other methods on the private SEU-SmokeVeh (results were copied from the reference papers). Unlike existing works, which only use one or two metrics, we applied four popular frame-wise metrics for evaluation. From Table 3, several observations can be made as follows. First, from the first and the third rows, we find that YOLOv5s obtains the highest detection ratio (or recall ratio) while the performance gap between YOLOv5s and our YOLOv5 tiny is not as large as the one in the LaSSoV image data. This suggests that the lightweight YOLOv5 tiny is enough to learn the main features of obvious smoke regions while YOLOv5s can struggle with hard cases. Second, increasing the detection threshold to 0.5 for YOLOv5s and YOLOv5 tiny largely improves precision and degrades detection rates, while still boosting F1 scores. Third, (ST)\({}^{2}\)Net-suffix is consistently superior to (ST)\({}^{2}\)Net-prefix for our cascaded framework, which may be explained by the motion information in high-level layers being better for classification. We fixed (ST)\({}^{2}\)Net-suffix as the (ST)\({}^{2}\)Net module in the remaining content unless otherwise specified. Fourth, our cascaded method significantly outperforms the others in Precision and F1, while still keeping reasonable detection rates. Specifically, based on the results of YOLOv5 tiny(0.2), our method obtains around a 25% gain in F1 score. Last but not least, the R-VLBC [7] method that leverages hand-crafted local pattern features \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Detector** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]:0.95** & **Params (M)** & **FLOPs** & **Speed-CPU (ms)** & **Speed-GPU (ms)** \\ \hline MobileNetv2- & 64.52 & 22.96 & 4.6 & 36.7 & 470.6 & 13.1 \\ YOLOv5s & 87.3 & 39.8 & 7.2 & 16.5 & 134.5 & 18.5 \\ YOLOv5n & 80.9 & 33.1 & 1.9 & 4.5 & 61.5 & 13.2 \\ Our YOLOv5 tiny & 79.8 & 32.1 & 1.2 & 2.8 & 48.9 & 12.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation of different deep learning detectors on LaSSoV. All the detectors are trained on the training set of LaSSoV and the results are reported on the test set of LaSSoV with image size 640. The speeds are reported on three randomly-selected smoke images with average inference time. achieves the worst performance in all metrics, which indicates the limitations of these features. To further investigate their performance in different scenes, Figure 9 illustrates the comparison of F1 score in each scene. The traditional R-VLBC method is consistently worse than deep learning methods especially in scene #3 (cross roads). There exist many varied vehicle shadows in scene #3, which largely confuse those static smoke detectors. YOLOv5s(0.2) and YOLOv5 tiny(0.2) perform well and similarly in simple scenes (i.e., scene #1 and #2). Scene #4 shows a partially wet road and the wet regions can be easily mistaken as smoke regions by YOLOv5s and YOLOv5 tiny, leading to low F1 scores. Our method is significantly superior to other methods in all scenes thanks to the (ST)2 Net classifier. Specifically, our method improves the F1 of YOLOv5s by around 43% in scene #3, indicating that (ST)2 Net is robust to different roads. Figure 10 further illustrates some detection samples in our LaSSoV-video test set. The yellow bounding boxes denote those false positives of YOLOv5 tiny with scores larger than 0.4. These false positives are even difficult for human beings in static images, which are