# Cascaded Vehicle Matching and Short-Term Spatial-Temporal Network for Smoky Vehicle Detection
## 5 Experiments
### Ablation Study  
Since our framework included two auxiliary modules compared to pure YOLO detectors, we conducted an evaluation of these two modules, presented in Table 4. From Table 4, we can see that the vehicle matching module slightly improves YOLOv5 tiny in almost all metrics except for the detection rate (a slight degradation), and the (ST)2 Net module achieves larger performance gains than the vehicle matching module. Specifically, for the F1 scores, the vehicle matching module contributes a 2.5% improvement while the (ST)2 Net module contributes 22%. We evaluated several other common spatial-temporal video classification models including long-short term memory with CNN features (CNN-LSTM) [49], temporal segment network [50] (i.e., temporal average with CNN features, CNN-Avg), and temporal concatenation with CNN features (CNN-Cat). We replaced our (ST)2 Net-suffix as one of them for the evaluation. All their backbones were the same as in our (ST)2 Net model (i.e., ResNet-18); the CNN features are the output of the last pooling layer in ResNet-18. The results are shown in Figure 11. Several observations can be made as follows. First, our (ST)2 Net-suffix obtains the best F1 score and Precision which suggests that the abstract motion information may be captured by 3D convolutions. Second, CNN-Avg and CNN-Cat perform similarly in all metrics since they actually use the same static CNN features of three frames. Last but not least, an LSTM model upon CNN features keeps high recall but is not effective in distinguishing smoke regions. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**YOLOv5 tiny** & **Vehicle Matching** & **(ST)2 Net** & **DR** & **FAR** (\(\downarrow\)) & **Precision** & **F1** \\ \hline \(\surd\) & & & 0.9087 & 0.1436 & 0.3967 & 0.5370 \\ \(\surd\) & \(\surd\) & & 0.8815 & 0.1211 & 0.4255 & 0.5616 \\ \(\surd\) & \(\surd\) & \(\surd\) & 0.7713 & 0.0192 & 0.7936 & 0.7801 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Evaluation of the individual modules of our method on LaSSoV-video test set. Figure 10: Detection samples in LaSSoV-video test set. Yellow bounding boxes denote those false positives of YOLOv5 tiny, and green bounding boxes refer to those results of our cascaded method. In other words, the yellow boxes are filtered by the vehicle matching module and (ST)2 Net. As mentioned above, the default temporal input length is 3 for our (ST)\({}^{2}\)Net modules. In fact, this short temporal extent is unusual in other similar tasks such as video action recognition [49; 50; 51]. Thus, we evaluated the temporal length as shown in Figure 12. We do not show the false alarm rates since all of these models obtain less than 0.02 in FAR. We took ResNet-18 to fine tune an image classification model for length 1. As can be seen in Figure 12, the F1 score and precision are improved when increasing the length but are saturated at length 3. The detection rate is slightly degraded with increasing input length. We show some temporal extent samples in Figure 13 to investigate why more frames are not helpful for performance. We find that most of the extented frame regions are not related to smoke, which is mainly caused by the high-speed vehicles and low frame rate cameras. As known in image classification, a large input size is helpful for performance. To verify this, we rescaled the input to 256 \(\times\) 256, and randomly cropped 224 \(\times\) 224 regions to train our (ST)\({}^{2}\)Net-suffix model with temporal length 3. Replacing the default (ST)\({}^{2}\)Net-suffix with this one, we achieved 0.7552, 0.0176, 0.8037, and 0.7763, in DR, FAR, Precision,  
Figure 11: The comparison of different short-term spatial-temporal models in metrics of DR, precision, and F1 score. Figure 12: Evaluation of temporal length for our (ST)\({}^{2}\)Net-suffix. Figure 13: Temporal extent examples for detected smoke regions in time \(t\). and F1, respectively. Compared to the results in Table 3, we find that only Precision is slightly improved while there is an obvious degradation in DR or F1.