# A Review of Video Object Detection: Datasets, Metrics and Methods
## 3 Video Object Detection Methods
### Other Methods  
Apart from the frameworks described above, some methods are presented that are based on a combination of multiple methods described above [24; 121]. The method in [24] is based on the optical flow and tracking methods. The methods in [121] (Attentional LSTM) and [122] (TSSD) are based on the attention and LSTM methods. In addition, these other methods appear in the literature [36; 83; 84; 85; 86; 87; 88; 89; 90]. The methods in [83] and [87] discuss ways to align and enhance feature maps. While the method in [90] studied the effect of the input image size by selecting a size to achieve a better speed-accuracy trade-off. The method in [83] named STSN (spatiotemporal sampling networks) is shown in Figure 10. This method aligns feature maps between adjacent frames. Similar to the FGFA method in [22], it relies on the idea that detection on a single frame would have difficulties dealing with noise sources such as motion blur and video defocus. Multiple frames are thus utilized for feature enhancement to achieve better performance. Unlike FGFA, which uses the optical flow method to align feature maps, deformable convolution is employed for feature alignment in [83]. First, a sharing feature extraction network is applied to extract feature maps on a current frame and adjacent frames. Then, the two feature maps are concatenated per channel and a deformable convolution is performed. The result of the deformable convolution is used as the offset for the second deformable convolution operation to align the feature maps. Furthermore, augmented feature maps are obtained by aggregating the features in the same way as FGFA. Compared with FGFA, STSN uses deformable convolution to align the features of two adjacent frames implicitly. Although it is not as intuitive as the optical flow method, it is also found to be effective. According to the experimental results reported, STSN still achieved a higher mAP than FGFA (78.9% vs. 78.8%) without relying on the optical flow information. In addition, without the assistant of the temporal post-processing, STSN obtained a better performance than the D&T baseline [81], 78.9% vs. 75.8%. Different from [83], by using the deformable convolution to propagate the temporal information, the Spatial-Temporal Memory Network (STMN) was considered in [87], which involved an RNN architecture with a Spatial-Temporal Memory module (STMM) to incorporate the long-term temporal information. The Spatial-Temporal Memory Network (STMN) operates in an end-to-end manner to model the long-term information and align the motion dynamics for video object detection. STMN  
Figure 10: STSN (spatiotemporal sampling networks) framework [83]. Figure 9: D or T framework [80]. is the core module in STMN, a convolutional recurrent computation unit which fully utilizes the pretrained weights learned from static image datasets such as ImageNet [93]. This design is essential to address the practical difficulties of learning from video datasets, which largely lack the diversity of objects within the same category. STMM receives the feature maps of a current frame at time step \(t\) and the spatial-temporal memory \(M_{t-1}^{*}\) with the information of all the previous frames. Then, the spatial-temporal memory \(M_{t}^{*-}\) of the current time step is updated. In order to capture the information of both later frames and previous frames at the same time, two STMMs are used for bidirectional feature aggregation to produce the memory \(M\), which is employed for both classification and bounding box regression. Therefore, the feature maps are propagated and aggregated by combining the information across multiple video frames. Evaluated on the ImageNet VID dataset, STMN has achieved the current start-of-the-art accuracy. All the algorithms described above start from how to propagate and aggregate feature maps. In [90], video object detection was examined from another point of view. Similar to [123], the effect of input image size on the performance of video object detection was studied in [90].