# A Review of Video Object Detection: Datasets, Metrics and Methods
## 1 Introduction  
Video object detection involves detecting objects using video data as compared to conventional object detection using static images. Two applications that have played a major role in the growth of video object detection are autonomous driving [1; 2] and video surveillance [3; 4]. In 2015, video object detection became a new task of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC2015) [5]. With the help of ILSVRC2015, studies in video object detection have further increased. Earlier attempts in video object detection involved performing object detection on each image frame. In general, object detection approaches can be grouped into two major categories: (1) one-stage detectors and (2) two-stage detectors. One-stage detectors (e.g., [6; 7; 8; 9; 10; 11; 12]) are often more computationally efficient than two-stage detectors (e.g., [13; 14; 15; 16; 17; 18; 19; 20; 21]). However, two-stage detectors are shown to produce higher accuracies compared to one-stage detectors. However, using object detection on each image frame does not take into consideration the following attributes in video data: (1) Since there exist both spatial and temporal correlations between image frames, there are feature extraction redundancies between adjacent frames. Detecting features in each frame leads to computational inefficiency. (2) In a long video stream, some frames may have poorquality due to motion blur, video defocus, occlusion, and pose changes [22]. Detecting objects from poor quality frames leads to low accuracies. Video object detection approaches attempt to address the above challenges. Some approaches make use of the spatial-temporal information to improve accuracy, such as fusing features on different levels, e.g., [22; 23; 24; 25]. Some other approaches focus on reducing information redundancy and improving detection efficiency, e.g., [26; 27; 28]. Initially, video object detection approaches have relied on handcrafted features, e.g., [29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42]. With the rapid development of deep learning and convolutional neural networks, deep learning models have been shown to be more effective than conventional approaches for various tasks in computer vision [43; 44; 45; 46; 47; 48; 49; 50], speech processing [51; 52; 53; 54; 55], and multi-modality signal processing [56; 57; 58; 59; 60; 61]. A number of deep learning-based video object detection approaches were developed after the ILSVRC2015 challenge. The training is normally done offline. The testing phase on modern GPUs even of complex networks has been shown to meet the 30 frames per sec rate of video, e.g., [26], allowing the real-time deployment of networks. The great value of video object detection approaches is further presented in some specific applications. For example, hand segmentation [62; 63] is well realized with the help of the optical flow to enhance the feature maps as per the video object detection method [28]. Human pose estimation in videos [64] is another successful application, which draws lessons from [22; 28] to solve the motion blur, occlusion and other specific challenges occurring in videos. Furthermore, instance-level human parsing [65] starts from the similar approaches. Mutual assistance of tracking and detection [26] is well employed in multiple people tracking [66]. Deep learning-based video object detection approaches can be divided into flow based [22; 27; 28; 67; 68; 69], LSTM (Long Short Term Memory)-based [70; 71; 72; 73], attention-based [74; 75; 76; 25; 77; 78], tracking-based [79; 78; 79; 80; 26; 81; 82] and other methods [83; 84; 85; 86; 87; 88; 89; 90]. A review of these approaches is provided in this paper. Section 2 covers the existing datasets and evaluation metrics for video object detection. Then, in Section 3, the existing video object detection approaches are described. The accuracy and processing time of these approaches are compared in Section 4. Section 5 mentions the future trends or needs relating to video object detection.