# Temporal Action Detection in Untrimmed Videos from Fine to Coarse Granularity
## 2 Previous Works  
Temporal action detection in videos is an active area of research. The earlier works focus on hand-crafted feature representations for action. Recently, much progress of action detection has been facilitated by convolutional neural network (CNN) feature. Meanwhile, the methodologies of spatial object detection were extended to temporal action detection. As described in Section 1, most of the prior action detection methods fall into four types of frameworks: classification then post-processing, proposal then classification, single-stream, and temporal upsampling. In this section, we outline the major contributions of these frameworks.  
Classification then Post-processing. [PERSON] et al. [2] used the C3D [21], a type of 3D Convolutional Neural Network architecture, to extract features of video segments and trained LSTM to classify the segments. Finally, they determined the temporal boundaries of actions by post-processing. [PERSON] et al. [22] firstly conducted temporal sliding window scanning to split a video into short segments. Then, they adopted CNN features and the Fisher Vector [23] of improved Dense Trajectories (iDT) [24] to represent each segment. Finally, they trained multiple one-vs-all Support Vector Machine (SVM) to perform segment-level recognition and generated the detection results using the predictions of segments.  
Figure 2: Overview of our proposed detection method. In each of proposal and classification steps, our method performs classification from segment-level to window-level.  
Proposal then Classification. [PERSON] et al. [3] exploited three segment-level C3D networks to address the problems of proposal, classification and localization, respectively, and used the classification network to serve as initialization for the localization network. [PERSON] et al. [4] trained a C3D via multi-task learning to perform proposal, classification and localization in parallel. [PERSON] et al. [25] computed fully-convolutional 3D CNN features and proposed temporal regions, then conducted pooling of the features within these 3D regions to predict the categories of actions. Also some works [18; 26; 27] focused on generating high quality action proposals. [PERSON] et al. [26] and [PERSON] et al. [27] used LSTM networks to encode a video stream and produced proposals inside the video stream. [PERSON] et al. [18] proposed to jointly predict action proposal and to refine the temporal action boundary by temporal coordinate regression.  
Single-stream. [PERSON] et al. [9] trained deep recurrent architecture based on enforcing semantic constraints on intermediate modules that are gradually relaxed as learning progresses, and processed the video in a single stream, which directly outputs the temporal boundaries of actions and corresponding action categories. [PERSON] et al. [8] extracted snippet-level action score features via CNN and 3D CNN networks and designed a single shot action detector network, in which base layers are used to reduce the temporal dimension of the input data; anchor layers output multiple scale feature map associated with anchor instances and prediction layers are used to predict the categories, locations, and confidences of anchor instances.  
Temporal Upsampling. [PERSON] et al. [13] designed a Convolutional-De-Convolutional (CDC) network that places CDC filters on top of C3D network. The proposed CDC filter performs temporal upsampling and spatial downsampling simultaneously to predict actions at the frame-level. [PERSON] et al. [14] proposed a Temporal Preservation Convolution (TPC) network that equips C3D with TPC filters. TPC filter can fully preserve the temporal resolution and downsample the spatial resolution simultaneously, enabling frame-level action localization with the minimal loss of time information.