Temporal-scale Convolutional Networks for Human Action Recognition Based on Key-Frame Extraction

Human action recognition is an important part of intelligent video analysis. Recently, deep learning has made significant progress in this field, and state-of-the-art methods are based on the two-stream convolutional networks. In long-term action recognition, existing approaches mainly use video frames obtained by averaging or sampling as input, which may lose important information in the sampling interval. By defining the amount of video information, we propose a method of segment division and key frame extraction for action recognition is proposed, where multi-temporal-scale two-stream networks are used to extract features. We achieve 94.2% accuracy on a widely used action recognition benchmark (Split1 of UCF-101). Introduction Human action recognition is an important part of intelligent video analysis. The core is to segment video into image sequence and classify the sequence by extracting the temporal and spatial features. In the early years, hand-crafted features were commonly used to achieve action classification, such as using spatio-temporal descriptors [1]. In recent years, with the great success of the deep neural network in the field of image recognition, the use of deep learning for video behavior recognition has achieved remarkable results. One of the major difference between video and static images is that video has temporal components. To extract temporal features, video action recognition needs to deal with a large number of video frames and get implied cues. Existing researches usually divide the video into several segments, and extract features of each detected segment. The early methods often use a single Convolutional Network (ConvNet) to extract features, and the results are obtained based on the mean value of the extracted results. The two-stream method [2] introduce optical flow as an auxiliary input, and use two ConvNet architectures to extract the temporal as well as the spatial information in the segments, respectively. LRCN [3] introduced a Recurrent Neural Network (RNN) to extract time characteristics. The above methods use average sampling to obtain the frame which is need to be detected. Most existing researches focus on how to extract temporal information, with little attention to the quality of detected frames. Whether the detected frame contains the key information of the current action is an important factor that affects the result of action recognition. Here we call the video frame or optical flow sequence that contains key action information “key frame”, and call the sampling interval “segment”. The difference between video and static images is not just the temporal components. Because video comes from continuous shooting, it is easy to have disturbances such as motion blur, loss of focal length and so on. Even video compression can bring noise. However, there is a difference in the amount of information between different video frames. Meanwhile, the amount of motion information contains among different video segments is often very different because of the uneven movement of natural behavior. Traditional key frame extraction methods are mostly based on global feature [4], such as brightness histogram and gradient histogram. It is difficult to find fragments of key information in action sequence with small scene change.