Deep convolutional networks have achieved great success for image recognition. However, for action recognition in videos, their advantage over traditional methods is not so evident. We present a general and flexible video-level framework for learning action models in videos. This method, called temporal segment network (TSN), aims to model long-range temporal structures with a new segment-based sampling and aggregation module. This unique design enables our TSN to efficiently learn action models by using the whole action videos. The learned models could be easily adapted for action recognition in both trimmed and untrimmed videos with simple average pooling and multi-scale temporal window integration, respectively. We also study a series of good practices for the instantiation of TSN framework given limited training samples. Our approach obtains the state-the-of-art performance on four challenging action recognition benchmarks: HMDB51 (71.0 ), UCF101 (94.9 ), THUMOS14 (80.1 ), and ActivityNet v1.2 (89.6 ). Using the proposed RGB difference for motion models, our method can still achieve competitive accuracy on UCF101 (91.0 ) while running at 340 FPS. Furthermore, based on the temporal segment networks, we won the video classification track at the ActivityNet challenge 2016 among 24 teams, which demonstrates the effectiveness of TSN and the proposed good practices.
To overcome the limitation of local descriptors, several mid-level representations have been proposed for action recognition @cite_29 @cite_10 @cite_25 @cite_31 @cite_67 @cite_15 @cite_19 . Raptis @cite_29 grouped similar trajectories into clusters, each of which was regarded as an action part. Jain @cite_10 extended the idea of discriminative patches into videos and proposed discriminative spatio-temporal patches for representing videos. Zhang @cite_31 proposed to discover a set of mid-level patches in a strongly-supervised manner. Similar to 2-D poselet @cite_37 , they tightly clustered action parts using human joint labeling, dubbed . Wang @cite_25 proposed a data-driven approach to discover those effective parts with high motion salience, known as motionlet . Zhu @cite_67 proposed a two-layer representation for action recognition. The weakly-supervised actons were learned via a max-margin multi-channel multiple instance learning framework. Wang @cite_15 proposed a multiple level representation called as MoFAP by concatenating motion features, atoms, and phrases. Sadanand @cite_19 presented a high-level video representation called as Action Bank by using a set action templates to describe the video content. In summary, these mid-level representations have the merits of representative and discriminative power, but still depends on the low-level hand-crafted features.
@cite_67 @cite_37 @cite_15 @cite_29 @cite_19 @cite_31 @cite_10 @cite_25
2139893168 2535410496 2175640409 2055753778 2063153269 2137591992 1998605630 2139594308
This paper proposes a multi-level video representation by stacking the activations of motion features, atoms, and phrases (MoFAP). Motion features refer to those low-level local descriptors, while motion atoms and phrases can be viewed as mid-level "temporal parts". Motion atom is defined as an atomic part of action, and captures the motion information of video in a short temporal scale. Motion phrase is a temporal composite of multiple motion atoms defined with an AND OR structure. It further enhances the discriminative capacity of motion atoms by incorporating temporal structure in a longer temporal scale. Specifically, we first design a discriminative clustering method to automatically discover a set of representative motion atoms. Then, we mine effective motion phrases with high discriminative and representative capacity in a bottom-up manner. Based on these basic units of motion features, atoms, and phrases, we construct a MoFAP network by stacking them layer by layer. This MoFAP network enables us to extract the effective representation of video data from different levels and scales. The separate representations from motion features, motion atoms, and motion phrases are concatenated as a whole one, called Activation of MoFAP. The effectiveness of this representation is demonstrated on four challenging datasets: Olympic Sports, UCF50, HMDB51, and UCF101. Experimental results show that our representation achieves the state-of-the-art performance on these datasets. This paper proposes motionlet, a mid-level and spatiotemporal part, for human motion recognition. Motion let can be seen as a tight cluster in motion and appearance space, corresponding to the moving process of different body parts. We postulate three key properties of motion let for action recognition: high motion saliency, multiple scale representation, and representative-discriminative ability. Towards this goal, we develop a data-driven approach to learn motion lets from training videos. First, we extract 3D regions with high motion saliency. Then we cluster these regions and preserve the centers as candidate templates for motion let. Finally, we examine the representative and discriminative power of the candidates, and introduce a greedy method to select effective candidates. With motion lets, we present a mid-level representation for video, called motionlet activation vector. We conduct experiments on three datasets, KTH, HMDB51, and UCF50. The results show that the proposed methods significantly outperform state-of-the-art methods. A citing sentence is one that appears in a scientific article and cites previous work. Citing sentences have been studied and used in many applications. For example, they have been used in scientific paper summarization, automatic survey generation, paraphrase identification, and citation function classification. Citing sentences that cite multiple papers are common in scientific writing. This observation should be taken into consideration when using citing sentences in applications. For instance, when a citing sentence is used in a summary of a scientific paper, only the fragments of the sentence that are relevant to the summarized paper should be included in the summary. In this paper, we present and compare three different approaches for identifying the fragments of a citing sentence that are related to a given target reference. Our methods are: word classification, sequence labeling, and segment classification. Our experiments show that segment classification achieves the best results. This paper proposes motion atom and phrase as a mid-level temporal part'' for representing and classifying complex action. Motion atom is defined as an atomic part of action, and captures the motion information of action video in a short temporal scale. Motion phrase is a temporal composite of multiple motion atoms with an AND OR structure, which further enhances the discriminative ability of motion atoms by incorporating temporal constraints in a longer scale. Specifically, given a set of weakly labeled action videos, we firstly design a discriminative clustering method to automatically discover a set of representative motion atoms. Then, based on these motion atoms, we mine effective motion phrases with high discriminative and representative power. We introduce a bottom-up phrase construction algorithm and a greedy selection method for this mining task. We examine the classification performance of the motion atom and phrase based representation on two complex action datasets: Olympic Sports and UCF50. Experimental results show that our method achieves superior performance over recent published methods on both datasets.