# Fine-Grained Action Recognition by Motion Saliency and Mid-Level Patches
## 2 Related Work  
The studies on action recognition are numerous, such as fine-grained action recognition [1; 2; 9; 10; 26; 27] and coarse-grained action recognition [28; 29]. In the following, we review the work that is most related to the current study. [PERSON] et al. [1] presented a database on kitchen operations to evaluate fine-grained action classification. It is composed of 65 different actions that take place continuously within 8 hours of recording. This database is mainly used for fine-grained action recognition [3; 9; 10; 11; 26; 30; 31; 32; 33]. In the literature [1], two schemes for action recognition are proposed, one is based on the human pose and the other is based on the DT feature. On the basis of these and upon further study, [PERSON] et al. [27] proposed a hand-centric approach for fine-grained activity classification and composite activity recognition. [PERSON] et al. [9] proposed a representation and classification pipeline that seamlessly incorporated localized semantic information for fine-grained action recognition. However,  
Figure 2: Our fine-grained action recognition method, which includes motion region segmentation based on motion saliency, object proposal generation by the idea of the Huffman algorithm, mid-level patch detector training with K-means clustering and SVM, and action recognition with a graph structure. these methods [1; 9; 27] require explicit object detection or pose estimation. Interaction Part Mining [10] involves a fine-grained action recognition pipeline by mid-level part mining. It applies Max-N pooling to keep important candidate parts, which would generate redundant information. Following the impressive results of deep learning on the task of image classification [34] and other research fields [35; 36], many efforts have been made to train deep networks for the task of action recognition. [PERSON] et al. [3] proposed a temporal pooling function, which uses the evolution of video temporal structure to represent the videos. [PERSON] et al. [30] proposed a new network structure, which encodes spatiotemporal features by sharing weights between the learned parameters. [PERSON] et al. [2] proposed an end-to-end fine-grained action detection system based on RNNs (Recurrent Neural Networks), which parses the interactive objects frame by frame in a video to describe a specific action for fine-grained action recognition. These algorithms achieved good recognition effects on the fine-grained actions while fusing the DT features and the CNN (Convolutional Neural Network) features, but the results of using the two features respectively were not satisfactory. There is work that utilized spatiotemporal segments to capture different interaction components of human action and represent videos [11; 37; 38]. Some work [9; 10] focused on the problem of modeling context information between actions and objects and investigated how to parse or track operated objects. [PERSON] et al. [39] detected proposals at the frame-level and scored them with static and motion CNN features, and then tracked high-scoring proposals throughout the video to generate spatiotemporal proposals. Multi-level representation of action is popular in action recognition because it can extract the participation components of action in different granularities [40]. [PERSON] et al. [11] extracted mid-level elements from region proposals to represent actions. [PERSON] et al. [41] proposed a hierarchical spatiotemporal segmentation method for action recognition and location. Spatiotemporal segments included the whole human body and body parts with different granularities. The action participation segments-based method is also used in this paper. Our mid-level patches include human body parts and motion objects that participate in action, and the static objects can be excluded based on motion saliency. Mid-level patch mining employs object proposals to guide the detection of patches.