Human behaviour recognition with mid‐level representations for crowd understanding and analysis

IET Image ProcessingVolume 15, Issue 14 p. 3414-3424 ORIGINAL RESEARCH PAPEROpen Access Human behaviour recognition with mid-level representations for crowd understanding and analysis Bangyong Sun, Bangyong Sun College of Printing, Packaging Engineering and Digital Media, Xi'an University of Technology, Xi'an, Shaanxi, 710048 China The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, Shaanxi, 710119 ChinaSearch for more papers by this authorNianzeng Yuan, Nianzeng Yuan College of Printing, Packaging Engineering and Digital Media, Xi'an University of Technology, Xi'an, Shaanxi, 710048 ChinaSearch for more papers by this authorShuying Li, Corresponding Author Shuying Li angle_lisy@163.com School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, Shaanxi, 710121 P.R. China Correspondence Shuying Li, School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, 710121, Shaanxi, P.R. China. Email: angle_lisy@163.com Siyuan Wu, The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China. Email: wusiyuan@opt.ac.cnSearch for more papers by this authorSiyuan Wu, Corresponding Author Siyuan Wu wusiyuan@opt.ac.cn orcid.org/0000-0002-9041-8587 The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, Shaanxi, 710119 China Correspondence Shuying Li, School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, 710121, Shaanxi, P.R. China. Email: angle_lisy@163.com Siyuan Wu, The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China. Email: wusiyuan@opt.ac.cnSearch for more papers by this authorNan Wang, Nan Wang orcid.org/0000-0001-5601-7838 The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, Shaanxi, 710119 China University of Chinese Academy of Sciences, 19A Yuquanlu, Beijing, 100049 P. R. ChinaSearch for more papers by this author Bangyong Sun, Bangyong Sun College of Printing, Packaging Engineering and Digital Media, Xi'an University of Technology, Xi'an, Shaanxi, 710048 China The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, Shaanxi, 710119 ChinaSearch for more papers by this authorNianzeng Yuan, Nianzeng Yuan College of Printing, Packaging Engineering and Digital Media, Xi'an University of Technology, Xi'an, Shaanxi, 710048 ChinaSearch for more papers by this authorShuying Li, Corresponding Author Shuying Li angle_lisy@163.com School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, Shaanxi, 710121 P.R. China Correspondence Shuying Li, School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, 710121, Shaanxi, P.R. China. Email: angle_lisy@163.com Siyuan Wu, The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China. Email: wusiyuan@opt.ac.cnSearch for more papers by this authorSiyuan Wu, Corresponding Author Siyuan Wu wusiyuan@opt.ac.cn orcid.org/0000-0002-9041-8587 The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, Shaanxi, 710119 China Correspondence Shuying Li, School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, 710121, Shaanxi, P.R. China. Email: angle_lisy@163.com Siyuan Wu, The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China. Email: wusiyuan@opt.ac.cnSearch for more papers by this authorNan Wang, Nan Wang orcid.org/0000-0001-5601-7838 The Key Laboratory of Spectral Imaging Technology CAS, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, Shaanxi, 710119 China University of Chinese Academy of Sciences, 19A Yuquanlu, Beijing, 100049 P. R. ChinaSearch for more papers by this author First published: 25 February 2021 https://doi.org/10.1049/ipr2.12147AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Crowd understanding and analysis have received increasing attention for couples of decades, and development of human behaviour recognition strongly supports the application of crowd understanding and analysis. Human behaviour recognition usually seeks to automatically analyse ongoing movements and actions in different camera views by using various machine learning methodologies in unknown video clips or image sequences. Compared to other data modalities such as documents and images, processing video data demands much higher computational and storage resources. The idea of using middle level semantic concepts to represent human actions from videos is explored and it is argued that these semantic attributes enable the construction of more descriptive methods for human action recognition. The mid-level attributes, initialized by a cluster processing, are built upon low level features and fully utilize the discrepancies in different action classes, which can capture the importance of each attribute for each action class. In this way, the representation is constructed to be semantically rich and capable of highly discriminative performance even paired with simple linear classifiers. The method is verified on three challenging datasets (KTH, UCF50 and HMDB51), and the experimental results demonstrate that our method achieves better results than the baseline methods on human action recognition. 1 INTRODUCTION Visual recognition and understanding [1-5] of human behaviours have been widely used in crowd analysis [6-8] and security surveillance. Moreover, the development of human action recognition can support the application of crowd understanding and analysis. In simple terms, the objective of human behaviour recognitions is to correctly classify the video into its action category, and the video is fragmented to contain only one implementation of human movement. In more general cases, the purpose of human action recognition is focused on performing the continuous recognition on every appeared human action of the input video from start to end. Human action recognition task is of significant meaning in several applications. For instance, a train station surveillance system can automatically recognize suspicious activities like “people suddenly run panicky ”or "a person waving his/her harms with swords in hands.” Recognition of human actions is also helpful in the real-time monitoring of patients, children and elderly persons. What is more important, a high performance action recognition system can make the construction of vision-based intelligent environments and gesture-based human computer interfaces become possible. Though a lot of enthusiasm has been generated in the vision community by recent advances in machine recognition of human actions [9-12], there are still several important issues left to be addressed. One of the most crucial issues is to find algorithms that can robustly overcome the variability of features with the same action class label. This is because there are large variations in performance. For example, hands waving motions can differ in speed and radian length. In addition, there are anthropometric differences between individuals. Moreover, similar observations can be made for different actions, especially for non-period actions or actions that are adapted to the environment (e.g. walking with a dog or handing somebody something). A thorough human action recognition framework should be able to tolerate variations within one class and distinguish actions of different classes. For increasing numbers of action classes, this will even be more challenging as the overlap between classes will be higher. Many previous action recognition frameworks directly match low-level features with action class labels [13-16], in which abundant visual spatiotemporal information can hardly be generalized by the raw low-level features. To overcome the aforementioned drawback, recent works show that attributes built upon the raw low-level features can act as higher semantic concepts and bridge the gap between low-level features and action class labels. Some works [17-21] even treated attributes as middle-level features for classification or learning problems. Lampert et al. [22] proposed the direct attribute prediction model which was learned to predict the presence of each attribute, and then used these predictions to train object models. Liu et al. [23] treated the action attributes as latent variables, whose classifiers are pre-trained by linear SVMs with outputs being the inputs of latent SVM. The aforementioned methods have proved that the attributes are functional in many ways and effective for recognizing actions. In this work, we simultaneously consider the relationship between low-level features and attributes to bridge the gap between low-level features and action class labels. Attributes can put up a semantically expressive bridge between raw data and higher level representations as they focus on depicting characters of all instances within the same class rather than naming one directly. But the intra-class variability can still make the attributes to be inaccurate for specific members of the training classes. As a result, some action samples may be associated with slightly different sets of attributes even if they share the same action class label. To address this problem, we construct a mid-level representation by combining low level features and attributes together. Figure 1 shows the procedure to generate the mid-level representation of our method. Each action attribute corresponds to a set of low-level features which share the same action class label. Meanwhile, every mid-level representation contains both intra-class and inter-class information, which make it very discriminative. The contributions of this paper would be summarized as follows: The mid-level representations can conquer the appearance variation across cameras to some extent and represent the discriminative information of motion patterns in the video sequences The proposed model is discriminative by making the same actions more similar after the middle-level representation while different motion patterns are more distinct. In this case, it can reduce the negative impact of the similar actions Extensive experiments have been conducted on three action datasets to verify the performance of the proposed method. The experimental results show that our mid-level representation of human action patterns is capable of constructing a powerful action recognition framework. In addition, our method can give prominent performance on several standard datasets (KTH, UCF50, HMDB51). FIGURE 1Open in figure viewerPowerPoint Illustration of our method. Our method not only introduces a set of attributes to bridge the gap between low-level features and action classes, but also formulate the connection between attributes and action classes The rest of this paper is organized as follows. In Section II, we review recent works on action recognition. In Section III, we introduce the proposed method in detail, including the construction of mid-level representation and the whole algorithm framework for classifying human actions. Then we present experimental results on several data bases for video based action recognition in Section IV. Finally, we conclude this paper in Section V. 2 RELATED WORKS The recognition of human activities from video sequences is one of the most encouraging applications of computer vision which effectively support the crowd understanding and analysis. In recent years, this task has attracted the attention of researchers from academia, security agencies, industry and the general populace as well. According to the features used for recognizing human actions, the methods in related works can be mainly classified into two categories: global feature-based methods and local feature-based methods. And method based on neural network also obtain attention because of their outstanding performance in many computer vision tasks. Moreover, methods of crowd understanding and analysis are essential to be stated. 2.1 Global feature based method Global feature based methods [24-26] encode the visual observation as a whole and the region of interest (ROI) is usually detected through background subtraction or tracking. Common global features are obtained from edges, silhouettes or optical flow [27]. The representations are powerful since they encode much of the information. The work of Bobick and Davis [27, 28] is one of the earliest methods utilizing silhouettes. They extracted silhouettes from a single view and accumulated the differences between subsequent frames of a motion clip with the generated binary motion energy image (MEI) and motion history image. Instead of silhouette, the observation within the ROI can also be described with motion information—optical flow [28]. Flow methods are usually executed when background subtraction cannot be performed. Efros et al. [28] calculated optical flow in person-centered video frames on sports footage, where persons in the frame were very small. There are also a lot of work combining flow and shape descriptors together, which can overcome the limitations of single representation. Optical flow was employed in ref. [29], and it researched what makes a flow algorithm good for action recognition. In addition, the spatio-temporal volume approach is also a global representation, despite the fact that this approach shares many similarities with local approaches. 2.2 Local feature based method Local representations describe the object as a collection of individual patchs. To estimate the local features, spatiotemporal interest points should be detected first. After that, local cubes will be calculated around these points. Then, the cubes are combined into a final feature. Finally, bag-of-feature approaches are always followed. Local features are less sensitive to noise and partial occlusion, and do not strictly require background subtraction or tracking. However, as they depend on the extraction of a sufficient amount of relevant interest points, pre-processing is sometimes indispensable. To detect the interest points or salient regions, a large variety of methods [30-34] have been put forward. Laptev et al. [28] modified the 2D spatial interest points to detect 3D spatiotemporal interest points for action recognition. The detected local features preserve some rotation invariance, which is robust to the succeeding action recognition. Wang et al. [34] presented a detector to obtain spatio-temporal interest points by employing a Geometric Algebra for action recognition in multi-channel videos. Yi et al. [32] developed the salient point detector by taking advantage of trajectory space information based multiple kernel learning. After the detection of interesting points or salient regions, many methods have been proposed to describe them. Scovanner et al. [35] extended the SIFT descriptor to 3D-SIFT, by using the spatiotemporal information which is denoted by a sub-histogram. Laptev et al. [36] applied histograms of gradient orientations (HoG) and histograms of optic flows to describe the information of local motion and cubes of the space-time neighbourhoods of detected interest points, and Klaser et al. [37] ameliorated HoG to the 3D case. 2.3 Deep learning based method C2LSTM (Correlational Convolutional) [38] can effectively perceive the spatial features, temporal dependencies and motion structure. Zufan et al. [39] address the human action recognition by applying Conv-LSTM and FC-LSTM (full connection) which are combined with different attention mechanisms. AGC-LSTM [40] (attention enhanced graph convolutional) can explore the relationship and capture discriminative features in time-space domain. Meng et al. [41] propose a novel sample fusion network to achieve data augmentation for skeleton-based human action recognition (SHAR). Graph convolutions technology are considered to address SHAR, G3D [42] applys cross-time-space edges as skip connections which can propagate direct information for disentangling the importance of nodes. 2.4 Crowd understanding and analysis Besides human behaviour recognition, crowd understanding and analysis have many subproblems such as crowd counting and person re-identification. Wang et al. [6] proposed a large-scale benchmark for crowd localization and counting named NWPU-Crowd. Synthetic data [7] are also adopted for pixel-wise crowd understanding. In ref. [8], pixel shuffle decoder and density-aware curriculum learning are designed to obtain the potential of crowd counting models. Moreover, Zhao et al. [43] present PISNet to address the interference of other pedestrians for person re-identification. And Li et al. [44] focus on different and similar components for person re-identification. 3 PROPOSED METHOD In this section, details about the proposed method are introduced. Framework of our method is shown in Figure 2. In III-A, we illustrate the ‘low-level’ features used in this work. In III-B, we present our method for human action recognition in detail, which explicitly employs the attribute-based mid-level representation. In III-C, we give the optimization procedures for the recognition method. FIGURE 2Open in figure viewerPowerPoint Flowchart of the proposed model. Attributes are created from low-level features, then mid-level representations are constructed with the attributes, finally mid-level representations are used to learn the action recognition model 3.1 Low-level features To extract low-level features, we follow the procedure described in ref. [45] and use the features named ‘action bank.’ In this implementation, action bank is obtained using the action spotting detector [46] following the previous work [45]. As described in [45], action bank is comprised of many individual action detectors sampled broadly in semantic space as well as viewpoint space and is constructed to be semantically rich. What's more, it is capable of highly discriminative performance. So it is capable of being the basis of a powerful action recognition method. 3.2 Attribute-based mid-level representations Let X ∈ R d × N represent the raw low-level features extracted from action video clips. Each column of X corresponds to the feature representation of a human action and N is the number of training samples. The low-level action bank feature x i ∈ X represents the i-th sample from training set. y i is the corresponding action class label of x i . Most previous works just represent actions with raw extracted features and simply connect the feature vectors to action class labels. In this case, the raw features cannot convey all the information the recognition needs, so that there exists a gap between the low-level features and class labels. In this paper, we consider that human actions can be better described by attribute-based mid-level representations, since these mid-level representations can act as higher semantic concepts and bridge the gap between low-level features and action class labels. Attributes are mainly used to describing an instance (e.g. rather than naming one. In one human action recognition context, for example, a ‘leg motion’ may be shared between ‘jogging’ action and ‘running’ action. This kind of descriptions of shared components or parts between similar or different classes can be called as attributes. And attribute learning has been popularized recently as an effective method for image and video understanding [22]. In this paper, we design an action attribute space A d × r as a semantic metric space in which each column denotes one kind of information of an action class. The attribute space is composed of r attributes C = c i 1 r , (1)and each attribute is mined from a group of low-level features that share the same action class label by using clustering algorithm. The clustering centers are defined as attributes because they are more discriminative than low-level features and able to capture the inherent intra-class variably of each action class. Thus, they are capable of representing the complete information of action classes. Given a training set D = x i , y i i = 1 N , y i ∈ − 1 , + 1 , (2)where + 1 denotes the positive samples and − 1 denotes the negative ones, we try to build a classification model to recognize an unknown action x. To this end, both attributes and low-level features are used to generate mid-level representations Φ C ( X ) , which represent the action video clips with higher-level concepts. The mid-level representation Φ C ( x ) is defined as Φ C x = s x , c 1 , s x , c 2 , … , s x , c r , (3) where c i ∈ A d × r is the attribute generated from X by K-means clustering, and r is the total number of cluster centers. s ( x , c ) is a similarity measure function, and we measure the similarity by Gaussian Hausdorff function, that is, s x , c = exp − x − c 2 / δ , (4) where δ is set as the average distance between all pairs of instances. For the training set D, we define Φ C X = Φ C x 1 ; Φ C x 2 ; … ; Φ C x N T , (5)where N is the number of training samples. For each action class l, there is a label matrix Y l = y 1 l , y 2 l , … , y N l T , (6)and y i l ∈ − 1 , 1 , (7)where + 1 denotes the positive samples and − 1 denotes the negative ones. To recognize all different kinds of action classes, we learn a classifier for each action class. And the classifier is denoted as f l Φ C X = w l T Φ C X , (8)where w 1 ∈ R r × 1 is the parameter vector, w l T denotes the transpose of w l , f l is a linear model. And given the training set D, the maximum likelihood estimate for the weight vector w ̂ , is that which minimizes the sum squared error: J s s e w = ∑ i = 1 N f l − y i 2 . (9) Let W denote the matrix of parameter vectors, [ w 1 , w 2 , … , w m ] . To find the best parameter matrix W, we optimize all loss functions and regularize them with l 2 , 1 -norm: min W W T ϕ C X − Y 2 , 1 , (10)where W = w 1 , w 2 , … , w m , (11) Y = Y 1 , Y 2 , … , Y m T , (12)and m is the total number of action classes. If Z ∈ R D × T , then ∥ Z ∥ 2 , 1 is the l 2 , 1 -norm of matrix Z, that is, Z 2 , 1 = ∑ d ∑ t Z d t 2 . (13) This norm first computes the l 2 norm of the loss function matrix in each dimension, then computes the l 1 norm which is well known for sparse solutions. The regularization achieves the minimum if and only if W T Φ C X = Y , (14)for all corresponding elements. Thus, the l 2 , 1 -norm has an important property so that it can provide the score of the mid-level representation, and find the best parameter matrix W. However, there is still a latent defect in this formulation. The K-means clustering technique is a non-convex method, so the process to generate action attributes may result in a suboptimal performance. To address this problem, the proposed method ameliorates the formulation in the following subsection. 3.3 Optimization method So far, we assume that the attributes C are available. However, it may lead to a suboptimal performance because of leaving the learning model out of accounting this situation. One alternative way is to learn W , C simultaneously. So our model is extended as follows: min W , C W l T Φ C X − Y 2 , 1 + β Θ X 1 m , C , (15)where β is the parameter to measure the influence of the clustering process, and Θ ( { X } 1 m , C ) is the optimization function to solve the sub-optimization problem of clustering process. The rationale behind this model is that by minimizing the integrated objective function, we can find a set of perfect attributes C and the classification model W. The model W can predict the data correctly with a large margin and minimize the loss of mutual information caused by feature merging. In this paper, we exploit the positive instances to upgrade the attributes C as Θ X 1 p , C = ∑ i = 1 p min j = 1 , … , k H x i , c j , (16) where { X } l p represents the positive instances for the l − t h action class label, and H x , c = x − c 2 / p , (17)is the square of average Euclidean distance. Although the objective function of Equation (5) is not-convex for a given W or C only, it is convex with respect to both W and C. Therefore, it is difficult to get a global optimal solution directly. In this case, it is necessary to solve the objective function of Equation (5) by iterating over one variable, while fixing the other variable. It is not guaranteed that the global solution will be obtained, while the local optimal solution is satisfactory likewise. To simplify this process, we adopt an efficient block coordinate descend algorithm. Specifically, we first optimize the objective function with respect to W when attributes C are fixed, then optimize it with respect to C when W are fixed. These two procedures are repeated until convergence. Algorithm 1 summarizes the pseudo-code of block coordinate descend algorithm. ALGORITHM 1. Attribute-based mid-level representation method Input: Φ C ( X ) , Y , β , Output: W , C 1: Perform clustering process to the positive features of each action class label to initialize attributes C. 2: While not converged, do 3: While not converged, do 4: Fix C, update W ← E q u a t i o n ( 4 ) 5: End while 6: Fix W, update C ← E q u a t i o n ( 6 ) 7: End while When W is fixed, we need to solve a optimization which is non-convex and non-differentiable. Let g ( C ) denote the objective function of Equation (5) and C ( 0 ) denote the current solution. Here, we employ the sub-gradient method to find a refined solution C ¯ , that is, g ( C ¯ ) ≤ g ( C ) . Let ∇ C denote the subgradient of C, we update C ( u ) as, C u + 1 = C u − α ∇ g C u , ∀ u = 0 , 1 , … , v . (18)where α is the step size and v is the number of iterations. Because the subgradient method is not a descent method, it is common to keep tracking C until we find the best candidate, that is, the one with the smallest function value, C ¯ = arg min C u , u = 0 , … , v g C u . (19) It is evident that we will have g ( C ¯ ) ≤ g ( C ) . When g ( C ¯ ) = g ( C ) , the algorithm stops and outputs C ¯ is as the final result. 4 EXPERIMENTS Before learning the optimal process, an important step is feature extraction. In this paper, as mentioned above, we use the action bank features as the low-level features. After extracting low-level features, block coordinate descend algorithm is employed to train the model. To verify the effectiveness and robustness of the proposed method, we conduct experiments on three representative datasets, that is, KTH, UCF50 and HMDB51 using different experimental settings. For KTH, UCF50, and HMDB51 datasets, we employ two methods to predict the label of the test samples in order to better evaluate the algorithm. The first method is called ‘one-versus-rest vote’ method. As can be seen from Figure 3, the mid-level representation Φ C ( x ) is treated as the input of all classifiers, action label of the representation is chosen to hold the maximum output of all classifiers by using the ‘response evaluation function.’ FIGURE 3Open in figure viewerPowerPoint Diagram of the one-versus-rest vote evaluation method. Each action mid-level representation is treated as the input of all action class classifiers, then the outputs of the classifiers are used as inputs of the response evaluation function to find the maximum output, whose corresponding classifier action label is the mid-level representation's action label Another evaluation method is called ‘Precision/Recall’ method. Figure 4 shows the illustration of the ‘Precision/Recall’ method. In this method, for the response of every action classifier, we will calculate the recall ( R) and precision ( P), then the average precision ( A P ) is calculated as A P = ∫ 0 1 P R d R . (20) FIGURE 4Open in figure viewerPowerPoint Illustration of the ‘Precision/Recall’ evaluation method For the aforementioned evaluation methods, average classification accuracy is served as the final measurement of the classification performance in thi