Three‐dimensional human activity recognition by forming a movement polygon using posture skeletal data from depth sensor

ETRI JournalVolume 44, Issue 2 p. 286-299 ORIGINAL ARTICLEFree Access Three-dimensional human activity recognition by forming a movement polygon using posture skeletal data from depth sensor Dinesh Kumar Vishwakarma, Corresponding Author Dinesh Kumar Vishwakarma dvishwakarma@gmail.com orcid.org/0000-0002-1026-0047 Department of Information Technology, Delhi Technological University, Delhi, India Correspondence Dinesh Kumar Vishwakarma, Department of Information Technology, Delhi Technological University, Delhi, India. Email: dvishwakarma@gmail.comSearch for more papers by this authorKonark Jain, Konark Jain JP Morgan Chase, Mumbai, IndiaSearch for more papers by this author Dinesh Kumar Vishwakarma, Corresponding Author Dinesh Kumar Vishwakarma dvishwakarma@gmail.com orcid.org/0000-0002-1026-0047 Department of Information Technology, Delhi Technological University, Delhi, India Correspondence Dinesh Kumar Vishwakarma, Department of Information Technology, Delhi Technological University, Delhi, India. Email: dvishwakarma@gmail.comSearch for more papers by this authorKonark Jain, Konark Jain JP Morgan Chase, Mumbai, IndiaSearch for more papers by this author First published: 25 January 2022 https://doi.org/10.4218/etrij.2020-0101AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Human activity recognition in real time is a challenging task. Recently, a plethora of studies has been proposed using deep learning architectures. The implementation of these architectures requires the high computing power of the machine and a massive database. However, handcrafted features-based machine learning models need less computing power and very accurate where features are effectively extracted. In this study, we propose a handcrafted model based on three-dimensional sequential skeleton data. The human body skeleton movement over a frame is computed through joint positions in a frame. The joints of these skeletal frames are projected into two-dimensional space, forming a "movement polygon." These polygons are further transformed into a one-dimensional space by computing amplitudes at different angles from the centroid of polygons. The feature vector is formed by the sampling of these amplitudes at different angles. The performance of the algorithm is evaluated using a support vector machine on four public datasets: MSR Action3D, Berkeley MHAD, TST Fall Detection, and NTU-RGB+D, and the highest accuracies achieved on these datasets are 94.13%, 93.34%, 95.7%, and 86.8%, respectively. These accuracies are compared with similar state-of-the-art and show superior performance. 1 INTRODUCTION Human activity recognition (HAR) is a widely studied area in the field of computer vision. Its applications include video analysis, surveillance systems, robotics, and control of multimedia devices, assisting the sick or disabled, and different systems consisting of human–computer interfaces [1, 2]. Most multimedia contents are available in the form of videos and images. These images and videos may be analyzed in the aspect of the human being. The current multimedia device Microsoft Kinect sensor provides an excellent road map for human–machine interaction, and in every aspect, human activities are at the center. Currently, most studies on HAR have mainly been focused on learning and recognizing activities from videos captured by a single vision camera, that is, using RGB image videos [1, 3]. Although the use of two-dimensional (2D) cameras provides an easy and cheap way of collecting data, the cluttered background, occlusions, cast shadows, varying lighting conditions, and changes in viewpoint [1, 2] can vary the way activities are perceived. These drawbacks lead to poor recognition rates in HAR when 2D imagery is used. The use of range cameras can significantly alleviate these challenges. The ability of the field camera to capture the depth feature of videos is useful since perceiving depth leads to better recognition performance [4]. For example, in the frontal view, actions, such as eating and talking on the phone, may be the same in the 2D camera extracted skeletons; however, using range cameras like Microsoft Kinect, we can use the depth map sequence to classify these activities in a better way [5]. Currently, earlier range sensors were either too costly, provided subpar estimation, or were problematic to use on human subjects. The recent release of Microsoft Kinect [2] addresses these issues by providing both an RGB image and depth image streams. The Kinect was initially targeted for the entertainment market. It has now excited considerable interest within the vision and robotics community owing to its broad applications, such as human–computer interaction, security and surveillance, and multimedia data analysis [3, 6, 7]. Shotton and others [8] proposed a method to swiftly and precisely estimate three-dimensional (3D) position coordinates of skeletal joints from a single depth image captured by the Kinect. The method can provide an accurate estimation of 3D skeletal joint locations at 200 frames per second on the Xbox 360 GPU. This skeletal joint information is constructive in solving computer vision problems with human subjects. Using skeletal joint information, we can easily address the task of HAR compared with the use of RGB imagery. There are some useful real-world applications of HAR, such as healthcare, e-commerce, smart homes, and surveillance systems [9-15]. There are different types of sensors used to capture human activity, such as video cameras, wearable sensors, thermal, and wireless devices [16-22]. Most methods employed for HAR can be classified into three categories: silhouette-based, point-based, and skeleton-based HAR. Silhouette-based HAR involves using binary silhouettes (binary contours) to represent various human activities. Point-based or object-based HAR uses a particular object or point in the video and classifies the data using the object's trajectory in 2D or 3D Euclidean space [23]. Skeleton-based HAR uses the skeletal joints extracted from the video sequence. It uses their respective position and their change of position in each frame to formulate the classification among human activities. Although these methods are widely used, we use the skeleton-based method owing to its advantages over the other two methods. The silhouette-based method has the advantage of being exhaustive in extracting the region of interest (ROI) since it uses contours of the whole person moving. However, it cannot distinguish activities where the silhouettes do not differ by a considerable amount. For example, in activities, such as brushing teeth and talking on the phone, the silhouettes would be similar and produce low performance in HAR. The point-based method, although it considers every frame of the movement, still does not yield high accuracy in HAR if the activities have similar most-moving-joint trajectories. For example, activities, such as high arm wave and hammer, may have similar trajectories of the arm, resulting in low accuracy. The skeleton-based method corrects all these inadequacies by being exhaustive, considerate of every small movement by tracking the movement of the joints instead of the person itself. It also considers the relative motion of the joints with respect to each other. The skeleton-based method also allows us to visualize the activities better than the point-based method. This is because skeletons formed in each frame depict the activity in a more reasonable way than just the trajectory of a part of the person. In this study, we employed the skeleton-based method and developed an algorithm to extract from the 3D data points of joints of the skeleton to a 2D polygon depicting the plotted joints' movements in three frames: the first, the middle, and the last. The 2D polygon is converted to a one-dimensional (1D) vector consisting of 360 features, each being the distance from the centroid of the polygon of the points on the polygon inclined at the angles equal their respective indices. Next, we obtained the most-moving joint in each example and stored the movement of the joint using the angle with respect to the horizontal in each of the 20 frames selected as a 1D vector. Then, 381 features are passed through principal component analysis (PCA) and the examples are classified against each other using the linear support vector machine (SVM). Our main contribution is threefold. First, we present a unique algorithm using both skeleton-based and point-based methods for data classification. Second, we propose the reduction of the 3D data to 1D, which proves to be less computationally expensive. Third, our method can effectively differentiate activities with minor differences, and the whole system runs at real time. We used the following datasets: MSR-Action3D [24], Berkeley MHAD [25], TST Fall detection dataset v2 [26], and NTU-RGB+D [27]. 2 RELATED WORK In the past few decades, a broad spectrum full of inspirational ideas has been proposed in the area of HAR. Various researchers have used the skeleton-based method of classification and recognition of activities in HAR. This section presents a review of the study conducted in the skeleton-based method of HAR. Chen and others [28] operated on the so-called depth motion maps (DMMs) from three projection views (front, side, and top). They captured motion signals and used local binary patterns (LBPs) to gain their feature representation, which is also employed for object recognition for cloud video [29]. However, their method is computationally expensive and consumes much time for the algorithm to learn the classifiers. Rahmani and others [29] combined the discriminative information from depth images and 3D joint positions to classify the activities. Although they could achieve high speeds in implementing their algorithm, the approach lacked in the accuracy of the classification conducted. Chen and others [30] developed a 3.5D demonstration of a depth video for action. The 3.5D graph of the depth video consists of a set of nodes corresponding to the joints of the human body. Zhuang and others [31], similar to this study, extracted features from the coordinates of human body joints and classified them using machine learning. They combined a Markov blanket with the wrapper method for sequential data feature selection and proposed an algorithm using four sets of human motion data and two machine learning methods (Hidden Markov Model (HMM) and Dynamic Bayesian Network (DBN)). However, they used just a part of the data to classify their sequential data, which may have led to a loss of information. Chen and others [32] engaged skeleton joint locations in 3D spatial bins to build histograms of 3D joint locations (HOJ3D) as features for action recognition. A skeleton joint-based method for recognizing person-to-person interaction using the graph model has been presented [33]. It preserved the complex spatial evidence over joints. An evolutionary algorithm for selecting the best subset of skeleton joints based on the topological structure of a skeleton has been presented, which lead to a fair increase in recognition rates [32]. A modeled human action through a spatiotemporal hierarchy of 3D skeletal features inspired by human anatomy [33] has been presented. They employed linear dynamic systems for learning the dynamics of the above-stated features. Ofli and others [25] formed and used the Berkeley MHAD dataset in the same study to develop recognition accuracies through Mocap (Motion Capture) data using methods, such as 1-NN, 3-NN, and K-SVM. Xu et al. [34] investigated fall prediction based on human biomechanics equilibrium and body postural characteristics by analyzing 3D skeletal joint data from a depth camera sensor Kinect. Yang and Tian [35] proposed a 3D joint position difference in consecutive frames and temporal differences in all frames for representing action. They could capture all information on the activity using only 3D joint locations. Wang and others [36] also used a similar approach to the previous one; however, they introduced the depth histogram-based feature. Low-frequency Fourier components were used as features in the temporal dimension. A human body joint using depth features [37] was used for detecting human activities very effectively. Similarly, a depth silhouette-based method is also effective [38]. Indoor and outdoor human activities, such as object localization, human gait recognition, spatiotemporal contents detection, motion recognition, and text recognition, have been conducted [39-46]. 3 PROPOSED MODEL The proposed method considers the approach of reducing the dimensionality of a 3D video (RGB images + Depth data) to a 2D polygon representing the motion of the joints in the activity. The 2D polygon is converted to a 1D vector of the distance of the polygon's points from its centroid sorted with respect to the angle subtended by the point considered from the centroid with respect to the x-axis. We also use another set of features: the most moved joint's angle of movement in 20 consecutive frames of the activity. Figure 1 shows the flow diagram of the proposed algorithm. FIGURE 1Open in figure viewerPowerPoint Schematic block diagram depicting the algorithm proposed through example of activities such as draw tick and tennis serve Figure 2 shows the dimension reduction technique used for the two activities: golf swing and pick up and throw. Figure 2A,D show the 3D data from the video captured with depth data of the activity golf swing and activity pick up and throw, respectively. The color contour represents the depth data. As shown in Figure 2B,E, the red polygon represents the 2D boundary plot generated for each activity. As shown in Figure 2C,F, the plot's ordinate of the points corresponding to the integer angles in degrees (−180 to +180) generates the 361 element vector. Thus, a 3D video was represented as a 1D vector for the SVM for training and testing. The dimensional reduction enabled the computer to visualize the videos captured by MS-Kinect into a more elegant form of a total of 382 numbers in a matrix. We used the data arranged in the form of a n × 3 matrix. Here, n is the total number of special coordinates given for the sample, which is equal to the number of frames multiplied by the number of joints, with each column representing the x, y, and z coordinates of the skeleton's joints, respectively. The number of joints j identified varied in each dataset: 20 in MSRAction3D, 25 in TST Fall Detection v2, and 43 in Berkeley MHAD. We separated each coordinate per frame. Thus, converting the n × 3 matrix to an n j × j × 3 matrix. Now, we select the first, middle, and last frames empirically to represent the movement in its initial, maximum moved, and final stage, respectively. Then, we plot them together and determine the boundary of the scatter plot using the 'boundary' function provided in the MATLAB tool [47]. It generates a closed boundary around the points, which has a default shrink factor of 0.5, and its precise value is not crucial. The algorithm used for determining the boundary of the set of points can be found in Algorithm 1. Notice in Figure 3 the similarity in the polygons generated for activity "sidekick" in Figure 3A,B and for activity "high arm wave" in Figure 3C,D. Additionally, note the dissimilarities in Figure 3B,C, which are representative of the two activities performed by the same test subject. The boundary's vertices were extracted, and the coordinates of the centroid of the given polygon, G = G x G y G z were calculated using 1. The distance of the points on the polygon from the centroid was taken as the representation of the polygon in a 1D form using the distance formula in (2). Thus, the distance was sorted with respect to the angle of inclination of the given point from the origin, generating a distance from the origin versus angle of the point with respect to the centroid and the x- and y-axes, θ (calculated using (3) and (4)), plot. D and Θ are the 1D matrices containing all the distances and corresponding angles, respectively. FIGURE 2Open in figure viewerPowerPoint Examples of dimensionality reduction in two activities: golf swing and pick up and throw FIGURE 3Open in figure viewerPowerPoint Some samples of the polygon generated from the MSRAction3D dataset: (A) activity number 15: sidekick performed by test subject number 10. (B) Activity number 15: sidekick performed by test subject number 7. (C) Activity number 1: high arm wave performed by test subject number 7; and (D) activity number 1: high arm wave performed by test subject number 9 G = mean X mean Y mean Z . (1) d = x − G x 2 + y − G y 2 + z − G z 2 . (2) θ = tan − 1 y − G y x − G x . (3) D = d , Θ = θ . (4) As shown in Figure 4, note the dissimilarities between the two activities 1D plot, like the number of peaks and trenches. We passed the distance vector into a linear SVM by taking the feature vector as 361 points from the plot corresponding to each integer angle from −180° to +180°. The SVM produced promising results. However, the machine still had problems with activities, such as draw x and draw a circle, whose movements could not be adequately represented using the three-frame techniques. Figure 5A,C shows polygon boundaries of two activities: activity 7 (draw x) and activity 9 (draw a circle) in the MSRAction3D dataset. These figures are strikingly similar to each other. Therefore, it will produce similar 1D feature vectors, which may lead to worse experimental results. As shown in Figure 5B,D, we can note the difference in the two activities using the trajectory of the most moved joint method. The SVM will now have a set of 20 new features to effectively discriminate between the two classes. Therefore, we follow the approach of determining the most moving joint in the body and determine its trajectory in 20 consecutive frames of the sample. We represented the positions of the joint in these 20 frames by storing the inclination of the points to the x-axis with respect to the origin in a row vector of 20 entries. FIGURE 4Open in figure viewerPowerPoint Samples of the 1D distance vector versus the corresponding inclination angle plot. The plots correspond to Figure 3: (A) activity number 15: sidekick performed by test subject number 10. (B) Activity number 15: sidekick performed by test subject number 7. (C) Activity number 1: high arm wave performed by test subject number 7; and (D) activity number 1: high arm wave performed by test subject number 9 FIGURE 5Open in figure viewerPowerPoint Polygon and feature vector of two activities: (A), (B) draw x activity and (C), (D) draw circle activity The most moving joint is determined by computing the covariance matrix ((5), (6), and (7)) of the matrix containing the x, y, and z coordinates of each joint in all frames of the sample as its columns. A single number computed the trace of each matrix to represent the covariance of the joint in the sample activity. The joint with the maximum trace is considered the most moved joint because its trajectory is the most scattered, as depicted by the trace of the covariance of its x, y, and z coordinates. The sample covariance of N observations of K variables is the K − by − K matrix: Q = q ij . cov X , Y = 1 n 2 ∑ ∑ 1 2 x i − x j y i − y j . (5) σ X = cov X , X . (6) q ij = 1 N − 1 ∑ ( X ij − mean X j ( X ik − mean X k . (7)Then, we concatenated the original 361 entries long vector with the index of the most moved joint and the 20-entry vector formed in the above step. The produced feature vector now had 361 + 1 + 20 = 382 entries. Figure 6 shows the similarity of the vector graphs of the same activity and the difference between the graphs of the two activities. Figure 6A,B has one peak, whereas Figure 6C,D has plateau-type structure. FIGURE 6Open in figure viewerPowerPoint Samples of the 20-feature vector generated for the most moved joint in activities (A) sidekick performed by test subject number 10. (B) Sidekick performed by test subject number 7. (C) High arm wave performed by test subject number 7 and (D) high arm wave performed by test subject number 9 We then employed PCA on the matrix formed by the number of samples used per action and their respective 382 features. We passed the eigenvectors generated from PCA into the SVM and obtained promising results. Algorithm 2 summarizes the proposed method. 4 EXPERIMENT AND RESULTS To test the performance of the proposed algorithm, we conduct an experiment using four publicly available datasets: MSR-Action3D [24], Berkeley MHAD [25], TST Fall detection dataset v2 [26], and NTU-RGB+D [27]. The recognition accuracy of each action and activity is computed using the following equation. Accuracy = TP + TN TP + TN + FN + FP × 100 ( In Percentage ) . (8)Here, TN, TP, FN, and FP are True Negative, True Positive, False Negative, and False Positive, respectively. For each dataset, a confusion matrix is predicted through SVM, and the overall accuracy on the dataset is computed. To verify the effectiveness of the proposed algorithm, we compared the overall recognition accuracy on each dataset with the state-of-the-art on similar datasets. 4.1 MSR action 3D dataset Li et al. [24] introduced the MSR Action 3D Dataset in June 2010. It consists of 20 activities performed by 10 people, each performing three repetitions of the activity. Thus, about 30 samples of each activity are obtained. The resolution is 320 × 240, and the video is shot in 15 frames per second mode. The dataset consists of 567 action samples. These actions are selected to know the various activities that are generally undertaken while using gaming consoles and cover different movements related to the torso, legs, and arms. Figure 7 shows the description of these activities, and Table 1 presents the name of these activities. FIGURE 7Open in figure viewerPowerPoint Twenty activities in the dataset: Row 1 left to right: high arm wave, horizontal arm wave, hammer, hand catch, forward punch; Row 2 left to right: high throw, draw x, draw tick, draw circle, hand clap; Row 3 left to right: two-hand wave, side boxing, bend, forward kick, side kick; Row 4 left to right: jogging, tennis swing, golf swing, pickup and throw TABLE 1. Activities of MSR action 3D dataset Label Name of activity Label Name of activity 1 High arm wave 11 Two-hand wave 2 Horizontal arm wave 12 Side boxing 3 Hammer 13 bend 4 Hand catch 14 Forward kick 5 Forward punch 15 Side kick 6 High throw 16 jogging 7 Draw x 17 Tennis swing 8 Draw tick 18 Tennis serve 9 Draw circle 19 Golf swing 10 Hand clap 20 Pick up and throw Classification result: In the experiments, we used a linear SVM for classifying data in the 20 classes. The validation technique used was fivefold validation. Figure 8 shows the details of the classification result from the confusion matrix. FIGURE 8Open in figure viewerPowerPoint Classification result of MSR 3D action dataset The overall recognition accuracy achieved 94.13%, which is shown in the Table 2 with bold text. To verify the effectiveness of the proposed algorithm, we compared this efficiency with that of similar state-of-the-art algorithms. It is presented in Table 2. TABLE 2. Comparison of accuracy with similar state-of-the-art on MSR action 3D dataset Method Validation technique Accuracy Li et al. [24] 1/3 holdout validation 86.50% Chen et al. [28] 50% holdout 87.90% Rahmani et al. [29] 50% holdout 82.70% Cai et al. [48] 50% holdout 91.01% Xu et al. [49] 1/3 holdout validation 82.87% Zhuang et al. [31] 10-fold validation 83.37% Jalal et al. [50] LOSO 92.40% Ours Fivefold validation 94.13% Bold means accuracy achieved by the proposed model. Our algorithm misclassifies activity 2 (horizontal arm wave) and activity 3 (Hammer) as activity 5 (Forward Punch) because the actions have similar movements; thus, generating similar polygons and leading to confusion in the SVM. However, the algorithm performs almost perfectly in classifying activity 14 (Forward Kick): 99.63% and activity 15 (Side Kick): 99.79% because of their categorically unique polygon representations. It is worth noting that the most moved joint criteria must have played a vital role in classifying these two activities almost correctly because these activities use the joints, such as the knee and foot, more than the upper body joints. The proposed algorithm outperforms all other state-of-the-art algorithms by making a better representation of certain complex activities in the dataset, such as activities 7, 8, and 9 (draw x, draw a circle, and draw tick), and by considering the most moved joint and its movement in many frames. It means that even if the person's stance and skeletons may be the same in the activities, the proposed algorithm has success in classifying them accurately. The videos are shot in one view mode only. However, multiple view data may provide better recognition accuracy because a different view may capture a different aspect of the activity performed. For example, activity number 5, forward punch, could have been better represented if the side view of the activity was also available. Another possible problem that the dataset may have is that it is shot in a closed environment and activity, such as golf swing and tennis serve, would generally be performed outdoors; thus, the dataset may have different background disturbances. The algorithm may be unable to work as well as it has worked on indoor conditions. 4.2 Berkley MHAD dataset Ofli and others [25] introduced the Berkeley MHAD dataset in 2013. The dataset consists of 11 activities performed by 7 males and 5 female subjects having the age range of 23 to 30 years except for one elderly subject. All the subjects performed 5 repetitions of each action, giving a total of about 660 action sequences, which tally to about 82 min of total recording time. The dataset contains a staggering number of 43 joints, which is more than double of the joints considered by the MSRAction3D dataset. This proves to be somewhat of a beneficial drawback. Beneficial because more number of joints produces a more precise skeletal representation. The drawback also comes from the increased number of joints since the algorithm takes more storage to run, and the time complexity of the algorithm, in general, is O(n). Therefore, the algorithm takes almost twice as much time as when it classifies MSRAction3D samples. Figure 9 and Table 3 show the actions and labels, respectively. FIGURE 9Open in figure viewerPowerPoint Eleven activities as performed in the dataset from left to right: jumping, jumping jacks, bending, punching, waving two hands, waving one hand, clapping, throwing, sit down/stand up, sit down, and stand up TABLE 3. Berkeley MHAD action dataset Label Name of activity Label Name of activity 1 Jumping 7 Clapping 2 Jumping jacks 8 Throwing 3 Bending 9 Sit down 4 Punching 10 Stand up 5 Waving two hands 11 Sit down and stand up 6 Waving one hand As shown from the confusion matrix Figure 10, the algorithm has difficulties classifying activity 9 (sit down/sit up) as it misclassified it as activity 10 (sit down) and activity 3 (bending) a few times. The reason for this might be the complexity of activity 9, which cannot be adequately represented by the three frames technique. Additionally, activities 10 and 11 are giving an erroneous result with each other because activities (sit down and stand up) are the same activities performed in the reverse order. This is a possible drawback in our algorithm since it does not consider the time dimension of the activity. Table 4 presents the comparison results with other state-of-the-art on this dataset. The comparison parameters are the highest recognition accuracy achieved through various methods [25, 32, 51, 52] and validation techniques, such as one-vs-all (OVA), leave-one-out-cross validation (LOOCV), and leave-one-subject-out (LOSO). To verify the effectiveness of the proposed algorithms, we compared the recognition accuracy of 93.34% with many existing algorithms. FIGURE 10Open in figure viewerPowerPoint Classification result of Berkeley MHAD action dataset TABLE 4. Comparison of accuracy with similar state-of-the-art on Berkley MHAD dataset Method Validation technique Accuracy Ofli et al. [25] OVA 79.93% Foggia et al. [51] LOOCV 82.00% Guo et al. [52] LOOCV 86.83% Chen et al. [32] LOSO 92.41 Ours Fivefold validation 93.34% Abbreviations: LOOCV, leave-one-out-cross validation; LOSO, leave-one-subject-out; OVA, one-vs-all. As shown in Table 4, the proposed method's result surpasses the results of other state-of-the-art methods. This dataset had a plethora of 43 joints for each frame, which is beneficial for our algorithm since it can generate more accurate and constricted boundary polygons for the given activity, representing the activity more adequately. It can also be seen that the algorithm performs remarkably well for activities 5 and