Human activity recognition with analysis of angles between skeletal joints using a RGB‐depth sensor

ETRI JournalVolume 42, Issue 1 p. 78-89 ORIGINAL ARTICLEFree Access Human activity recognition with analysis of angles between skeletal joints using a RGB-depth sensor Ömer Faruk İnce, Ömer Faruk İnce Center for Intelligent and Interactive Robotics, Korea Institute of Science and Technology, Seoul, Rep. of KoreaSearch for more papers by this authorIbrahim Furkan Ince, Ibrahim Furkan Ince Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of KoreaSearch for more papers by this authorMustafa Eren Yıldırım, Mustafa Eren Yıldırım orcid.org/0000-0002-0662-2770 Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of Korea Department of Electrical and Electronics Engineering, Bahçeşehir University, Istanbul, TurkeySearch for more papers by this authorJang Sik Park, Corresponding Author Jang Sik Park jsipark@ks.ac.kr orcid.org/0000-0003-1794-7631 Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of Korea Correspondence Jang Sik Park, Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of Korea. Email: jsipark@ks.ac.krSearch for more papers by this authorJong Kwan Song, Jong Kwan Song Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of KoreaSearch for more papers by this authorByung Woo Yoon, Byung Woo Yoon Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of KoreaSearch for more papers by this author Ömer Faruk İnce, Ömer Faruk İnce Center for Intelligent and Interactive Robotics, Korea Institute of Science and Technology, Seoul, Rep. of KoreaSearch for more papers by this authorIbrahim Furkan Ince, Ibrahim Furkan Ince Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of KoreaSearch for more papers by this authorMustafa Eren Yıldırım, Mustafa Eren Yıldırım orcid.org/0000-0002-0662-2770 Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of Korea Department of Electrical and Electronics Engineering, Bahçeşehir University, Istanbul, TurkeySearch for more papers by this authorJang Sik Park, Corresponding Author Jang Sik Park jsipark@ks.ac.kr orcid.org/0000-0003-1794-7631 Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of Korea Correspondence Jang Sik Park, Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of Korea. Email: jsipark@ks.ac.krSearch for more papers by this authorJong Kwan Song, Jong Kwan Song Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of KoreaSearch for more papers by this authorByung Woo Yoon, Byung Woo Yoon Department of Electronics Engineering, Kyungsung University, Pusan, Rep. of KoreaSearch for more papers by this author First published: 13 November 2019 https://doi.org/10.4218/etrij.2018-0577Citations: 22AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Human activity recognition (HAR) has become effective as a computer vision tool for video surveillance systems. In this paper, a novel biometric system that can detect human activities in 3D space is proposed. In order to implement HAR, joint angles obtained using an RGB-depth sensor are used as features. Because HAR is operated in the time domain, angle information is stored using the sliding kernel method. Haar-wavelet transform (HWT) is applied to preserve the information of the features before reducing the data dimension. Dimension reduction using an averaging algorithm is also applied to decrease the computational cost, which provides faster performance while maintaining high accuracy. Before the classification, a proposed thresholding method with inverse HWT is conducted to extract the final feature set. Finally, the K-nearest neighbor (k-NN) algorithm is used to recognize the activity with respect to the given data. The method compares favorably with the results using other machine learning algorithms. 1 INTRODUCTION Monitoring and recognition of human activity patterns collected by motion sensors is currently a popular research topic. Human activity recognition (HAR) has been used in several different domains such as robotics 1-3, computer engineering 4, 5, healthcare 6, natural sciences 7, and industrial applications 8, 9. Understanding of human activity involves activity recognition and activity pattern discovery. HAR aims to provide highly accurate detection of human activities by adopting a predefined activity model. To do so, a high level conceptual model must first be built and run by structuring an appropriately pervasive system. In contrast, discovery of activity patterns is more closely related to identifying unknown patterns directly from low-level sensor data without applying any predefined models or assumptions. Previous research has proven that machine learning methodologies work efficiently to classify different activities from sensor data 10-12. Some types of sensors used in HAR systems are digital cameras, depth sensors, wearable sensors, and gyro sensors 13-19. Sensor-based systems require two main steps. The first step is the calculation of relevant features based on a sensor's acquired data. In the second step, a chosen classification algorithm defines the activity in accordance with the features obtained in the first step. Common features contain statistics extracted from time-domain signal analysis, frequency-domain analysis, and wavelet analysis, which is also known as time-frequency analysis. Vision and other signal-based sensors 20-24 are used to merge technologies with advanced worldwide practical applications and play major roles in science and industry. These technologies provide comfort in daily activities and enhance the quality of life. This paper proposes a biometric system that uses angles between skeletal joints to recognize human activities in 3D space based on RGB-depth sensor data, which could be useful for elderly care and video surveillance systems. Simply stated, the system obtains particle joint angle pairs and stores them using the sliding kernel method. After creating a feature set, Haar coefficients of the features are obtained, and the number of features is then reduced by applying an averaging dimension reduction technique. A proposed thresholding method using the inverse Haar wavelet transform (HWT) is applied to enhance the signals for better classification. HWT is commonly applied to filter out data noise, reduce data size, and detect singularities. Thus, HWT is highly effective in time series data processing. Finally, the k-nearest neighbor (k-NN) algorithm is applied to classify the real-time data originating from the sensor. This paper is organized as follows; Section 1 provides a brief introduction of the proposed study. In Section 2, HAR applications regarding previous approaches are reviewed. In Section 3, the proposed algorithm for human activity recognition is discussed in detail. In Section 4, the experimental environment is presented. In the final section, the advantages and limitations of the proposed study and further improvements for better performance are discussed. 2 PREVIOUS APPROACHES USING HAR Human activity recognition has been intensively studied over the past several years. In 25, re-identification of persons in two different mediums was accomplished using SIFT and Bag-of-Features. The use of skin joint features obtained from RGB-depth sensors using HAR to track humans was proposed in 26. The authors in 27, 28 used local and hybrid features for facial analysis. In 29, motion detection was achieved in real-time using multiple cameras. The majority of HAR studies can be categorized under two types: vision-based and sensor-based methods. A vision-based study is presented in 30 in which the authors estimate the simultaneous pose and shape of articulated objects by using a single depth camera. First, 3D transformations of each skeletal joint are illustrated using a twist map and exponential maps, and the articulated deformation model constructed from the maps is then combined with a probabilistic model to carry out pose tracking. In 31, the authors suggested using the representation of intermediate body parts to map the estimation of poses into a per-pixel classification problem and then restructure the resulting body parts to construct confidence-scored 3D proposals of multiple body joints. Human pose estimation based on a single depth camera was proposed in 32, and relies on the correlations among articulated and generalized Gaussian kernels. The approach consists of embedding the kinematic skeleton into the Gaussian kernels and constructing tree-structured templates from several multivariate Gaussian kernels with quaternion-based rotation. In 33 a new skeleton-based method is proposed to describe the spatio-temporal aspects of an activity data sequence via the Minkowski and cosine distances between 3D skeletal joints. In 34, multifeatures along with a hidden Markov model (HMM) are used with a single camera for a healthcare application. Spatial-temporal features for HAR were evaluated in 35, 36. In 37, graph formulation is employed for abnormal activity recognition. Although there are some RGB-based studies in the literature, the applications suffer in environments that are totally dark or where illumination changes are present, despite the use of a multi-camera system consisting of eight cameras installed to view a room from every possible angle and to overcome an issue with subject occlusion, for example 33. However, unlike RGB-based methods, depth-based methods are invariant to illumination changes. In sensor-based studies, authors have proposed the use of multiple accelerometers, wearable sensors, and other types of sensors. In 38, the authors invented a system which is based on systematic performance analysis of motion-sensor-captured behavior for human activity recognition via smart phones. Sensory data sequences using smart phones were collected while participants in the experiment performed typical and daily activities. An activity unit was then characterized by time, frequency, and wavelet domain features. By means of various classification algorithms, both personalized and generalized models were created and performed activity recognition. Sikder and others proposed a new distance metric called the log-sum distance to calculate the difference between two sequences of positive numbers 39. Basically, the log-sum distance measures the motion data gathered from daily activities. Wearable sensors were used for human behavior analysis in 40. Another study used pyroelectric sensors for abnormal activity detection 41. In contrast, internet of things (IoT)-based HAR systems have also been proposed. Subasi and others proposed an intelligent m-healthcare system using IoT technology 42. The motivation was to provide pervasive human activity recognition using data mining algorithms. Additionally, deep learning algorithms have been recently applied to vision-based HAR systems by Wang, Simonyan and Zisserman, and Karpathy and others in 2014 43-45. The authors turn the task of action recognition into one of image classification to handle over-fitting problems caused by the small number of annotated training examples. Because some actions are highly associated with certain objects and static poses, the static appearance by itself could be a useful solution. Recently, long-short term memory (LSTM) networks have been used in different approaches. Inspired by the success of the weightlessness feature, Tao and others developed two-directional features for bidirectional long short-term memory (BLSTM) to augment learning in HAR systems. For higher accuracy, a new classifier named the multi-column BLSTM (MBLSTM) that combines various acceleration signal features for activity recognition is presented 46. 3 PROPOSED HUMAN ACTIVITY RECOGNITION MODEL 3.1 Problem definition In this study, the main motivation is to develop a HAR system which has low cost and high efficiency. To do so, RGB-depth based system (a camera-based HAR system) using skeletal angle information is proposed. The proposed approach obtains eight different angles of skeletal joints. To obtain the angle information, skeleton detection and tracking is first performed using Microsoft's Kinect SDK 2.0. Because each joint is defined in the SDK, 3D angles can be calculated. Joint pairs are selected with respect to the relevance of basic human activities as defined in related studies. Previous approaches 47, 48 show that there is a direct relevance between angle motion sequences and human activities. In particular, the main motivation for using angles is that angles are scale and rotation invariant. The flowchart of the proposed method is shown in Figure 1. Figure 1Open in figure viewerPowerPoint Flowchart of the proposed method 3.2 Flow analysis of the proposed algorithm 3.2.1 Angles between joints A Kinect v2 sensor can detect and track 25 skeletal joints. After the skeleton is detected and tracked, the key part of the method lies in selecting features to create feature vectors. Ofli and others have shown that during standing actions and walking, the right and left knees and the elbow are the most informative features of the human skeleton 49. In addition, Uddin and others have shown that angles between shoulders, elbows, knees, and the crotch provide useful informative features for 3D human activity recognition 47. Moreover, Thang and others have used angle pairs of shoulders, elbows, and knees. In the proposed approach, angle pairs of hipbones are also added. The selected angle pairs used in this study are shown in Figure 2. Figure 2Open in figure viewerPowerPoint Human skeleton and selected angle pairs The value of the angle between two different joints and can be calculated by distinguishing the locations of the two joints with respect to a reference joint r in 3D space. The formula is denoted as follows: (1) In this equation, represents the distance between joint and the reference joint r in 3D space. Similarly, represents the distance between joint and the reference joint r in 3D space. The dot between the vectors indicates the dot product. Lastly, and represent the lengths of these vectors, respectively. 3.2.2 Sliding Kernel Because human activity occurs in the time domain, it can be considered as a two dimensional problem. It is thus necessary to observe the angle patterns of each joint angle in the time domain to recognize an action. Therefore, storing angle swings in the time domain using the sliding kernel method is beneficial. Angle data of each joint in the time domain can be denoted as follows: (2) (3) (4)where is the angle value of a joint in the frame. The number of features obtained in each frame is equal to , where n represents the kernel size. In this paper, the sliding kernel is achieved using a queue structure. In each frame, eight joint angles are obtained by the Kinect camera and they are stored in the queue structure. Here the capacity of the queue structure is another parameter of the system, which denotes the kernel size. If the queue capacity is 16, then 16 8 = 128 features are stored in the queue. Because the queue structure is a first-in first-out (FIFO) data structure, it behaves as a sliding kernel in the time domain. In our study, the kernel size (in other words the queue capacity) is set to 16, which yields optimum results with respect to time complexity and performance. 3.2.3 Discrete Haar Wavelet Transform (DWT) Haar wavelets are square-shaped functions developed to organize data using frequency. Data are transformed from the spatial domain to the frequency domain, and each component on the corresponding resolution scale is stored by wavelet transformation. The basic Haar function , with scaling function , is defined using the following relationships: (5) (6) (7) (8) The functional relevance between the wavelet function and scaling function can be described as follows: (9)and (10) Haar wavelet transform basically consists of averaging and differencing. For an input dataset with elements, the HWT simply takes the average of each pair of components of the dataset and places them in the first half of a new string (low-pass band). The average differences between each pair of components compose the other half of the new string (high-pass band). This process repeats itself until there are detail coefficients and one final sum consisting of low-pass values. Assume that x is a vector with length N, and , where N is a power of 2. The number of steps required is: (11) The calculation of low-pass band coefficients and high-pass band coefficients proceeds as follows: (12)and (13) This procedure is also reconstructive because: (14)and (15) In this paper, the number of elements for each input signal is 128, which means that the step number is After determining the Haar coefficients of the feature vectors, averaging-based dimension reduction is applied. 3.2.4 Dimension reduction with averaging After 128 Haar coefficients are produced, the averaging-based dimension reduction method is conducted to decrease the computational cost and eliminate redundancy. The average of two likely HWT candidates in the low-pass module or in the high-pass module will correspond to new elements in the new set of vectors. For instance, the average of the first and second elements will correspond to the first index of the new vector, the average of the third and fourth elements will correspond to the second index of the new vector, and so on. The equation for reducing the dimensions is denoted as follows: (16)and (17)where f is the feature vector for each reduction. The first subscript of x represents the number of dimension steps, and the second subscript is the index number of a feature vector. Additionally, represents the step number. In this paper, we use averaging for dimension reduction in the frequency domain. After we obtain the Haar coefficients, we apply the averaging method to reduce the size of the frequency graph of the Haar coefficients. After dimension reduction in the frequency domain, we apply the inverse transform to obtain feature coefficients in the time domain. This is necessary to reduce the dimensions of the feature vectors with minimal loss of information. 3.2.5 Proposed thresholding with inverse HWT As mentioned in the Haar Wavelet Transform section, the original signal can be easily reconstructed by applying the reverse averaging and differencing operation without losing information. However, it is possible to obtain a higher degree of compression by setting a non-negative threshold value . This method is called lossy compression. Using a threshold will make any low-pass band elements correspond to zero if their magnitude in the transformed signal is less than or equal to the threshold value. By this means, the number of zeros in the transformed signal will increase, which will provide a high level of compression. For lossless compression, the threshold value is set to zero. If lossy compression is required or is advantageous, then approximations of the original signal are constructed. Setting the value of requires care, as there is a tradeoff between the threshold value and compressed signal quality. The conventional threshold equation is (18)and (19)where x represents any Haar coefficient located in the high-pass band. The proposed thresholding method provides a useful technique to enhance the signal quality while performing inverse HWT. The formulation is (20) (21) (22)where is the mean of the Haar coefficients in both the low-pass and high-pass bands. 3.2.6 Transformation of feature vectors First, feature vector values contain the information of eight different joint angles. These values are stored (for 16 frames) until 128 features are obtained. To retain the important information stored, data are evaluated in the frequency domain. For time series evaluation, HWT is widely used in signal processing, because HWT compresses the data without losing important information (red signals in Figure 3). In the frequency domain, Haar coefficients are averaged for dimension reduction (green signals in Figure 3). Thus, the important information is kept and redundancy will be eliminated. In other words, the power spectrum of each signal will be reduced, and this provides a low dynamic range evaluation of Haar coefficients. After dimension reduction, a novel thresholding method is proposed. In this method, the Haar coefficients below the mean of all Haar coefficients are replaced with the mean of all Haar coefficients, but the sign of each replaced coefficient is kept. Through the inverse HWT, the feature extraction process is performed (purple signals in Figure 3). In summary, a new feature vector contains the characteristics of the raw data, but at a different scale and shape. In Figure 3, purple signals represent the results of the thresholding methods combined with inverse HWT. Even though both methods eliminate redundancy and provide low dynamic range evaluation, it is clear that the proposed method creates more peaks and valleys in the data, which indicate more efficient classification performance. Figure 3Open in figure viewerPowerPoint Thresholding comparison: (A) conventional method and (B) proposed method 4 EXPERIMENTAL RESULTS AND CONSIDERATIONS 4.1 Experimental environment and dataset The dataset for this study was created by Kyungsung University, Department of Electronic Engineering. For implementation of the proposed HAR system, Microsoft Visual Studio 2013, C# with Microsoft Kinect SDK 2.0, Vitruvius, and the Accord.net machine learning framework are used. For comparisons with different parameters of k-NN and other algorithms such as Random Forest (RF) and genetic algorithm (GA), Weka data mining software is used. Lastly, Python 3.6.4 with Tensorflow is also used for the LSTM network performance evaluation. This dataset contains information regarding 10 different people whose height, weight, and clothing are different. Each activity is recorded for each person (user) two times to construct the training set and one time to construct the test set. The reason is that test sequences include different angle variations of the same activities, which are used to evaluate the accuracy of the proposed system in real-world conditions. Moreover, for k-fold cross validation, training and test sequences were combined, and k was set to 10. Because each person performs an activity at a different pace, the number of instances for each activity is different. The number of instances for each activity used in the dataset is shown in Table 1. Table 1. Number of instances used in the dataset for each activity Activity Train Test Number of instances Number of instances Standing 2018 848 Walking 2060 955 Eating 1728 926 Calling 1692 873 Lifting 2264 832 Handclapping 2224 859 Sitting 2183 969 Total 14 169 6262 For the preparation of the training and test sets, the Microsoft Kinect v2 sensor is placed at a height of 1.70 m. During the creation of the training set, a user is recorded in two directions, which are ±45° with respect to the camera. However, the users are recorded for the entire area between ±45° and −45° while preparing the test set. The key motivation behind the use of test activity patterns not used for training is that the proposed method provides rotation invariance. The dataset setup is presented in Figure 4. Figure 4Open in figure viewerPowerPoint Dataset setup for training and testing Finally, the dataset is composed of two activity primitives, which are posture and motion. The list of primitives is given in Table 2 and images of each activity in the dataset are shown in Figure 5. Table 2. Activity primitives and their members Posture Motion Standing Walking Calling Eating Sitting Lifting — Handclapping Figure 5Open in figure viewerPowerPoint HAR-activity examples from dataset 4.2 Comparison of different k-NN parameters As mentioned previously, the k-NN algorithm is used as the classifier in this paper. The main reason for choosing k-NN as the classifier is that k-NN is a lazy learner. This means that the k-NN algorithm does not model the data while training. However, it remembers the data during the test phase. Another reason is that k-NN is quite effective in a large number of applications; however, as the number of dimensions increases, k-NN performance deteriorates. Because there are various parameters associated with k-NN, a short explanation of which parameters are used for this study follows. First, because k-NN looks for the nearest neighbor in a cluster by calculating the distance between attributes, one of the main parameters is distance. There are various types of distance metrics, such as the Euclidean distance, Manhattan distance, or Minkowski distance. Each of these metrics has advantages and disadvantages, and selecting which to use largely depends on the dataset. In this study, the Euclidean distance was selected based on the analysis results discussed below. After choosing the distance metric, assigning the value of k (the number of closest training examples to be considered as similar to a test sample) is the next task, and assigning the value of k also depends on the dataset. There are some ways to find the optimum k value, such as cross-validation and resampling. Assigning k is a critical task. A small k value will allow noise to have a higher influence on the data. In contrast, a large k value increases computational complexity and obviates the main philosophy behind k-NN (attributes which are close to each other could have similar densities or classes). In the proposed study, the k value assigned is 1. To determine the best performance, the data were tested with different k values and different distance metrics. Performance results from this comparison are shown in Table 3. Table 3. Performance using different k values and metrics Metric k value Accuracy (%) F-1 (%) Precision (%) Recall (%) Euclidean distance 1 86.1 85.7 86.0 86.1 3 85.6 85.1 85.4 85.6 5 84.9 84.3 84.8 84.9 9 84.2 83.6 84.0 84.2 Manhattan distance 1 85.9 85.6 85.8 85.9 3 85.6 85.1 85.4 85.6 5 84.8 84.2 84.6 84.8 9 84.1 83.6 84.0 84.2 Minkowski distance 1 86.1 85.7 86.0 86.1 3 85.6 85.1 85.4 85.6 5 84.9 84.3 84.8 84.9 9 84.2 83.6 84.0 84.2 4.3 Comparison of different classifiers In this section, the comparison of different classifiers is presented. For this evaluation, k-NN, RF, and LSTM networks are tested on the same dataset. The motivation for comparing k-NN and RF is that there is a close relationship between these two algorithms 50. Simply stated, both algorithms can be viewed as weighted neighborhood schemes. They create models from training data to make predictions for new observations by checking the neighborhoods, and formalize a weight function. RF also provides a variable ranking mechanism that can be used to select important variables. Because k-NN is a nonparametric model, it is usually a good classifier for many situations in which the joint distribution is unknown or difficult to model parametrically. This is especially true for high-dimension datasets. However, there is some dissimilarity as well. For instance, RF produces an in-memory classification model which does not require database lookups, while k-NN uses on-the-spot learning that requires extensive computations, which makes k-NN inefficient for classifying large databases. Apart from this, eager learners such as RF cannot easily model decision spaces with complicated decision boundaries; in contrast, k-NN performs instance-based learning which leads to accurate performance if a well-tuned k-NN model is used. Additionally, k-NN is a lazy learning scheme, and only stores the input data during the training process; thereby, training is essentially spontaneous 51. The parameters used for k-NN were given in the previous section. For RF, the size of each bag was set to 100, and 100 trees were used. Additionally, the number of folds for back fitting was set to 0 (no back fitting). In addition to these two algorithms, LSTM was included in this comparison. The reason is that deep learning algorithms have become very popular recently in the machine learning field, and they usually achieve higher accuracy rates compared to other classifiers. In this study, a many-to-one recurrent neural net (RNN) architecture with two LSTM cells was used. The number of hidden layers was set to 32, and the learning rate and lambda loss amount were set to 0.0015 and 0.0025, respectively. A comparison was also made between the feature vectors obtained by; (i) proposed method with proposed thresholding method, (ii) proposed method with conventional thresholding method, (iii) dimension reduced data with averaging method, and (iv) dimension reduced data with GA. The performances of these feature vectors using the k-NN classifier are shown in Figure 6 based on confusion matrices. Briefly, a confusion matrix evaluates the quality of the predictions of