Advances in human action recognition: an updated survey

IET Image ProcessingVolume 13, Issue 13 p. 2381-2394 Review ArticleFree Access Advances in human action recognition: an updated survey Syed A.R. Abu-Bakar, Corresponding Author syed@fke.utm.my Computer Vision, Video and Image Processing Research Lab, School of Electrical Engineering, Faculty of Engineering, Universiti Teknologi Malaysia, MalaysiaSearch for more papers by this author Syed A.R. Abu-Bakar, Corresponding Author syed@fke.utm.my Computer Vision, Video and Image Processing Research Lab, School of Electrical Engineering, Faculty of Engineering, Universiti Teknologi Malaysia, MalaysiaSearch for more papers by this author First published: 17 October 2019 https://doi.org/10.1049/iet-ipr.2019.0350Citations: 8AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Research in human activity recognition (HAR) has seen tremendous growth and continuously receiving attention from both the Computer Vision and the Image Processing communities. Due to the existence of numerous publications in this field, undoubtedly, there have been a number of review papers on this subject that categorise these techniques. Many of the recent works have started to tackle more challenging problems and these proposed techniques are addressing more realistic real-world scenarios. Conspicuously, an updated survey that covers these methods is timely due. To simplify the categorisation, this study takes a two-layer hierarchical approach. At the top level, the categorisation is based on the basic process flow of HAR, i.e. input data-type, features-type, descriptor-type, and classifier-type. At the second layer, each of these components is further subcategorised based on the diversity of the proposed methods. Finally, a remark on the coming popularity of deep learning approach in this field is also given. 1 Introduction Human activity or human action recognition (HAR) has been one of the most active research fields in both the computer vision and image processing societies. With an ever-increasing effort in proposing better techniques and approaches in tackling still-unsolved complex problems, it is foreseen that this trend will continue for many years to come. As a result, researchers across the globe have been actively engaged in HAR field. The primary goal of HAR is to develop an automated system that would mimic the human visual system in understanding and describing the human actions in a given scene. With the higher demands and dependency on machine intelligence, the subject of HAR has become a very important role as it has a wide spectrum of applications ranging from surveillance, automatic video indexing, man–machine interactions, and elderly-care, just to name a few, and thus, possessing high commercial values. Despite having received great attention by various researchers from various groups of interests, the challenges in solving real-world HAR problems still remain. Some of these challenges are due to background clutter, changes in lighting and illumination, occlusions be it self-occlusion or with other objects, camera view-dependent, frame resolution, different in scale and appearance, as well as the nature of the action itself. Additionally, the inter- and intra-variations in the human action add another dimension of complexity to the problem. Needless to say, being a heavily researched topic, a score of survey or review papers have been published previously covering a plethora of proposed approaches and methods [1-7]. From these works, it can be seen that categorising these available methods has never been an easy task since there is no clear line of separations between them. Moreover, as newer methods are proposed, some of these categorisations are no longer relevant or valid. Hence, the purpose of this paper is to compile recent existing methods as a continuation effort from previous surveys so that up-to-date development on HAR continues to benefit researchers who are fascinated to work in this field. In achieving this latter goal, this paper will take a simple and basic approach in grouping these methods, i.e. this survey will categorise based on the processing stages in HAR which includes image or data input, feature extraction, descriptor formation, and action classifier. This survey will not cover the various standard data sets used in the benchmarking of HAR algorithms since an extended survey on these data sets is already available [8]. However, these data sets will be mentioned in order to highlight the diversity of HAR approaches (see Tables 1–3). Table 1. Existing data sets based on input type Input Data set Multiview Multimodal MultiActor Scene/object #Actions #Subjects RGB KTH 6 25 Weizmann 10 9 BIT interaction ✓ 8 NA UT interaction ✓ 6 NA Hollywood 2 audio ✓ ✓ 12 NA UCF Sports ✓ 10 NA YouTube Action ✓ ✓ 11 NA Olympic Sports ✓ 16 NA UCF 50 ✓ ✓ 50 NA UCF 101 audio ✓ ✓ 101 NA PASCAL VOC 2012 10 NA Stanford 40 ✓ 40 NA Willow 7 7 NA UIUC 14 8 HMDB 51 ✓ ✓ 51 NA i3DPost ✓ 12 8 IXMAS ✓ 11 10 MOBOT – 6a [9] audio 8 8 TRECVID MED ✓ 15 NA CCV ✓ 20 NA RGB + depth + skeleton MSR action 3D ✓ 20 10 MSR action 3D Ext ✓ 20 23 MSR daily activity ✓ 16 10 Florence 3D 9 10 UT – Kinect 10 10 UCF – Kinect 16 16 Huawei/3D life session 1 ✓ audio 16 17 Huawei/3D life session 2 ✓ audio 17 14 3D action pair [10] 6 10 Berkeley MHAD inertia, audio 11 12 UTD – MHAD [11] inertia 27 8 CAS – YNU – MHAD [12] inertia 10 10 NUCLA action 3D ✓ 10 10 UWA 3D activity II [13] ✓ 30 10 MMI [14] ✓ ✓ 22 22 MV – TJU [15] ✓ 22 20 Multi-type [16] ✓ 12 20 CAD – 120 ✓ 10 4 Table 2. Types of features extracted with respect to data set and their corresponding highest achievement Features Data set Performance achieved (%) References Global Silhouette Wiezmann 100 [17, 18] KTH 97.1 [17] Optical Flow KTH 95 HMDB51 48.4 [19] YouTube 82.3 HOG Wiezmann 88.3 [20] Shape + Trajectories Wiezmann 98.2 [21] Depth Motion Map MSR Action 3D 100 [22] UT – Kinect 90.9 [22] NUCLA 80 [13] UWA 3D 52.2 [13] MSR daily 85 [22] Skeleton/Joints MSR Action 3D 97.6 [23] Florence 3D 95.3 [24] UT – Kinect 99.2 [24] MSR Daily 78.7 [25] MV – TJU 93.9 [15] Multi-Type 76.7 [16] NTU –RGBD 57.7 [26] MSRC – 12 95.8 [27] Local STIP KTH 98.2 [28] Weizmann 99.6 [29] Olympic Sport 77.6 [30] Hollywood 2aa Based on mean average precision (mAP). 78.5 [31] UCF50 93.2 [31] UCF101 84.5 [32] UCF Sports 93.8 [33] YouTube 89.7 [31] HMDB51 62.1 [34] IXMAS (m-view) 94.7 [31] MV – TJU (m-view) 95.8 [15] Dense Trajectory Hollywood 2aa Based on mean average precision (mAP). 67.5 [35] UCF50 94.5 [36] UCF101 97.3 [37] UCF Sports 95.3 [38] YouTube 87.9 [38] IXMAS (m-view) 84.3 [39] TRECVID (S/O)bb Include scene or object. 87.5 [40] CCV (S/O) 70.8 [40] STIP + LBP KTH 89.4 [41] HMDB51 35.2 STIP + SIFT UCF101 84.5 [32] LBP + moment KTH 96 Weizmann 91.4 [41] IXMAS (m-view) 83.2 Block Matching KTH 89.2 [42] Global + Local STIP + texture KTH 90.0 [43] HMDB51 38.94 STIP + 3D shape Huawei 3D Life #1 86.8 Huawei 3D Life #2 82.3 [44] NTU-RGBD 66.6 UT – Kinect 93.0 a Based on mean average precision (mAP). b Include scene or object. Table 3. Types of classifiers and their performance Classifier References Best performance (References) Supervised Approach SVM and all its variants [9, 10, 12-14, 16, 19, 23, 26, 28, 35-39, 42, 44-67] UCF50 – 94.5% [36] UCF101 – 97.3 [37] UCF Sports – 95.3% [38] Probability Based [21, 27, 29, 33, 68-71] KTH – 98.2% [28] Wiezmann – 99.6% [29] Hollywood2 – 78.5% [31] MSR 3D – 97.6% [23] UT-Kinect – 99.2% [24] MSRC 12–95.8% [27] IXMAS – 94.7% [31] YouTube – 89.7% [31] Multi-Task Learning [15, 24, 25, 30, 31, 72-74] Collaborative Representative and variants [75-78] Dynamic Programming [70, 79] Unsupervised Approach kNN [11, 17, 18, 41, 80-82] Wiezmann – 100% [17, 18] Clustering [20] KTH – 97.1% [17] Semi-supervised Approach Image-to-Video Adaptation [32] UCF101 – 84.5% Deep Learning Approach Convolution Neural Network [22, 40, 83] MSR 3D – 100% [22] UTD-MHAD – 91.2% [83] RNN [84] Sub-JHMDB – 88.3% ELM [85] Olympic sports – 89.7% Therefore, the structure of this paper is organised as follows: Section 2 covers the related work particularly some of the previous surveys and their categorisations. Section 3 presents the latest approaches of HAR based on the processing flow as mentioned above. Finally, Section 4 will conclude the paper. 2 Related work on previous surveys The first survey or review paper on HAR was presented in an IEEE Workshop by Aggarwal and Cai [1] in 1997 and later on published in a journal in 1999 [2]. In their work, they generalised human motion analysis as a process flow that involves body structure analysis, body parts tracking and HAR. Each of these components was treated as an independent subject of research. Thus, in HAR, they categorised it into two categories viz. state–space-based and template matching based. State–space approaches model the human motion as a sequence of Markov chain using the well-known hidden Markov model (HMM). The states in HMM are the extracted features which could be points, lines or two-dimensional (2D) blobs. Hence, given a sequence of features, HMM tries to match the model that best fits the action. One advantage of such an approach is that variation in the motion duration is not a problem as the system is capable to handle it. The disadvantage of this approach; however, is that it involves a non-linear model search for global optimum during the training phase which results in high computation time. On the other hand, in the template-matching approaches, 2D features are first extracted either in the form of optical flow, motion energy images (MEIs) or motion history images (MHIs). Next, feature vectors or descriptors are generated by concatenating features from each frame in the sequence. In order to maintain consistency in motion duration as well as to avoid large feature vector, feature normalisation and reduction may be applied. These features are then compared to those pre-stored descriptors for classifying different actions. On the downside, template matching is highly sensitive to motion duration, albeit these approaches require a less computational cost. A generalisation of the template-matching technique is the exemplar-based method which has the ability to handle some motion variations [45]. Turaga et al. [3] presented another survey that covered up to the year 2008. In their work, they distinguished the different notions between action recognition and activity recognition. They suggested action as a simple motion pattern that lasts for short duration of time (in the order of tens of seconds) performed by a single person, whereas an activity as a composite of few actions, thus more complex, and lasts for a much longer period and involves interaction among several people. Hence, they categorised action recognition into three categories viz. (i) non-parametric approaches, (ii) volumetric approaches, and (iii) parametric approaches. As for the activity recognition, they categorised it also into three categories, i.e. (i) graphical model approaches (ii) syntatic approaches, and (iii) knowledge-based approaches. Prior to the recognition part, they classified the input of the system or the low-level features into four types: (i) optical flow, (ii) trajectories, (iii) blobs, shapes, and Silhouettes, and (iv) filter response. The advantage of optical flow is that under certain assumptions the pixel intensity can be assumed to be invariant under the displacement between consecutive frames. However, in practice, its computation is susceptible to noise and changes in illumination. Trajectories refer to tracked points which suffer from occlusions, noise and background clutter, and as a result, a robust tracking algorithm is needed for reliable tracking. Blobs, shapes and human silhouettes can be obtained through background subtraction. However, they perform poorly in the presence of (self) occlusions. The final class is the filter based that is getting popular due to its ability to partially overcome the problem of occlusions and illumination changes. Examples of these approaches are the histogram of oriented gradient (HOG), spatiotemporal interest points (STIP), 3D Harris detector and 3D Gabor filter banks. The non-parametric approaches essentially work on the blobs, shapes and human silhouette inputs. In the 2D template methods, each frame is dealt with individually, and a sequence of video frames for the 3D methods. To reduce the huge amount of data, data reduction technique such as the principal component analysis (PCA) is utilised. The volume approaches make use of the filter-based and optical flow classes of features and the one that has been successful in its performance is the part-based which is to work on some selected key points and their surrounding pixels. Codewords or descriptors are then generated to represent the action and a classifier such as support vector machine (SVM) is then used to classify the action. In another approach, instead of using a classifier, a direct comparison for the similarity of the subvolume patches or cuboids from the action template has also been considered. The parametric approaches try to model the action through some probabilistic framework such as the popular HMMs and some other variants such as the linear and non-linear dynamical systems. In dealing with activity recognition, graphical models encode more complex conditional dependencies than the traditional HMM. Other models such as Bayesian network, dynamic belief network, and Petri nets have also been proposed in the literature. These approaches, however, have not been so popular because of their intractability and complexity properties. The syntactic approaches follow the rules of grammars in language modelling. In this model, activities are likened as sentences whereas action or activity primitives are taken as words. Using a set of linguistic rules, the likelihood of a complete and proper sentence (activities) based on words (action) constructions is being parsed to conform to a given grammar (activity model). Examples of these approaches include context-free grammar (CFG) techniques and stochastic grammars techniques. Finally, in the knowledge-based approaches, logic-based rules that describe some common sense domain knowledge to describe activities are employed. One of the drawbacks of these logic-based approaches is the constraint satisfaction checks which will be computationally expensive. As an alternative, ontology approaches for a specific domain such as bank monitoring, surveillance, etc. have also been proposed. Poppe [4] presented a survey on HAR methods extending from 1988 to 2010 with a different overview of action classification. He differentiated HAR-based on the image representation in a hierarchical topology, i.e. either global or local representation at the top level and more subcategories at the lower levels. Global representation is a top–down strategy that makes use of a complete human body in a form of either silhouette, optical flow or edges. Background subtraction or tracking has always been the technique used in order to get the whole human figure. However, global representation suffers from many drawbacks such as sensitive to noise, partial occlusions, and inconsistency with viewpoints. In order to partially alleviate these problems, the approach is further divided into global grid-based representation and space–time volumes. The grid-based representation spatially partitions the observation into smaller blocks of areas or cells and formulate descriptors for each cell. Matching is done at these cells levels. The space–time volumes stack a complete human body posture collected over a certain period of time to form a 3D spatiotemporal volume (STV) with time as the third dimension. Similar to the grid-based representation, weighted moments are calculated from smaller 3D patches. They are then concatenated to form a global descriptor. Albeit, these approaches require accurate alignment in the background subtraction. On the other hand, the local representation uses local descriptors or patches to describe human actions. Such approaches avoid the aforementioned drawbacks in the STV methods, and better at handling issues on viewpoint, changes in illumination and partial occlusions. These patches can be obtained by methods such as dense sampling or space–time interest point detectors. Descriptors are then formed using either a small 2D window for an image or 3D cuboid for a volume. From these image representation, Poppe classified human action into either direct classification classes, which deal directly on each image frame without taking the temporal aspect into account, or temporal state–space classifications that consider the temporal variations in actions. Examples of the direct classification methods include k-nearest neighbour (k-NN) and discriminative classifiers such as SVM and boosting. On the contrary, techniques such as dynamic time warping (DTW), HMM, conditional random fields (CRFs) and maximum entropy Markov models fall under the temporal state–space approaches. Another significant review paper was published by Aggarwal and Ryoo [5] in 2011. In this paper, they concentrated more on high-level activity recognition approaches. To be more specific, out of the four different levels of activities that they defined; gestures, actions, interactions, and group, they focused their review on the last three. Their work covered techniques that were published between 1975 and 2009. At the top level, they separated the HAR into single-layered approaches and hierarchical approaches. Single-layered approaches attempt to recognise an activity directly from the sequence of images, whereas in hierarchical approaches the activity is decomposed into primary actions or atomic units (a gesture such as ‘stretching an arm’ or ‘raising a leg’ is known as an atomic unit). Hence, an action can be considered as a combination or a sequence of these atomic units. This category is then subdivided into three classes; namely statistical, syntactic, and description-based approaches. Statistical approaches utilise probabilistic state-based models (such as HMM) in order to represent and recognise particular human activities. Syntactic approaches model the sequence of human action using atomic-level units by means of grammar syntax such as stochastic CFG. The description-based approaches represent the human activities by describing the relationship of the underlying atomic units temporally, spatially, and logically. As for the single-layered approaches, they are further sub-classified into either space–time or sequential approaches. Each of these two is further divided into sub-sub-categories. The space–time approaches are divided into space–time volume approaches, trajectory approaches, and space–time feature approaches while the sequential approaches are divided into exemplar-based and state-based approaches. The space–time volume approaches consider stacks of 2D images which form 3D image volumes similar to the approaches described in [1, 2]. The trajectory approaches use patterns generated from tracking the 2D or 3D points corresponding to the human joint positions. The space–time local feature approaches use local descriptors extracted from 3D space–time volume to represent and recognise human activities. These 3D volumes or trajectories or features are then compared in terms of their similarities with a known (trained) template corresponding to a particular activity. In general, space–time approaches do not require background subtraction operation and their features are partially scaled, rotational, and translational invariants. The exemplar-based approaches (or template matching) treat an input video as a sequence of observations or feature vectors that will then be compared to precomputed templates of sequences in determining the class of activities that the given feature vectors belong to. On the other hand, state model-based approaches model the feature vectors as a series of atomic units using a composition of set of states such as the HMM and the dynamic Bayesian network. This statistical model generates a state sequence representing a particular action with a certain probability that will be used to compute the most probable class of activity. A similar but more recent review paper was conducted by Cheng et al. [6]. They followed the same approach as in [5] for the categorisation of human activities but extended their survey to techniques that were published up to the year 2012. Additionally, they also provided a comprehensive report on the performance of these methods and even presented those methods that give the best accuracy with respect to different data sets. Interested readers can refer to their seminal work in [6]. The latest survey discussed in this paper is the survey carried out by Vrigkas et al. [7] which covered works on HAR that spanned from 1993 to 2014. Their approach to classify human action is somewhat different from the previous ones. They first described that human activities are composed of several components based on complexity. These components are gestures, atomic actions, human-to-object, and human-to-human interactions, group actions, behaviours, and events. In their work, the categorisation was based on the type of input involved. Input that involves only images or video i.e. a single type of data is categorised as unimodal approaches, whereas inputs that involve several different sources fall under the category of multimodal approaches. They further subcategorised the unimodal category into space–time methods, stochastic methods, rule-based methods, and shape-based methods. Space–time category involves space–time features as well as trajectories which include 3D volumes, STIP, and optical flow fields. Stochastic methods involve predicting the sequence of states. Some examples include HMM, Markov random field (MRF) and hidden CRF techniques. In the rule-based approaches, an activity is modelled using sets of rules or attributes that describe an event. Examples of rule-based include the linguistic-/grammar-based rules and the CFG rules techniques. Finally, the shape-based approaches attempt to recognise an action by extracting pose and appearance. Human silhouettes, skeleton joints, and depth motion map (DMM) are usually the preferred inputs used in these approaches. In a different setting, the multimodal category is subdivided into three categories: affective methods, behavioural methods, and social-networking methods. In the affective methods, human's affective states such as facial expression, physiological changes, and speech are taken into consideration in understanding and recognising the underlying human activity. Other cues that have also been used are EEG signals and pupillary reflex. The behavioural methods on the other hand endeavour to form a behaviour attributes from a non-verbal cue such as gesture, facial expression, and auditory properties. These methods work on the premises that the ability to comprehend human behaviour and its changes provides some clues in understanding complex activities. Finally, the recognition methods are based on social networking framework that involves interactions with another person, or a few persons, or a group of people, or even with objects. To complete the review, the authors also present the advantages and disadvantages of each of these approaches. Additionally, discussion and comparison on different available data sets were also tabulated for better understanding. 3 Updated review The enormous efforts by researchers on HAR field have made it difficult, as can be viewed in the previous section, to distinctly and exclusively classify a given method. More often than not, we will find that a given method may be categorised differently in a different context as can be found in the previous surveys; thus, forming a unique classification strategy that covers all existing techniques something which is hard to achieve. Therefore, instead of introducing another set of classification framework, our aim is to put together these existing categories into a more general perspective that somehow encapsulates previous works and more importantly able to embrace current approaches. Before we proceed we would like to clarify some of the terms used in this article. In the past, some works, such as Turaga et al. [3], established different meanings to ‘action’ and ‘activity’. Particularly, ‘action’ has been defined as a simple complete human motion such as walking, waving, jumping, and jogging and on the other hand, ‘activity’ has been defined as a combination of a sequence of actions to form a meaningful human action such as walking towards a table, picking up an apple and taking a bite, or two persons walking towards each other and giving a hug. While action normally is performed by a single person, an activity may involve more than a person. Likewise, an activity may involve not only person-to-person interaction (PPI) but also person-to-object interaction. Aggarwal and Ryoo [5] extended further and defined the movement of the limbs of the body such as the lifting of the hand or swinging of the arm as an atomic action unit. Following this definition, it is obvious then several atomic action units form action, and a series of actions form an activity. In this review, however, as a generalisation, we will not differentiate between action and activity, and unless otherwise stated, we will use these two words interchangeably. The basic canonical method in HAR consists of input data, feature extraction, descriptor formation, and classification which follows sequentially from left to right as shown in Fig. 1. Hence, in this survey, we will analyse and discuss current methods in the manner of this flow. On doing so, we will cover methods not only meant for video sequences but also those techniques that have been designed for still images as well. Fig. 1Open in figure viewerPowerPoint General process flow for canonical HAR methods 3.1 Data input Initially, research on HAR revolves around complete human-centric movements as captured by the optical camera. With the advancement in the sensor technology, researchers have utilised different sensors as input to the system such as the RGB-D cameras which include the depth information as well as joints positions depicting humans ‘skeletons’ also known as the stick figure. 3.1.1 RGB or colour based By far, RGB, or colour-based data is still the popular choice of input for two simple reasons. First, these cameras are easily available and widely used in surveillance, webcam, and embedded devices and second, they are still the cheapest image capturing devices compared to others. Hence, many of the current methods such as in [30, 34, 38, 46, 47, 68, 72] use the colour-based images as input to their systems. All these methods directly exploit the intensity values from the red, green, and blue channels of the image. An example of an RGB or colour-based image is shown in Fig. 2a. Fig. 2Open in figure viewerPowerPoint Types of input (a) RGB- or colour-based image, (b)Its corresponding depth-based image (depth map). Different depthis represented by a different shade of grayscale,(c) Its correspondingskeleton-based image [86] 3.1.2 RGB-based + Depth Another type of input which is gaining considerable interest among researchers is the RGB-D cameras such as the Microsoft Kinect [87] and Asus Xtion PRO LIVE [88] devices. Besides giving the intensity values of all objects in the scene, these cameras also provide the depth information in the form of depth map images; thus, providing a readily available 3D data with respect to the viewpoint. The additional depth information provides better handling of pose estimation problems, increases the robustness to changes in illumination and simplifies the segmentation of foreground (human in this case) from the background. Some of the HAR works that make use the depth information with high accuracy can be found in [22, 44, 48, 75, 80]. An example of a depth map image is illustrated in Fig. 2b. 3.1.3 Skeleton based The third category of input data is the skeleton-based image (see Fig. 2c). Similar to the depth map images, skeleton images are generated by Kinect-like depth sensors. This type of sensors represents the human in a form similar to a stick figure with a number of joint positions (shown in short green and blue lines in Fig. 2c) portraying the joints of human limbs (Kinect provides up to 25 joint positions [89]). Using skeleton-based images in HAR somewhat alleviates many problems faced by the RGB or colour-based images such as changes in illumination, background clutter, pose variability, and appearance (texture) dependency. With the advent of portable and affordable depth sensor devices, there has been an increase of interest in recognising human activity by means of human skeleton images [10, 24, 25, 73]. 3.1.4 Multi-modal based Apart from the intensity-based input dev