# Key-Skeleton-Pattern Mining on 3D Skeletons Represented by Lie Group for Action Recognition

Authors: Guang Li; Kai Liu; Wenwen Ding; Fei Cheng; Boyang Chen
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

The human skeleton can be considered as a tree system of rigid bodies connected by bone joints. In recent researches, substantial progress has been made in both theories and experiments on skeleton-based action recognition. However, it is challenging to accurately represent the skeleton and precisely eliminate noisy skeletons from the action sequence. This paper proposes a novel skeletal representation, which is composed of two subfeatures to recognize human action: static features and dynamic features. First, to avoid scale variations from subject to subject, the orientations of the rigid bodies in a skeleton are employed to capture the scale-invariant spatial information of the skeleton. The static feature of the skeleton is defined as a combination of the orientations. Unlike previous orientation-based representations, the orientation of a rigid body in the skeleton is defined as the rotations between the rigid body and the coordinate axes in three-dimensional space. Each rotation is mapped to the special orthogonal groupSO(3). Next, the rigid-body motions between the skeleton and its previous skeletons are utilized to capture the temporal information of the skeleton. The dynamic feature of the skeleton is defined as a combination of the motions. Similarly, the motions are represented as points in the special Euclidean group SE(3). Therefore, the proposed skeleton representation lies in the Lie group (SE(3)×⋯×SE(3), SO(3)×⋯×SO(3)), which is a manifold. Using the proposed representation, an action can be considered as a series of points in this Lie group. Then, to recognize human action more accurately, a new pattern-growth algorithm named MinP-PrefixSpan is proposed to mine the key-skeleton-patterns from the training dataset. Because the algorithm reduces the number of new patterns in each growth step, it is more efficient than the PrefixSpan algorithm. Finally, the key-skeleton-patterns are used to discover the most informative skeleton sequences of each action (skeleton sequence). Our approach achieves accuracies of 94.70%, 98.87%, and 95.01% on three action datasets, outperforming other relative action recognition approaches, including LieNet, Lie group, Grassmann manifold, and Graph-based model.


---

## Body

## 1. Introduction

Human action recognition is currently the most dynamic research topic in the field of computer vision, owing to its applications in intelligent surveillance, video games, robotics, and other fields. Several approaches have been proposed to recognize human action from RGB video sequences over the past few decades [1], but their performance is unsatisfactory because RGB data are very sensitive to factors such as perspective changes, occlusions, and background clutter. Although significant research results have been achieved, human action recognition remains a challenging problem.Because the human skeleton can generally be regarded as an articulated system of rigid segments, which are connected by joints, human action can be viewed as a continuous evolution of the spatial configuration, which is constructed by these rigid segments [2]. Therefore, if human skeleton sequences can be accurately extracted from RGB videos, action recognition can be performed by classifying these sequences. However, it is very difficult to accurately extract a skeleton sequence from RGB videos [3]. With the advent of cost-effective RGB-D cameras, it has become easier to extract the three-dimensional (3D) human skeleton from depth maps. Although this improves the appearance and viewpoint variations to a certain extent [4–7], the following two challenges cause large intraclass variations and remain unresolved. First, different people can perform the same action in different ways. Second, the 3D human skeleton is sometimes imprecise because depth maps include noisy information. However, a psychological research found that humans can easily recognize an action from a pose sequence [8]. According to the work of [8], Yang et al. considered that actions can be classified by a single key pose [9]. This suggests that a set of key skeletons can be used to perform action classification rather than the entire skeleton sequence. Since the representation of the key poses is robust to outlier poses, this approach should improve the accuracy of action recognition as long as the key poses are accurate.The general framework of the proposed approach is shown in Figure1. By observing human action in daily life, the orientations and motions of rigid bodies can include a lot of useful information for action recognition. In this paper, a new skeletal representation, which is composed of the static feature and dynamic feature, is proposed for 3D skeleton-based action recognition. The static feature is used to represent the spatial information in a given skeletont. To capture the scale-invariant spatial information, the orientations of the rigid bodies in the skeleton are employed to construct its static feature. In this work, the orientation of a rigid body in the skeleton is represented as six rotation matrices between the rigid body and the three coordinate axes in 3D space. The rotation matrices are mapped to the special orthogonal group SO(3)[10]. Next, the dynamic feature is employed to represent the temporal information of skeletont. The dynamic feature is composed of the rigid-body motions between skeletonst andt-1 and those between skeletonst and 1 (the three skeletons belong to the same sequence or action). The motions are represented as points in the special Euclidean group SE(3)[11]. Hence, skeletont is represented by a point in Lie Group (SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3)⋯×SO(3)),where the operation × represents the direct product between groups in group theory. Using the proposed skeleton representation, a human action (skeleton sequence) can be represented as points in the Lie group. However, it is typically a very complicated task to classify human actions represented by a Lie group directly. Many standard classification approaches, such as the support vector machine (SVM)[12] approach, are not directly applicable to Lie groups. To overcome the classification difficulties, the actions (skeleton sequences) are mapped from the Lie Group to its Lie algebra (se(3)×se(3)×⋯×se(3),so(3)×so(3)×⋯×so(3)), which are the elements of the tangent space of the manifold at the identity element. The Lie algebra is a vector space, which makes action classification easier.Figure 1
The general framework of the proposed method.An action (skeleton sequence) usually includes many noisy skeletons, which can reduce the action recognition accuracy. In this study, the key-skeleton-patterns are used to eliminate noisy skeletons from an action, and the remaining skeletons in the action are called the most informative skeleton sequence. First, a pattern is defined as a short skeleton sequence, which is not necessarily adjacent in the original skeleton sequences. If the short skeleton sequence appears in many skeleton sequences of an action class, the pattern is called the key-skeleton-pattern in that class. Next, to mine the key-skeleton-patterns, k-means is used to learn the symbolic dictionary from all skeletons in the dataset. Each symbol in the dictionary represents a class of similar skeletons, which means that each skeleton is quantized (represented) by a symbol in the dictionary. Then, a skeleton sequence can be represented as a symbol sequence. In this paper, probability is used to measure the distance between a skeleton and its corresponding symbol in order to minimize the effect of quantization errors (e.g., two different skeletons are quantized by the same symbol). Hence, each skeleton is represented by a distance-based probability, and an action is represented as a probability sequence. Then, a new pattern-growth algorithm named MinP-PrefixSpan is proposed to mine the key-skeleton-patterns of an action class from the symbol sequences and the probability sequences that correspond to the action class. Compared with the PrefixSpan algorithm, our algorithm achieves higher efficiency by reducing the number of new skeleton patterns in each growth step. Finally, the key-skeleton-patterns are utilized to eliminate noisy skeletons from the action in order to capture the most informative skeleton sequence of the action. An SVM is employed to classify the most informative skeleton sequences.The main contributions of this study are as follows. (1)To capture the scale-invariant skeletal information, the orientations of rigid bodies in a skeleton are utilized to construct the static feature. Different from previous orientation-based approaches, in this study, a rigid-body orientation is represented as six rotation matrices, and each rotation matrix is represented as a point in SO(3). (2) Traditional approaches based on Lie groups [5, 13, 14] only consider the spatial information of a skeleton but ignore the temporal information between different skeletons. Therefore, our approach employs the rigid-body motions between different skeletons to describe the temporal variation. Likewise, the motions can be represented as points on SE(3). (3) Traditional approaches also ignore the influence of noisy skeletons in an action on the accuracy of the action recognition. In this study, based on the PrefixSpan algorithm [15] in data mining, a new pattern-growth algorithm is proposed to mine the key-skeleton-patterns of each action class, and the key-skeleton-patterns are used to eliminate noisy skeletons.


## 2. Related Work

A brief overview of the related work on human action recognition approaches based on skeletons is provided in this section, and various sequential pattern-mining algorithms are reviewed.The existing skeleton-based action recognition approaches can be classified into three main categories. The first class of approach ignores the influence of noisy skeletons on the accuracy of action recognition. Slama et al. represented an action by an observability matrix, which was characterized by an element of a finite Grassmann manifold [16]. However, their method does not eliminate noisy skeletons from an action, and it is insufficient to estimate the approximation of an extended observability sequence with a finite Grassmann manifold. Ding et al. divided actions into subactions and used the profile hidden Markov model(HMM) to align them [13]. Although their approach accurately extracts the spatial features of an action, it does not solve the following two problems: eliminating noisy skeletons and reducing the time complexity of the profile HMMs. Liu et al. proposed a new spatiotemporal representation, called "Skepxels," to transform skeleton videos into images of flexible dimensions, and employed the resulting images to build a CNN-based framework for effective human action recognition[17]. Likewise, their approach does not eliminate noisy skeletons from an action. In this study, the key-skeleton-patterns of an action are utilized to eliminate noisy skeletons from the action as an approach to improve the accuracy of action recognition.The second class of approach ignores scale variations from subject to subject, which means that the spatial feature of an action cannot be accurately represented. Chaudhry et al. hierarchically divided the human skeleton into smaller parts and employed certain bio-inspired shape features to represent each part [18]. The temporal evolutions of these bio-inspired features are modeled by linear dynamical systems (LDSs). Although their approach takes full advantage of the correlation between the skeletal parts, it ignores the feature of the rigid bodies in a skeleton and the scale variations between different subjects. Xia et al. proposed a view-invariant representation of the human skeleton using histograms of 3D joint locations [19]. The temporal evolutions of this skeletal representation are modeled by a discrete HMM. However, their approach not only ignores the relativity between the rigid bodes in a skeleton but also the normalization of the skeleton data. Li et al. represented an action by a special graph based on the top-K relative variance of joint relative distance (RVJRD) [20]. One potential limitation of this approach is that the graph-based model does not handle scale variations, which may cause incorrect spatial information to be selected by the top-K RVJRD. In contrast, our proposed approach uses the orientations of the rigid bodies in a skeleton to capture scale-invariant skeletal features.The third class of approach ignores the temporal information of an action and treats the poses in the action independently. Evangelidis et al. used a local skeleton descriptor to encode the relative positions of joint quadruples [21]. The descriptor of an action was represented by a multilevel Fisher vector composed of the local skeleton descriptor in the action. However, the action descriptor not only ignores the temporal information between different skeletons but also has high time complexity. Huang et al. combined the Lie group structure with a deep network framework [22]. Their learning structure (LieNet) has a rotation mapping layer transforming the Lie group features into the traditional neural network model. One main limitation of this approach is that LieNet ignores the rich temporal information of human actions. Vemulapalli et al. described the relative geometry between the rigid-body parts using special Euclidean group SE(3)[5]. Therefore, the entire skeleton in an action can be represented as a point in SE(3). An action is represented as a curve in the Lie group (SE(3)×SE(3)⋯×SE(3)). Although their approach can accurately extract the spatial information of a skeleton, it ignores the temporal cues between the skeletons in an action and does not eliminate noisy skeletons from the action. Our proposed dynamic feature models the temporal structures of an action using the rigid-body motions between different skeletons in the action.Sequential pattern mining aims to discover frequent subsequences as patterns in a sequence database. Traditional sequential pattern mining algorithms [23–26] are usually used to mine frequent sequential patterns from deterministic databases. However, those approaches cannot be indirectly applied to uncertain data (or probabilistic data). Unfortunately, the existing pattern mining algorithm on an uncertain dataset [27, 28] is not adopted to our probabilistic sequence model. Therefore, considering the amount of noise in our uncertain datasets (probabilistic datasets), a new pattern-growth algorithm is proposed to mine the key-skeleton-patterns from the datasets.


## 3. Proposed Framework

### 3.1. Fundamental Concepts

In this subsection, a brief overview of the special Euclidean groupSE(3) and the special orthogonal group SO(3) is presented, which is necessary for further understanding of the Lie group. We refer the readers to [2, 10, 11] for a general introduction to Lie groups. Important notations are shown in Table 1.Table 1
Notations involving Lie groups.
Notation Description SO(3) Special orthogonal group  so(3) Lie algebra ofSO(3)  SO(3) Special Euclidean group  se(3) Lie algebra ofSE(3)  × Direct product of Lie group


#### 3.1.1. Special Orthogonal Group

The special orthogonal groupSO(3) is a Lie group, which can be represented by all 3×3 orthogonal matrices shown as follows:(1)SO3=A∈R3×3;ATA=I3,detA=1where I3 denotes the 3×3 identity matrix and A is a rotation matrix. In 3D space, a rotation A is an element of SO(3) and can transform a vector x=x1,x2,x3T to x′ by(2)x′=AxEvery groupSO(3) has an associated Lie algebra of SO(3) that is the tangent space around the identity element I3. The Lie algebra of SO(3), denoted by so(3), is a set of all real 33 skew-symmetric matrices as follows:(3)so3=M∈R3×3;Mt=-M. Given an element(4)Ω=0-m3m2m30-m1-m2m10∈so3 its vector form vec(Ω) is(5)vecΩ=m1,m2,m3T The exponential map expSO(3) from so(3) to SO(3) and the logarithm map logSO(3) from SO(3) to so(3), respectively, are(6)expSO3Ω=eΩ;logSO3A=logA.


#### 3.1.2. Special Euclidean Group

The special Euclidean group SE(3) is a Lie group, which is a set of 4 by 4 matrices(7)SE3=H∈R4×4,H=Ab01,A∈SO3,b∈R3. The matrix representation also makes SE(3) action on points c∈R3 by rotating and translating them:(8)c′=H·c=Ab01·c=Ac+b1.Every Lie Group SE(3) can be associated with a Lie algebra se(3), which is the tangent space of the Lie group SE(3) at the identity matrix I4. Note that se(3) is a 6D vector space that can be formed by 4 by 4 matrices(9)se3=Mv00,M∈R3×3,v∈R3,MT=M. Given an element(10)Ω=0-m3m2v1m30-m1v2-m2m10v30000∈se3, its vector form vec(Ω) is(11)vecΩ=m1,m2,m3,v1,v2,v3T. The exponential map expSE(3) from se(3) to SE(3) and the logarithm map logSE(3) from SE(3) to se(3), respectively, are(12)expSE3Ω=eΩ;logSE3H=logH.S O ( 3 ) × ⋯ × S O ( 3 ) and SE(3)×⋯×SE(3): the direct product × is used to combine multiple SO(3), which form a new Lie group K=SO(3)×⋯×SO(3) with identity element (I3,…,I3) and its Lie algebra k=so(3)×⋯×so(3). The exponential of (Ω1,…,ΩN)∈k and the logarithm map of (A1,…,AN)∈K, respectively, are given by(13)expKΩ1,…,ΩN=eΩ1,…,eΩN(14)logKA1,…,AN=logA1,…,logAN.The vector form of logK((A1,…,AN)) is(15)veclogKA1,…,AN=veclogA1,…,veclogAN. Similarly, a new Lie group K=SE(3)×⋯×SE(3) with identity element(I4,I4,…,I4) and its Lie algebra k=se(3)×se(3)×⋯×se(3) are formed by using the direct product ×. The exponential map of (Ω1,…,ΩN)∈k and the logarithm map of (H1,…,HN)∈K, respectively, are given by(16)expKΩ1,…,ΩN=eΩ1,…,eΩN(17)logKH1,…,HN=logH1,…,logHN. The vector form of logK((H1,…,HN)) is(18)veclogKH1,…,HN=veclogH1,…,veclogHN.


#### 3.1.3. Explanation of Fundamental Concepts

According to the concepts described in Section3.2.1, the orientation of a rigid body is represented by six rotation matrices. Mathematically, a rotation matrix is a point in SO(3); therefore, the orientation of a rigid body in skeleton t can be represented as six points in SO(3). Then the static feature, composed of the orientations of the rigid bodies in the skeleton, is represented as a point in the Lie group (SO(3)×SO(3)⋯×SO(3)), as shown in Figure 1.The motions of a rigid body is generally regarded as its rotations and translations in 3D space. Mathematically, the rotations and translations of a rigid body are defined asSE(3); therefore, a rigid-body motion between skeletons t and t-1 (or skeletons t and 1) can be represented as a point in SE(3). Then, the dynamic feature, which is composed of the rigid-body motions between skeletons t and t-1 and those between skeletons t and 1, is represented as a point in the Lie group (SE(3)×SE(3)⋯×SE(3)), as shown in Figure 1. A skeletal representation, composed of the static feature and dynamic feature, can be represent as a point in the Lie group (SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3).The wavy surface represents a Lie group(SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3)⋯×SO(3)) in Figure 1. A whole circle in the wavy surface represents an action (skeleton sequence). Each black dot in the circle represents a skeleton. Then, an action can be represented as points in the Lie group (the points are included in the same circle). To overcome the classification difficulties, an action (or a whole circle) is mapped from the Lie group to its Lie algebra (se(3)×se(3)×⋯×se(3),so(3)×so(3)×⋯×so(3)), as shown in Figure 1. In fact, the Lie algebra is a vector space.


### 3.2. Extraction of Skeleton Features

In this subsection, the static and dynamic features of a skeleton are represented as a point in the Lie group. LetZ=(C,E) be a skeleton. The set of bone joints is denoted by C=c1,c2,…,cN, and the set of rigid-body parts is denoted by E=e1,e2,…,eM, where c,e∈R3. Figure 2(a) shows an example of the human skeleton with 19 rigid-body parts and 20 bone joints.Figure 2
(a) Human skeleton with 19 rigid body parts and 20 bone joints. (b) Placing the hip center to the origin and aligning the x-axis with the ground plane projection of the vector from the left hip to the right hip.
(a) (b)


#### 3.2.1. Static Feature of Skeleton

By observing human action, the orientations of rigid bodies in a skeleton (pose) can include a lot of valuable information for action recognition. To describe the orientation of a given rigid bodyem, the global coordinate system is translated to the local coordinate system. Am,x, Am,y, and Am,z represent the three rotations that transform the rigid body em to the three coordinate axes, as shown in Figure 3(c). Their rotation relationship is shown as follows:(19)rx=Am,xem,ry=Am,yem,rz=Am,zem, where Am,x, Am,y, and Am,z∈SO(3). Similarly, Ax,m, Ay,m, and Az,m represent the three rotations that transform rx, ry, and rz to the rigid body em, respectively, as shown in Figure 3(d). Their rotation relationship is shown as follows:(20)em=Ax,mrx,em=Ay,mry,em=Az,mrz, where Ax,m, Ay,m, and Az,m∈SO(3). The six rotations can be used to describe the orientation of the rigid bodies.Figure 3
Representation of the orientation of rigid bodyem.
(a) (b) (c) (d)Given a skeletont, om(t)=Am,x(t),Ax,m(t),Am,y(t),Ay,m(t),Am,z(t),Az,m(t) is used to represent the orientation of em(t) in the skeleton. In this work, the skeletal staticfeature is defined as a set of the orientations of the rigid bodies in the skeleton as follows:(21)fot=o1t,o2t,…,oMt∈SO3×SO3×⋯×SO3, where M is the total number of rigid bodies in the human skeleton.


#### 3.2.2. Dynamic Feature of Skeleton

Rigid-body motion is generally regarded as rotations and translations in 3D space. Mathematically, the rotations and translations of a rigid body can be denoted bySE(3). In this study, SE(3) is employed to describe rigid-body motions between different skeletons. Let em(i)∈R3 be rigid body em in skeletoni and em(j)∈R3 be rigid body em in skeletonj(i=j).Given a pointk(i)∈em(i) and k(j)∈em(j) corresponding to k(i), we have(22)kj1=Hmi,jki1=Ami,jbmi,j01ki1, where Hm(i,j)∈SE(3) and Am(i,j) and bm(i,j) are the rotation and translation, which can transform em(i) to the position and orientation of em(j), respectively, as shown in Figure 4(b).Figure 4
Representation of the relative geometries of rigid bodyem at time instance t.
(a) (b) (c)Similarly, given a pointk(j)∈em(j) and k(i)∈em(i) corresponding to k(j), we have(23)ki1=Hmj,ikj1=Amj,ibmj,i01kj1, where Hm(j,i)∈SE(3) and Am(j,i) and bm(j,i) are the rotation and translation, which can transform em(j) to the position and orientation of em(i), respectively, as shown in Figure 4(c). dm(i,j)=Hm(j,i),Hm(i,j) is used to represent the motion of rigid body em between skeletons i and j. Then, the rigid-body motions between skeletons i and j can be represented by(24)frgi,j=d1i,j,d2i,j,…,dMi,j∈SE3×SE3×⋯×SE3,where M is the total number of the rigid bodies in the skeleton.In this study, our approach only considers the rigid-body motions between skeletonst andt-1 and those between skeletonst and 1. According to formula (24), the rigid-body motions between skeletonst andt-1 can be represented by frg(t,t-1). Similarly, the rigid-body motions between skeletonst and 1 can be represented by frg(t,1). Then, the skeletal dynamicfeature is defined as a set of the rigid-body motions in skeleton t as follows:(25)fmft=frgt,t-1,frgt,1.Skeletont is represented by the static feature and the dynamic feature as follows:(26)ft=fot,fmft, where fo(t) represents the static feature of the skeleton and fmf(t) represents its dynamic feature.


### 3.3. Skeleton Sequence Representation

#### 3.3.1. Lie Group Representation of Skeleton Sequence

Using the proposed skeletal feature, a skeleton sequence or an action is represented by(27)LG=ft,t∈1,T, where T is the total number of frames in the sequence and f(t)∈ Lie group SO(3)×⋯×SO(3),SE(3)×⋯×SE(3).


#### 3.3.2. Lie Algebra Representation of Skeleton Sequence

Since the most classification methods (such as SVM) cannot be directly applied to manifolds, to overcome these difficulties,f(t) is mapped to its Lie algebra (so(3)×⋯×so(3),se(3)×⋯×se(3)).The Lie algebra of fo(t) is given by(28)got=veclog⁡o1t,veclog⁡o2t,…,veclog⁡oMt∈so3×so3×⋯×so3 and the Lie algebra of frg(i,j) is given by(29)grgi,j=veclog⁡d1i,j,veclog⁡d2i,j,…,veclog⁡dMi,j∈se3×se3×⋯×se3. The Lie algebra of f(t) is given by g(t)=go(t),gmf(t)=go(t),grg(t,t-1),grg(t,1). A human action can be represented by the following Lie algebra structure:(30)LA=gt,t∈1,T where T is the total number of frames in the sequence.M is the total number of rigid bodies in skeletont. Given a rigid body i in skeleton t, vec(log(oi(t)) is the Lie algebra’s representation of the orientation of the rigid body, which is 18-dimensional vector(i∈[1,M]). go(t)=vec(log(o1(t)),vec(log(o2(t)),…,vec(log(oM(t)) is the Lie algebra’s representation of the static feature of skeleton t, which is a (18×M)-dimensional vector. vec(log(di(p,q)) is the Lie algebra’s representation of the motions of rigid body i between skeletons p and q, which is 12-dimensional vector. grg(t,t-1)=vec(log(d1(t,t-1)),vec(log(d2(t,t-1)),…,vec(log(dM(t,t-1)) is the Lie algebra’s representation of the rigid-body motions between skeletons t and t-1, which is a (12×M)-dimensional vector. grg(t,1)=vec(log(d1(t,1)),vec(log(d2(t,1)),…,vec(log(dM(t,1)) is the Lie algebra’s representation of the rigid-body motions between skeletons t and 1, which is a (12×M)-dimensional vector. gmf(t)=grg(t,t-1),grg(t,1) is the Lie algebra’s representation of the dynamic feature of skeleton t, which is a (24M=12M+12M)-dimensional vector. g(t)=go(t),gmf(t) is the Lie algebra’s representation of skeletont, which is a (42M=18M+24M)-dimensional vector. Hence, a human action can be seen as temporal evolutions of a 42M-dimensional vector.


### 3.4. Key-Skeleton-Pattern Mining

In the previous subsections, a skeleton sequence is represented as the Lie algebra structureLA=g(t),t∈[1,T], where T is the total number of frames in a skeleton sequence. However, a skeleton sequence can include many noisy skeletons, which reduce the accuracy and efficiency of the action recognition. In this subsection, the key-skeleton-patterns are used to eliminate noisy skeletons in a skeleton sequence in order to capture the most informative skeleton sequences.


#### 3.4.1. Formal Definitions

To mine the key-skeleton-patterns, classic k-means is used to quantize all skeletons represented by the Lie algebra to K symbol. LetS=s1,…,sK be a set containing K symbol and C=c1,…,cK be a set of centroid. Then, a skeleton sequence can be represented as a symbol sequence L=l1,…,lT(li∈S,i∈[1,T]). Since different skeletons may be quantized as the same symbol, to minimize the effect of quantization errors, each skeleton g(i) in a sequence is represented by probability pi, which is used to measure the distance between skeleton g(i)∈LA and centroid ci∈C as follows:(31)pi=distci,gi-1∑j=1Kdistcj,gi-10⩽pi⩽1,where cj∈C correspond to symbol sj∈S. Equation (31) shows the distance inversely proportional to pi. Now, a skeleton sequence also can be represented by a probability sequence P=p1,…,pT.Definitions. Some terms are defined in this paper as follows (Important notations are in Table 2.).Table 2
Notations involving data mining.
Notation Description α Skeleton pattern  d Mining sequence  D|α Projected dataset  SUP(α,D|α) Support of skeleton patternα  ES(α,D|α) Expected support of skeleton patternαDefinition 1 (pattern).
α = α ( 1 ) , … , α ( m ) is a sequence that contains m symbols chosen from the dictionary, i.e., α(i)∈S.Definition 2 (mining sequence).
d = P , L , L A is a mining sequence applied to mine the Key-skeleton sequence. P is a probability sequence, which represents a skeletons sequence. L is the symbol sequence, which corresponds to P. LA is a skeleton sequence represented by the Lie algebraic structure.Definition 3 (projected dataset).
Given a patternα and a mining sequence dataset D(d∈D) of an action class, the α-projected dataset D|α is defined by the setd|d∈D∧α⊆d.L.Definition 4 (support).
For a patternα and a symbol sequence d.L∈D|α (where d is an element of D|α), let F(α,d.L) be an indicator variable with value 1 if α is a subsequence of the symbol sequence d.L, and 0 otherwise. For any pattern α, its support in D|α is denoted by SUP(α,D|α)=∑i=1D|α‍F(α,d.L)Definition 5 (expected support).
Given a patternα and a symbol sequence d.L (where d is an element of D|α), let pos=Find(α,d.L) be the positions where the pattern α takes up in d.L and let d.P(α)=∏j=1pos‍d.P[pos[j]] be the product of the probability. For any pattern α, its expected support in D|α is denoted by ES(α,D|α)=∑i=1D|α‍d.P(α)/D|α.Definition 6 (key-skeleton pattern).
Given a patternα and a mining sequence dataset D of an action class, if SUP(α,D|α) is larger than a threshold τsup and ES(α,D|α) is larger than a threshold τprob, then the pattern α is called a key-skeleton-pattern of that action class. A key-skeleton-pattern of length i is called an i-pattern.


#### 3.4.2. MinP-PrefixSpan Algorithm

In this subsection, a new pattern-growth algorithm, called MinP-PrefixSpan, is proposed to mine the key-skeleton-patterns of an action class by searching over the enormous space of the symbol sequences and probability sequences of the action class. The algorithm is shown in Algorithm1. In Lines 2-8, the dataset D|α is employed to construct a new projected dataset D|αe. In Lines 9-11, if αe is the key-skeleton-pattern, pattern αe is appended to K and symbol table T|αe is constructed by Trim(T|α,D|αe). In Line 12, MinP-PrefixSpan is recursively called to grow the key-skeleton-pattern until all key-sequence patterns are found.Algorithm 1:MinP-PrefixSpan(T|α,D|α,K).
Input:  α-projected dataset Dα, symbol table T|α, key-skeleton-pattern dataset K (1)  for each symbol e∈T|α do (2)   D|αe←∅ (3)   for each d∈D|α do (4)    pos=Find(α,d.L) (5)    ife∈d.L[pos[α]+1,⋯,d.L] then (6)     Append d to D|αe (7)    end if (8)   end for (9)   if SUP(αe,D|αe)≥τsup and ES(αe,D|αe)≥τes then (10)   T|αe←Trim(T|α,D|αe) (11)   Append αe to K (12)   MinP-PrefixSpan(T|αe,D|αe,K) (13)  end if (14)  Free D|αe and T|αe (15) end forIn Line 10, the trim algorithm is used to improve the efficiency of the MinP-PrefixSpan algorithm by eliminating nonkey-skeleton-patterns. Algorithm2 shows the implementation details of the trim algorithm. The trim algorithm mainly consists of the following two parts:Algorithm 2:Trim(T|α,D|αe).
Input:symbol table T|α, αe-projected dataset Dαe Output: symbol table T|αe (1) T|αe←∅ (2) for each symbol m∈T|α do (3)  if SUP(αem,D|αe)≥τsup and ES(αem,D|αe)≥τes then (4)   T|αe←T|αe∪m (5)  end if (6) end forT r i m i n g   r u l e s . Given a mining sequence dataset of an action class D and a pattern α (according to Definition 3, D|α is the α-projected dataset), two rules are proposed to trim nonkey-skeleton-patterns as follows:(1)
IfSUP(α,D|α)≤τsup, then pattern α is a non-key-skeleton-pattern.(2)
IfES(α,D|α)≤τes, then pattern α is a non-key-skeleton-pattern.P a t t e r n   g r o w t h . Referring to the pattern-growth method of Prefxispan, one symbol e is used to grow key-skeleton-pattern α and check the support and expected support of the pattern αe. A symbol table T|α is used to store each symbol e in order to reduce the number of new skeleton patterns in each growth step. An important property is found between symbol tables.Property 7 (symbol table).
If a key-skeleton-patternγ grows from α, then T|γ⊆T|α.Proof.
Let us denoteγ=αe(1) as the key-skeleton-pattern and e(1)∈T|α. Suppose e(2)∈T|γ and β=αe(1)e(2) also is a key-skeleton-pattern on D|α. Since SUP(αe(2),D|α)≥SUP(β,D|α) according to Definition 4, SUP(αe(2),D|α)≥τsup. Since ES(β,D|α)-ES(αe(2),D|α)≥0 according to Definition 5, ES(αe(2),D|α)≥τes. We conclude that e(2) belongs to T|α,which implies that T|γ⊆T|α.


### 3.5. Discovering the Most Informative Skeleton Sequence

The task of Algorithm3 is to discover the most informative skeleton sequences for all actions. Let J be a mining sequence dataset of all actions. K is a dataset used to store the key-skeleton-patterns of all action classes, and Q is a dataset used to store the most informative skeleton sequences of all actions. In Lines 2-11, the key-skeleton-patterns of each class action are mined from training dataset U and appended to dataset K. In Lines 12-22, the key-skeleton-patterns in dataset K are employed to discover the most informative skeleton sequence of the actions in dataset J, and all most informative skeleton sequences are stored in dataset Q(refer to Figure 5).Algorithm 3:Informative skeleton sequence (K,J).
Input:key-skeleton-pattern dataset K, mining sequence dataset of all actions J Output: The dataset of the most informative skeleton sequences Q (1)  Obtaining training dataset U from J (2)  K←∅; Q←∅ (3)  for the dataset of each action class D⊂ U do (4)   T is the table that includes all symbols in D (5)   for each symbol α∈T do (6)    if SUP(α,D|α)≥τsup and ES(α,D|α)≥τes then (7)     T|α←Trim(T,D|α) (8)     append α to K (9)     MinP-PrefixSpan(T|α,D|α,K) (10)   end if (11)  end for (12) end for (13) for each element d∈J do (14)  for each key-skeleton-pattern m∈K  do (15)   if m is a subsequence of d.L then (16)    patternpos=zero(len(d.L),1) (17)    position=Discoverlocation(m,d.L) (18)    for i∈position do (19)     patternpos[i]=1 (20)    end for (21)   end if (22)  end for (23)  for z=1 to Len(d.L) do (24)   if patternpos[i]==1 then (25)    append d.LA[z] to s (26)   end if (27)  end for (28)  Append s to Q (29) end for (30) return QFigure 5
Discovering most informative skeleton sequence.Dynamic Time Warping (DTW)[33] has excellent performance in searching for anoptimal alignment between time sequences. Therefore, for each action class, our model uses the action standardization algorithm proposed by the author of [5] to compute a nominal action and employs DTW to warp all the training or testing actions into this nominal action. SVMs are extensively used in computer vision to achieve excellent performances in image and video classifications. To achieve better classification results, a linear SVM is used to classify the most informative skeleton sequences.


### 3.6. Datasets

In this study, three standard 3D human action datasets are employed to study the effectiveness of the proposed method.M S R A c t i o n 3 D   d a t a s e t [34] can be captured using a depth camera similar to the Kinect device. This dataset consists of 20 actions of 10 subjects, with each action having two or three repetitions. In total, there are 557 action sequences. The dataset provides 3D locations of 20 joints. The horizontal and vertical locations of each skeleton joint are stored in the screen coordinates, and the skeleton’s depth position is stored in the global coordinates. Human actions in this dataset capture various types of motions, which are related to arms, legs, torso, and their combinations. Experiments on this dataset are challenging, but the dataset is widely applied to test the accuracy and robustness of recognition methods for various actions.U C K i n e c t - A c t i o n   d a t a s e t [19] is captured using a stationary Kinect sensor. It consists of 10 human actions obtained from daily life: walking, sitting down, standing up, picking up, carrying, throwing, pushing, pulling, waving, and clapping hands. Each human action is performed by 10 different subjects (nine males and one female) twice or thrice. In total, there are 199 action sequences. This dataset is very challenging. First, for some action sequences, parts of the human body are invisible because the body parts are out of the field of view. Second, subjects performed the same action using different limbs, such as waving the left hand and waving the right hand. Third, it is very difficult to capture the action sequences with invariance to the view point.G 3 D   d a t a s e t[23] consists of 663 sequences of 20 gaming actions captured by Kinect. Each actor performed each gaming action more than two twice. Although the dataset can provide synchronized video, depth, and skeleton data, skeleton data is only chosen in our experiment. The dataset is challenging because of the following two aspects: (1) if the body parts are occluded, the Kinect device gives inferred results, which may reduce the accuracy of the action recognition. (2) if two different actions have very small interclass variations, the two actions may easily interference with each other in the action recognition.


## 3.1. Fundamental Concepts

In this subsection, a brief overview of the special Euclidean groupSE(3) and the special orthogonal group SO(3) is presented, which is necessary for further understanding of the Lie group. We refer the readers to [2, 10, 11] for a general introduction to Lie groups. Important notations are shown in Table 1.Table 1
Notations involving Lie groups.
Notation Description SO(3) Special orthogonal group  so(3) Lie algebra ofSO(3)  SO(3) Special Euclidean group  se(3) Lie algebra ofSE(3)  × Direct product of Lie group


### 3.1.1. Special Orthogonal Group

The special orthogonal groupSO(3) is a Lie group, which can be represented by all 3×3 orthogonal matrices shown as follows:(1)SO3=A∈R3×3;ATA=I3,detA=1where I3 denotes the 3×3 identity matrix and A is a rotation matrix. In 3D space, a rotation A is an element of SO(3) and can transform a vector x=x1,x2,x3T to x′ by(2)x′=AxEvery groupSO(3) has an associated Lie algebra of SO(3) that is the tangent space around the identity element I3. The Lie algebra of SO(3), denoted by so(3), is a set of all real 33 skew-symmetric matrices as follows:(3)so3=M∈R3×3;Mt=-M. Given an element(4)Ω=0-m3m2m30-m1-m2m10∈so3 its vector form vec(Ω) is(5)vecΩ=m1,m2,m3T The exponential map expSO(3) from so(3) to SO(3) and the logarithm map logSO(3) from SO(3) to so(3), respectively, are(6)expSO3Ω=eΩ;logSO3A=logA.


### 3.1.2. Special Euclidean Group

The special Euclidean group SE(3) is a Lie group, which is a set of 4 by 4 matrices(7)SE3=H∈R4×4,H=Ab01,A∈SO3,b∈R3. The matrix representation also makes SE(3) action on points c∈R3 by rotating and translating them:(8)c′=H·c=Ab01·c=Ac+b1.Every Lie Group SE(3) can be associated with a Lie algebra se(3), which is the tangent space of the Lie group SE(3) at the identity matrix I4. Note that se(3) is a 6D vector space that can be formed by 4 by 4 matrices(9)se3=Mv00,M∈R3×3,v∈R3,MT=M. Given an element(10)Ω=0-m3m2v1m30-m1v2-m2m10v30000∈se3, its vector form vec(Ω) is(11)vecΩ=m1,m2,m3,v1,v2,v3T. The exponential map expSE(3) from se(3) to SE(3) and the logarithm map logSE(3) from SE(3) to se(3), respectively, are(12)expSE3Ω=eΩ;logSE3H=logH.S O ( 3 ) × ⋯ × S O ( 3 ) and SE(3)×⋯×SE(3): the direct product × is used to combine multiple SO(3), which form a new Lie group K=SO(3)×⋯×SO(3) with identity element (I3,…,I3) and its Lie algebra k=so(3)×⋯×so(3). The exponential of (Ω1,…,ΩN)∈k and the logarithm map of (A1,…,AN)∈K, respectively, are given by(13)expKΩ1,…,ΩN=eΩ1,…,eΩN(14)logKA1,…,AN=logA1,…,logAN.The vector form of logK((A1,…,AN)) is(15)veclogKA1,…,AN=veclogA1,…,veclogAN. Similarly, a new Lie group K=SE(3)×⋯×SE(3) with identity element(I4,I4,…,I4) and its Lie algebra k=se(3)×se(3)×⋯×se(3) are formed by using the direct product ×. The exponential map of (Ω1,…,ΩN)∈k and the logarithm map of (H1,…,HN)∈K, respectively, are given by(16)expKΩ1,…,ΩN=eΩ1,…,eΩN(17)logKH1,…,HN=logH1,…,logHN. The vector form of logK((H1,…,HN)) is(18)veclogKH1,…,HN=veclogH1,…,veclogHN.


### 3.1.3. Explanation of Fundamental Concepts

According to the concepts described in Section3.2.1, the orientation of a rigid body is represented by six rotation matrices. Mathematically, a rotation matrix is a point in SO(3); therefore, the orientation of a rigid body in skeleton t can be represented as six points in SO(3). Then the static feature, composed of the orientations of the rigid bodies in the skeleton, is represented as a point in the Lie group (SO(3)×SO(3)⋯×SO(3)), as shown in Figure 1.The motions of a rigid body is generally regarded as its rotations and translations in 3D space. Mathematically, the rotations and translations of a rigid body are defined asSE(3); therefore, a rigid-body motion between skeletons t and t-1 (or skeletons t and 1) can be represented as a point in SE(3). Then, the dynamic feature, which is composed of the rigid-body motions between skeletons t and t-1 and those between skeletons t and 1, is represented as a point in the Lie group (SE(3)×SE(3)⋯×SE(3)), as shown in Figure 1. A skeletal representation, composed of the static feature and dynamic feature, can be represent as a point in the Lie group (SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3).The wavy surface represents a Lie group(SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3)⋯×SO(3)) in Figure 1. A whole circle in the wavy surface represents an action (skeleton sequence). Each black dot in the circle represents a skeleton. Then, an action can be represented as points in the Lie group (the points are included in the same circle). To overcome the classification difficulties, an action (or a whole circle) is mapped from the Lie group to its Lie algebra (se(3)×se(3)×⋯×se(3),so(3)×so(3)×⋯×so(3)), as shown in Figure 1. In fact, the Lie algebra is a vector space.


## 3.1.1. Special Orthogonal Group

The special orthogonal groupSO(3) is a Lie group, which can be represented by all 3×3 orthogonal matrices shown as follows:(1)SO3=A∈R3×3;ATA=I3,detA=1where I3 denotes the 3×3 identity matrix and A is a rotation matrix. In 3D space, a rotation A is an element of SO(3) and can transform a vector x=x1,x2,x3T to x′ by(2)x′=AxEvery groupSO(3) has an associated Lie algebra of SO(3) that is the tangent space around the identity element I3. The Lie algebra of SO(3), denoted by so(3), is a set of all real 33 skew-symmetric matrices as follows:(3)so3=M∈R3×3;Mt=-M. Given an element(4)Ω=0-m3m2m30-m1-m2m10∈so3 its vector form vec(Ω) is(5)vecΩ=m1,m2,m3T The exponential map expSO(3) from so(3) to SO(3) and the logarithm map logSO(3) from SO(3) to so(3), respectively, are(6)expSO3Ω=eΩ;logSO3A=logA.


## 3.1.2. Special Euclidean Group

The special Euclidean group SE(3) is a Lie group, which is a set of 4 by 4 matrices(7)SE3=H∈R4×4,H=Ab01,A∈SO3,b∈R3. The matrix representation also makes SE(3) action on points c∈R3 by rotating and translating them:(8)c′=H·c=Ab01·c=Ac+b1.Every Lie Group SE(3) can be associated with a Lie algebra se(3), which is the tangent space of the Lie group SE(3) at the identity matrix I4. Note that se(3) is a 6D vector space that can be formed by 4 by 4 matrices(9)se3=Mv00,M∈R3×3,v∈R3,MT=M. Given an element(10)Ω=0-m3m2v1m30-m1v2-m2m10v30000∈se3, its vector form vec(Ω) is(11)vecΩ=m1,m2,m3,v1,v2,v3T. The exponential map expSE(3) from se(3) to SE(3) and the logarithm map logSE(3) from SE(3) to se(3), respectively, are(12)expSE3Ω=eΩ;logSE3H=logH.S O ( 3 ) × ⋯ × S O ( 3 ) and SE(3)×⋯×SE(3): the direct product × is used to combine multiple SO(3), which form a new Lie group K=SO(3)×⋯×SO(3) with identity element (I3,…,I3) and its Lie algebra k=so(3)×⋯×so(3). The exponential of (Ω1,…,ΩN)∈k and the logarithm map of (A1,…,AN)∈K, respectively, are given by(13)expKΩ1,…,ΩN=eΩ1,…,eΩN(14)logKA1,…,AN=logA1,…,logAN.The vector form of logK((A1,…,AN)) is(15)veclogKA1,…,AN=veclogA1,…,veclogAN. Similarly, a new Lie group K=SE(3)×⋯×SE(3) with identity element(I4,I4,…,I4) and its Lie algebra k=se(3)×se(3)×⋯×se(3) are formed by using the direct product ×. The exponential map of (Ω1,…,ΩN)∈k and the logarithm map of (H1,…,HN)∈K, respectively, are given by(16)expKΩ1,…,ΩN=eΩ1,…,eΩN(17)logKH1,…,HN=logH1,…,logHN. The vector form of logK((H1,…,HN)) is(18)veclogKH1,…,HN=veclogH1,…,veclogHN.


## 3.1.3. Explanation of Fundamental Concepts

According to the concepts described in Section3.2.1, the orientation of a rigid body is represented by six rotation matrices. Mathematically, a rotation matrix is a point in SO(3); therefore, the orientation of a rigid body in skeleton t can be represented as six points in SO(3). Then the static feature, composed of the orientations of the rigid bodies in the skeleton, is represented as a point in the Lie group (SO(3)×SO(3)⋯×SO(3)), as shown in Figure 1.The motions of a rigid body is generally regarded as its rotations and translations in 3D space. Mathematically, the rotations and translations of a rigid body are defined asSE(3); therefore, a rigid-body motion between skeletons t and t-1 (or skeletons t and 1) can be represented as a point in SE(3). Then, the dynamic feature, which is composed of the rigid-body motions between skeletons t and t-1 and those between skeletons t and 1, is represented as a point in the Lie group (SE(3)×SE(3)⋯×SE(3)), as shown in Figure 1. A skeletal representation, composed of the static feature and dynamic feature, can be represent as a point in the Lie group (SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3).The wavy surface represents a Lie group(SE(3)×SE(3)⋯×SE(3),SO(3)×SO(3)⋯×SO(3)) in Figure 1. A whole circle in the wavy surface represents an action (skeleton sequence). Each black dot in the circle represents a skeleton. Then, an action can be represented as points in the Lie group (the points are included in the same circle). To overcome the classification difficulties, an action (or a whole circle) is mapped from the Lie group to its Lie algebra (se(3)×se(3)×⋯×se(3),so(3)×so(3)×⋯×so(3)), as shown in Figure 1. In fact, the Lie algebra is a vector space.


## 3.2. Extraction of Skeleton Features

In this subsection, the static and dynamic features of a skeleton are represented as a point in the Lie group. LetZ=(C,E) be a skeleton. The set of bone joints is denoted by C=c1,c2,…,cN, and the set of rigid-body parts is denoted by E=e1,e2,…,eM, where c,e∈R3. Figure 2(a) shows an example of the human skeleton with 19 rigid-body parts and 20 bone joints.Figure 2
(a) Human skeleton with 19 rigid body parts and 20 bone joints. (b) Placing the hip center to the origin and aligning the x-axis with the ground plane projection of the vector from the left hip to the right hip.
(a) (b)


### 3.2.1. Static Feature of Skeleton

By observing human action, the orientations of rigid bodies in a skeleton (pose) can include a lot of valuable information for action recognition. To describe the orientation of a given rigid bodyem, the global coordinate system is translated to the local coordinate system. Am,x, Am,y, and Am,z represent the three rotations that transform the rigid body em to the three coordinate axes, as shown in Figure 3(c). Their rotation relationship is shown as follows:(19)rx=Am,xem,ry=Am,yem,rz=Am,zem, where Am,x, Am,y, and Am,z∈SO(3). Similarly, Ax,m, Ay,m, and Az,m represent the three rotations that transform rx, ry, and rz to the rigid body em, respectively, as shown in Figure 3(d). Their rotation relationship is shown as follows:(20)em=Ax,mrx,em=Ay,mry,em=Az,mrz, where Ax,m, Ay,m, and Az,m∈SO(3). The six rotations can be used to describe the orientation of the rigid bodies.Figure 3
Representation of the orientation of rigid bodyem.
(a) (b) (c) (d)Given a skeletont, om(t)=Am,x(t),Ax,m(t),Am,y(t),Ay,m(t),Am,z(t),Az,m(t) is used to represent the orientation of em(t) in the skeleton. In this work, the skeletal staticfeature is defined as a set of the orientations of the rigid bodies in the skeleton as follows:(21)fot=o1t,o2t,…,oMt∈SO3×SO3×⋯×SO3, where M is the total number of rigid bodies in the human skeleton.


### 3.2.2. Dynamic Feature of Skeleton

Rigid-body motion is generally regarded as rotations and translations in 3D space. Mathematically, the rotations and translations of a rigid body can be denoted bySE(3). In this study, SE(3) is employed to describe rigid-body motions between different skeletons. Let em(i)∈R3 be rigid body em in skeletoni and em(j)∈R3 be rigid body em in skeletonj(i=j).Given a pointk(i)∈em(i) and k(j)∈em(j) corresponding to k(i), we have(22)kj1=Hmi,jki1=Ami,jbmi,j01ki1, where Hm(i,j)∈SE(3) and Am(i,j) and bm(i,j) are the rotation and translation, which can transform em(i) to the position and orientation of em(j), respectively, as shown in Figure 4(b).Figure 4
Representation of the relative geometries of rigid bodyem at time instance t.
(a) (b) (c)Similarly, given a pointk(j)∈em(j) and k(i)∈em(i) corresponding to k(j), we have(23)ki1=Hmj,ikj1=Amj,ibmj,i01kj1, where Hm(j,i)∈SE(3) and Am(j,i) and bm(j,i) are the rotation and translation, which can transform em(j) to the position and orientation of em(i), respectively, as shown in Figure 4(c). dm(i,j)=Hm(j,i),Hm(i,j) is used to represent the motion of rigid body em between skeletons i and j. Then, the rigid-body motions between skeletons i and j can be represented by(24)frgi,j=d1i,j,d2i,j,…,dMi,j∈SE3×SE3×⋯×SE3,where M is the total number of the rigid bodies in the skeleton.In this study, our approach only considers the rigid-body motions between skeletonst andt-1 and those between skeletonst and 1. According to formula (24), the rigid-body motions between skeletonst andt-1 can be represented by frg(t,t-1). Similarly, the rigid-body motions between skeletonst and 1 can be represented by frg(t,1). Then, the skeletal dynamicfeature is defined as a set of the rigid-body motions in skeleton t as follows:(25)fmft=frgt,t-1,frgt,1.Skeletont is represented by the static feature and the dynamic feature as follows:(26)ft=fot,fmft, where fo(t) represents the static feature of the skeleton and fmf(t) represents its dynamic feature.


## 3.2.1. Static Feature of Skeleton

By observing human action, the orientations of rigid bodies in a skeleton (pose) can include a lot of valuable information for action recognition. To describe the orientation of a given rigid bodyem, the global coordinate system is translated to the local coordinate system. Am,x, Am,y, and Am,z represent the three rotations that transform the rigid body em to the three coordinate axes, as shown in Figure 3(c). Their rotation relationship is shown as follows:(19)rx=Am,xem,ry=Am,yem,rz=Am,zem, where Am,x, Am,y, and Am,z∈SO(3). Similarly, Ax,m, Ay,m, and Az,m represent the three rotations that transform rx, ry, and rz to the rigid body em, respectively, as shown in Figure 3(d). Their rotation relationship is shown as follows:(20)em=Ax,mrx,em=Ay,mry,em=Az,mrz, where Ax,m, Ay,m, and Az,m∈SO(3). The six rotations can be used to describe the orientation of the rigid bodies.Figure 3
Representation of the orientation of rigid bodyem.
(a) (b) (c) (d)Given a skeletont, om(t)=Am,x(t),Ax,m(t),Am,y(t),Ay,m(t),Am,z(t),Az,m(t) is used to represent the orientation of em(t) in the skeleton. In this work, the skeletal staticfeature is defined as a set of the orientations of the rigid bodies in the skeleton as follows:(21)fot=o1t,o2t,…,oMt∈SO3×SO3×⋯×SO3, where M is the total number of rigid bodies in the human skeleton.


## 3.2.2. Dynamic Feature of Skeleton

Rigid-body motion is generally regarded as rotations and translations in 3D space. Mathematically, the rotations and translations of a rigid body can be denoted bySE(3). In this study, SE(3) is employed to describe rigid-body motions between different skeletons. Let em(i)∈R3 be rigid body em in skeletoni and em(j)∈R3 be rigid body em in skeletonj(i=j).Given a pointk(i)∈em(i) and k(j)∈em(j) corresponding to k(i), we have(22)kj1=Hmi,jki1=Ami,jbmi,j01ki1, where Hm(i,j)∈SE(3) and Am(i,j) and bm(i,j) are the rotation and translation, which can transform em(i) to the position and orientation of em(j), respectively, as shown in Figure 4(b).Figure 4
Representation of the relative geometries of rigid bodyem at time instance t.
(a) (b) (c)Similarly, given a pointk(j)∈em(j) and k(i)∈em(i) corresponding to k(j), we have(23)ki1=Hmj,ikj1=Amj,ibmj,i01kj1, where Hm(j,i)∈SE(3) and Am(j,i) and bm(j,i) are the rotation and translation, which can transform em(j) to the position and orientation of em(i), respectively, as shown in Figure 4(c). dm(i,j)=Hm(j,i),Hm(i,j) is used to represent the motion of rigid body em between skeletons i and j. Then, the rigid-body motions between skeletons i and j can be represented by(24)frgi,j=d1i,j,d2i,j,…,dMi,j∈SE3×SE3×⋯×SE3,where M is the total number of the rigid bodies in the skeleton.In this study, our approach only considers the rigid-body motions between skeletonst andt-1 and those between skeletonst and 1. According to formula (24), the rigid-body motions between skeletonst andt-1 can be represented by frg(t,t-1). Similarly, the rigid-body motions between skeletonst and 1 can be represented by frg(t,1). Then, the skeletal dynamicfeature is defined as a set of the rigid-body motions in skeleton t as follows:(25)fmft=frgt,t-1,frgt,1.Skeletont is represented by the static feature and the dynamic feature as follows:(26)ft=fot,fmft, where fo(t) represents the static feature of the skeleton and fmf(t) represents its dynamic feature.


## 3.3. Skeleton Sequence Representation

### 3.3.1. Lie Group Representation of Skeleton Sequence

Using the proposed skeletal feature, a skeleton sequence or an action is represented by(27)LG=ft,t∈1,T, where T is the total number of frames in the sequence and f(t)∈ Lie group SO(3)×⋯×SO(3),SE(3)×⋯×SE(3).


### 3.3.2. Lie Algebra Representation of Skeleton Sequence

Since the most classification methods (such as SVM) cannot be directly applied to manifolds, to overcome these difficulties,f(t) is mapped to its Lie algebra (so(3)×⋯×so(3),se(3)×⋯×se(3)).The Lie algebra of fo(t) is given by(28)got=veclog⁡o1t,veclog⁡o2t,…,veclog⁡oMt∈so3×so3×⋯×so3 and the Lie algebra of frg(i,j) is given by(29)grgi,j=veclog⁡d1i,j,veclog⁡d2i,j,…,veclog⁡dMi,j∈se3×se3×⋯×se3. The Lie algebra of f(t) is given by g(t)=go(t),gmf(t)=go(t),grg(t,t-1),grg(t,1). A human action can be represented by the following Lie algebra structure:(30)LA=gt,t∈1,T where T is the total number of frames in the sequence.M is the total number of rigid bodies in skeletont. Given a rigid body i in skeleton t, vec(log(oi(t)) is the Lie algebra’s representation of the orientation of the rigid body, which is 18-dimensional vector(i∈[1,M]). go(t)=vec(log(o1(t)),vec(log(o2(t)),…,vec(log(oM(t)) is the Lie algebra’s representation of the static feature of skeleton t, which is a (18×M)-dimensional vector. vec(log(di(p,q)) is the Lie algebra’s representation of the motions of rigid body i between skeletons p and q, which is 12-dimensional vector. grg(t,t-1)=vec(log(d1(t,t-1)),vec(log(d2(t,t-1)),…,vec(log(dM(t,t-1)) is the Lie algebra’s representation of the rigid-body motions between skeletons t and t-1, which is a (12×M)-dimensional vector. grg(t,1)=vec(log(d1(t,1)),vec(log(d2(t,1)),…,vec(log(dM(t,1)) is the Lie algebra’s representation of the rigid-body motions between skeletons t and 1, which is a (12×M)-dimensional vector. gmf(t)=grg(t,t-1),grg(t,1) is the Lie algebra’s representation of the dynamic feature of skeleton t, which is a (24M=12M+12M)-dimensional vector. g(t)=go(t),gmf(t) is the Lie algebra’s representation of skeletont, which is a (42M=18M+24M)-dimensional vector. Hence, a human action can be seen as temporal evolutions of a 42M-dimensional vector.


## 3.3.1. Lie Group Representation of Skeleton Sequence

Using the proposed skeletal feature, a skeleton sequence or an action is represented by(27)LG=ft,t∈1,T, where T is the total number of frames in the sequence and f(t)∈ Lie group SO(3)×⋯×SO(3),SE(3)×⋯×SE(3).


## 3.3.2. Lie Algebra Representation of Skeleton Sequence

Since the most classification methods (such as SVM) cannot be directly applied to manifolds, to overcome these difficulties,f(t) is mapped to its Lie algebra (so(3)×⋯×so(3),se(3)×⋯×se(3)).The Lie algebra of fo(t) is given by(28)got=veclog⁡o1t,veclog⁡o2t,…,veclog⁡oMt∈so3×so3×⋯×so3 and the Lie algebra of frg(i,j) is given by(29)grgi,j=veclog⁡d1i,j,veclog⁡d2i,j,…,veclog⁡dMi,j∈se3×se3×⋯×se3. The Lie algebra of f(t) is given by g(t)=go(t),gmf(t)=go(t),grg(t,t-1),grg(t,1). A human action can be represented by the following Lie algebra structure:(30)LA=gt,t∈1,T where T is the total number of frames in the sequence.M is the total number of rigid bodies in skeletont. Given a rigid body i in skeleton t, vec(log(oi(t)) is the Lie algebra’s representation of the orientation of the rigid body, which is 18-dimensional vector(i∈[1,M]). go(t)=vec(log(o1(t)),vec(log(o2(t)),…,vec(log(oM(t)) is the Lie algebra’s representation of the static feature of skeleton t, which is a (18×M)-dimensional vector. vec(log(di(p,q)) is the Lie algebra’s representation of the motions of rigid body i between skeletons p and q, which is 12-dimensional vector. grg(t,t-1)=vec(log(d1(t,t-1)),vec(log(d2(t,t-1)),…,vec(log(dM(t,t-1)) is the Lie algebra’s representation of the rigid-body motions between skeletons t and t-1, which is a (12×M)-dimensional vector. grg(t,1)=vec(log(d1(t,1)),vec(log(d2(t,1)),…,vec(log(dM(t,1)) is the Lie algebra’s representation of the rigid-body motions between skeletons t and 1, which is a (12×M)-dimensional vector. gmf(t)=grg(t,t-1),grg(t,1) is the Lie algebra’s representation of the dynamic feature of skeleton t, which is a (24M=12M+12M)-dimensional vector. g(t)=go(t),gmf(t) is the Lie algebra’s representation of skeletont, which is a (42M=18M+24M)-dimensional vector. Hence, a human action can be seen as temporal evolutions of a 42M-dimensional vector.


## 3.4. Key-Skeleton-Pattern Mining

In the previous subsections, a skeleton sequence is represented as the Lie algebra structureLA=g(t),t∈[1,T], where T is the total number of frames in a skeleton sequence. However, a skeleton sequence can include many noisy skeletons, which reduce the accuracy and efficiency of the action recognition. In this subsection, the key-skeleton-patterns are used to eliminate noisy skeletons in a skeleton sequence in order to capture the most informative skeleton sequences.


### 3.4.1. Formal Definitions

To mine the key-skeleton-patterns, classic k-means is used to quantize all skeletons represented by the Lie algebra to K symbol. LetS=s1,…,sK be a set containing K symbol and C=c1,…,cK be a set of centroid. Then, a skeleton sequence can be represented as a symbol sequence L=l1,…,lT(li∈S,i∈[1,T]). Since different skeletons may be quantized as the same symbol, to minimize the effect of quantization errors, each skeleton g(i) in a sequence is represented by probability pi, which is used to measure the distance between skeleton g(i)∈LA and centroid ci∈C as follows:(31)pi=distci,gi-1∑j=1Kdistcj,gi-10⩽pi⩽1,where cj∈C correspond to symbol sj∈S. Equation (31) shows the distance inversely proportional to pi. Now, a skeleton sequence also can be represented by a probability sequence P=p1,…,pT.Definitions. Some terms are defined in this paper as follows (Important notations are in Table 2.).Table 2
Notations involving data mining.
Notation Description α Skeleton pattern  d Mining sequence  D|α Projected dataset  SUP(α,D|α) Support of skeleton patternα  ES(α,D|α) Expected support of skeleton patternαDefinition 1 (pattern).
α = α ( 1 ) , … , α ( m ) is a sequence that contains m symbols chosen from the dictionary, i.e., α(i)∈S.Definition 2 (mining sequence).
d = P , L , L A is a mining sequence applied to mine the Key-skeleton sequence. P is a probability sequence, which represents a skeletons sequence. L is the symbol sequence, which corresponds to P. LA is a skeleton sequence represented by the Lie algebraic structure.Definition 3 (projected dataset).
Given a patternα and a mining sequence dataset D(d∈D) of an action class, the α-projected dataset D|α is defined by the setd|d∈D∧α⊆d.L.Definition 4 (support).
For a patternα and a symbol sequence d.L∈D|α (where d is an element of D|α), let F(α,d.L) be an indicator variable with value 1 if α is a subsequence of the symbol sequence d.L, and 0 otherwise. For any pattern α, its support in D|α is denoted by SUP(α,D|α)=∑i=1D|α‍F(α,d.L)Definition 5 (expected support).
Given a patternα and a symbol sequence d.L (where d is an element of D|α), let pos=Find(α,d.L) be the positions where the pattern α takes up in d.L and let d.P(α)=∏j=1pos‍d.P[pos[j]] be the product of the probability. For any pattern α, its expected support in D|α is denoted by ES(α,D|α)=∑i=1D|α‍d.P(α)/D|α.Definition 6 (key-skeleton pattern).
Given a patternα and a mining sequence dataset D of an action class, if SUP(α,D|α) is larger than a threshold τsup and ES(α,D|α) is larger than a threshold τprob, then the pattern α is called a key-skeleton-pattern of that action class. A key-skeleton-pattern of length i is called an i-pattern.


### 3.4.2. MinP-PrefixSpan Algorithm

In this subsection, a new pattern-growth algorithm, called MinP-PrefixSpan, is proposed to mine the key-skeleton-patterns of an action class by searching over the enormous space of the symbol sequences and probability sequences of the action class. The algorithm is shown in Algorithm1. In Lines 2-8, the dataset D|α is employed to construct a new projected dataset D|αe. In Lines 9-11, if αe is the key-skeleton-pattern, pattern αe is appended to K and symbol table T|αe is constructed by Trim(T|α,D|αe). In Line 12, MinP-PrefixSpan is recursively called to grow the key-skeleton-pattern until all key-sequence patterns are found.Algorithm 1:MinP-PrefixSpan(T|α,D|α,K).
Input:  α-projected dataset Dα, symbol table T|α, key-skeleton-pattern dataset K (1)  for each symbol e∈T|α do (2)   D|αe←∅ (3)   for each d∈D|α do (4)    pos=Find(α,d.L) (5)    ife∈d.L[pos[α]+1,⋯,d.L] then (6)     Append d to D|αe (7)    end if (8)   end for (9)   if SUP(αe,D|αe)≥τsup and ES(αe,D|αe)≥τes then (10)   T|αe←Trim(T|α,D|αe) (11)   Append αe to K (12)   MinP-PrefixSpan(T|αe,D|αe,K) (13)  end if (14)  Free D|αe and T|αe (15) end forIn Line 10, the trim algorithm is used to improve the efficiency of the MinP-PrefixSpan algorithm by eliminating nonkey-skeleton-patterns. Algorithm2 shows the implementation details of the trim algorithm. The trim algorithm mainly consists of the following two parts:Algorithm 2:Trim(T|α,D|αe).
Input:symbol table T|α, αe-projected dataset Dαe Output: symbol table T|αe (1) T|αe←∅ (2) for each symbol m∈T|α do (3)  if SUP(αem,D|αe)≥τsup and ES(αem,D|αe)≥τes then (4)   T|αe←T|αe∪m (5)  end if (6) end forT r i m i n g   r u l e s . Given a mining sequence dataset of an action class D and a pattern α (according to Definition 3, D|α is the α-projected dataset), two rules are proposed to trim nonkey-skeleton-patterns as follows:(1)
IfSUP(α,D|α)≤τsup, then pattern α is a non-key-skeleton-pattern.(2)
IfES(α,D|α)≤τes, then pattern α is a non-key-skeleton-pattern.P a t t e r n   g r o w t h . Referring to the pattern-growth method of Prefxispan, one symbol e is used to grow key-skeleton-pattern α and check the support and expected support of the pattern αe. A symbol table T|α is used to store each symbol e in order to reduce the number of new skeleton patterns in each growth step. An important property is found between symbol tables.Property 7 (symbol table).
If a key-skeleton-patternγ grows from α, then T|γ⊆T|α.Proof.
Let us denoteγ=αe(1) as the key-skeleton-pattern and e(1)∈T|α. Suppose e(2)∈T|γ and β=αe(1)e(2) also is a key-skeleton-pattern on D|α. Since SUP(αe(2),D|α)≥SUP(β,D|α) according to Definition 4, SUP(αe(2),D|α)≥τsup. Since ES(β,D|α)-ES(αe(2),D|α)≥0 according to Definition 5, ES(αe(2),D|α)≥τes. We conclude that e(2) belongs to T|α,which implies that T|γ⊆T|α.


## 3.4.1. Formal Definitions

To mine the key-skeleton-patterns, classic k-means is used to quantize all skeletons represented by the Lie algebra to K symbol. LetS=s1,…,sK be a set containing K symbol and C=c1,…,cK be a set of centroid. Then, a skeleton sequence can be represented as a symbol sequence L=l1,…,lT(li∈S,i∈[1,T]). Since different skeletons may be quantized as the same symbol, to minimize the effect of quantization errors, each skeleton g(i) in a sequence is represented by probability pi, which is used to measure the distance between skeleton g(i)∈LA and centroid ci∈C as follows:(31)pi=distci,gi-1∑j=1Kdistcj,gi-10⩽pi⩽1,where cj∈C correspond to symbol sj∈S. Equation (31) shows the distance inversely proportional to pi. Now, a skeleton sequence also can be represented by a probability sequence P=p1,…,pT.Definitions. Some terms are defined in this paper as follows (Important notations are in Table 2.).Table 2
Notations involving data mining.
Notation Description α Skeleton pattern  d Mining sequence  D|α Projected dataset  SUP(α,D|α) Support of skeleton patternα  ES(α,D|α) Expected support of skeleton patternαDefinition 1 (pattern).
α = α ( 1 ) , … , α ( m ) is a sequence that contains m symbols chosen from the dictionary, i.e., α(i)∈S.Definition 2 (mining sequence).
d = P , L , L A is a mining sequence applied to mine the Key-skeleton sequence. P is a probability sequence, which represents a skeletons sequence. L is the symbol sequence, which corresponds to P. LA is a skeleton sequence represented by the Lie algebraic structure.Definition 3 (projected dataset).
Given a patternα and a mining sequence dataset D(d∈D) of an action class, the α-projected dataset D|α is defined by the setd|d∈D∧α⊆d.L.Definition 4 (support).
For a patternα and a symbol sequence d.L∈D|α (where d is an element of D|α), let F(α,d.L) be an indicator variable with value 1 if α is a subsequence of the symbol sequence d.L, and 0 otherwise. For any pattern α, its support in D|α is denoted by SUP(α,D|α)=∑i=1D|α‍F(α,d.L)Definition 5 (expected support).
Given a patternα and a symbol sequence d.L (where d is an element of D|α), let pos=Find(α,d.L) be the positions where the pattern α takes up in d.L and let d.P(α)=∏j=1pos‍d.P[pos[j]] be the product of the probability. For any pattern α, its expected support in D|α is denoted by ES(α,D|α)=∑i=1D|α‍d.P(α)/D|α.Definition 6 (key-skeleton pattern).
Given a patternα and a mining sequence dataset D of an action class, if SUP(α,D|α) is larger than a threshold τsup and ES(α,D|α) is larger than a threshold τprob, then the pattern α is called a key-skeleton-pattern of that action class. A key-skeleton-pattern of length i is called an i-pattern.


## 3.4.2. MinP-PrefixSpan Algorithm

In this subsection, a new pattern-growth algorithm, called MinP-PrefixSpan, is proposed to mine the key-skeleton-patterns of an action class by searching over the enormous space of the symbol sequences and probability sequences of the action class. The algorithm is shown in Algorithm1. In Lines 2-8, the dataset D|α is employed to construct a new projected dataset D|αe. In Lines 9-11, if αe is the key-skeleton-pattern, pattern αe is appended to K and symbol table T|αe is constructed by Trim(T|α,D|αe). In Line 12, MinP-PrefixSpan is recursively called to grow the key-skeleton-pattern until all key-sequence patterns are found.Algorithm 1:MinP-PrefixSpan(T|α,D|α,K).
Input:  α-projected dataset Dα, symbol table T|α, key-skeleton-pattern dataset K (1)  for each symbol e∈T|α do (2)   D|αe←∅ (3)   for each d∈D|α do (4)    pos=Find(α,d.L) (5)    ife∈d.L[pos[α]+1,⋯,d.L] then (6)     Append d to D|αe (7)    end if (8)   end for (9)   if SUP(αe,D|αe)≥τsup and ES(αe,D|αe)≥τes then (10)   T|αe←Trim(T|α,D|αe) (11)   Append αe to K (12)   MinP-PrefixSpan(T|αe,D|αe,K) (13)  end if (14)  Free D|αe and T|αe (15) end forIn Line 10, the trim algorithm is used to improve the efficiency of the MinP-PrefixSpan algorithm by eliminating nonkey-skeleton-patterns. Algorithm2 shows the implementation details of the trim algorithm. The trim algorithm mainly consists of the following two parts:Algorithm 2:Trim(T|α,D|αe).
Input:symbol table T|α, αe-projected dataset Dαe Output: symbol table T|αe (1) T|αe←∅ (2) for each symbol m∈T|α do (3)  if SUP(αem,D|αe)≥τsup and ES(αem,D|αe)≥τes then (4)   T|αe←T|αe∪m (5)  end if (6) end forT r i m i n g   r u l e s . Given a mining sequence dataset of an action class D and a pattern α (according to Definition 3, D|α is the α-projected dataset), two rules are proposed to trim nonkey-skeleton-patterns as follows:(1)
IfSUP(α,D|α)≤τsup, then pattern α is a non-key-skeleton-pattern.(2)
IfES(α,D|α)≤τes, then pattern α is a non-key-skeleton-pattern.P a t t e r n   g r o w t h . Referring to the pattern-growth method of Prefxispan, one symbol e is used to grow key-skeleton-pattern α and check the support and expected support of the pattern αe. A symbol table T|α is used to store each symbol e in order to reduce the number of new skeleton patterns in each growth step. An important property is found between symbol tables.Property 7 (symbol table).
If a key-skeleton-patternγ grows from α, then T|γ⊆T|α.Proof.
Let us denoteγ=αe(1) as the key-skeleton-pattern and e(1)∈T|α. Suppose e(2)∈T|γ and β=αe(1)e(2) also is a key-skeleton-pattern on D|α. Since SUP(αe(2),D|α)≥SUP(β,D|α) according to Definition 4, SUP(αe(2),D|α)≥τsup. Since ES(β,D|α)-ES(αe(2),D|α)≥0 according to Definition 5, ES(αe(2),D|α)≥τes. We conclude that e(2) belongs to T|α,which implies that T|γ⊆T|α.


## 3.5. Discovering the Most Informative Skeleton Sequence

The task of Algorithm3 is to discover the most informative skeleton sequences for all actions. Let J be a mining sequence dataset of all actions. K is a dataset used to store the key-skeleton-patterns of all action classes, and Q is a dataset used to store the most informative skeleton sequences of all actions. In Lines 2-11, the key-skeleton-patterns of each class action are mined from training dataset U and appended to dataset K. In Lines 12-22, the key-skeleton-patterns in dataset K are employed to discover the most informative skeleton sequence of the actions in dataset J, and all most informative skeleton sequences are stored in dataset Q(refer to Figure 5).Algorithm 3:Informative skeleton sequence (K,J).
Input:key-skeleton-pattern dataset K, mining sequence dataset of all actions J Output: The dataset of the most informative skeleton sequences Q (1)  Obtaining training dataset U from J (2)  K←∅; Q←∅ (3)  for the dataset of each action class D⊂ U do (4)   T is the table that includes all symbols in D (5)   for each symbol α∈T do (6)    if SUP(α,D|α)≥τsup and ES(α,D|α)≥τes then (7)     T|α←Trim(T,D|α) (8)     append α to K (9)     MinP-PrefixSpan(T|α,D|α,K) (10)   end if (11)  end for (12) end for (13) for each element d∈J do (14)  for each key-skeleton-pattern m∈K  do (15)   if m is a subsequence of d.L then (16)    patternpos=zero(len(d.L),1) (17)    position=Discoverlocation(m,d.L) (18)    for i∈position do (19)     patternpos[i]=1 (20)    end for (21)   end if (22)  end for (23)  for z=1 to Len(d.L) do (24)   if patternpos[i]==1 then (25)    append d.LA[z] to s (26)   end if (27)  end for (28)  Append s to Q (29) end for (30) return QFigure 5
Discovering most informative skeleton sequence.Dynamic Time Warping (DTW)[33] has excellent performance in searching for anoptimal alignment between time sequences. Therefore, for each action class, our model uses the action standardization algorithm proposed by the author of [5] to compute a nominal action and employs DTW to warp all the training or testing actions into this nominal action. SVMs are extensively used in computer vision to achieve excellent performances in image and video classifications. To achieve better classification results, a linear SVM is used to classify the most informative skeleton sequences.


## 3.6. Datasets

In this study, three standard 3D human action datasets are employed to study the effectiveness of the proposed method.M S R A c t i o n 3 D   d a t a s e t [34] can be captured using a depth camera similar to the Kinect device. This dataset consists of 20 actions of 10 subjects, with each action having two or three repetitions. In total, there are 557 action sequences. The dataset provides 3D locations of 20 joints. The horizontal and vertical locations of each skeleton joint are stored in the screen coordinates, and the skeleton’s depth position is stored in the global coordinates. Human actions in this dataset capture various types of motions, which are related to arms, legs, torso, and their combinations. Experiments on this dataset are challenging, but the dataset is widely applied to test the accuracy and robustness of recognition methods for various actions.U C K i n e c t - A c t i o n   d a t a s e t [19] is captured using a stationary Kinect sensor. It consists of 10 human actions obtained from daily life: walking, sitting down, standing up, picking up, carrying, throwing, pushing, pulling, waving, and clapping hands. Each human action is performed by 10 different subjects (nine males and one female) twice or thrice. In total, there are 199 action sequences. This dataset is very challenging. First, for some action sequences, parts of the human body are invisible because the body parts are out of the field of view. Second, subjects performed the same action using different limbs, such as waving the left hand and waving the right hand. Third, it is very difficult to capture the action sequences with invariance to the view point.G 3 D   d a t a s e t[23] consists of 663 sequences of 20 gaming actions captured by Kinect. Each actor performed each gaming action more than two twice. Although the dataset can provide synchronized video, depth, and skeleton data, skeleton data is only chosen in our experiment. The dataset is challenging because of the following two aspects: (1) if the body parts are occluded, the Kinect device gives inferred results, which may reduce the accuracy of the action recognition. (2) if two different actions have very small interclass variations, the two actions may easily interference with each other in the action recognition.


## 4. Experimental Results

The skeleton preprocess is as follows: a human action is composed of a continuous evolution of a series of skeletons. To make each skeleton view-invariant, all 3D joint coordinates in the skeleton are transformed to the coordinate system, which places the hip center at the origin. The entire skeleton will stop rotation until the global x-axis is aligned with the ground plane projection of the vector from the left hip to the right hip (refer to Figure2(b)).


### 4.1. Experiments on the MSRAction3D Dataset

Following the experimental protocol of [4], 20 actions in the MSRAction3D dataset are divided into three subsets AS1, AS2, and AS3, each including eight actions. AS1 and AS2 include actions with similar movement. AS3 groups include more complex actions. A half of the subjects are chosen for training, and the remaining subjects for testing. The experiment is run on ten different combinations of training and testing sets, and the mean performance is reported. Figure 6 shows that our approach outperforms various other representations. Our approach achieves a mean accuracy of 94.58% on the MSRAction3D dataset, outperforming other action recognition approaches, including Bag of 3D Points [4], Eigenjoints [29], and Lie group [5], which achieved accuracies of 74.7, 83.3, and 91.88%, respectively. Our approach performs better than the others both in distinguishing similar actions and in recognizing complex actions. This is mainly because the informative skeleton sequences, represented by the Lie group, are used to train SVM classifiers.Figure 6
Recognition rate on the MSRAction3D dataset based on AS1, AS2, and AS3.Following the experimental protocol of [16], the dataset containing all actions is tested. The experimental setting is more challenging than that of [4]. Our approach achieves an accuracy of 97.4%, outperforming other relative action recognition approaches, including Grassmann manifold [16], graph-based model [20], and Lie group [5], which achieved accuracies of 91.21, 92.2, and 92.46%, respectively, as shown in Table 3.Table 3
Recognition rate on the MSRAction3D dataset based on the experimental protocol of [16].
Approach Grassmann manifold[16] Graph-based model[20] Lie Group[5] Our approach Accuracy 91.21 92.2 92.46 97.4Figure7 shows the classification confusion matrix on the whole MSR-Action3D dataset. Most actions on the dataset can be correctly recognized by our approach, but classification errors occurred if two actions were extremely similar, such asdraw tick anddraw  X.Figure 7
The confusion matrix for action classification in entire the MSR-Action3D dataset.Matlab is used to run the experiments on a 3.60GHz Intel Core i7-4790 CPU machine. The average testing time of one action sequence in the dataset only costs 35.1ms, which is lower than that of Lie group(72.5ms).The reason is that the skeleton feature dimension of our approach(798-dimension) is lower than that of Lie Group(2052-dimension). However, since the authors of Grassmann Manifold and Graph-based model do not open the source code of their approaches, the average testing time of their approaches cannot be obtained.


### 4.2. Experiments on the UCKinect-Action Dataset

The recognition rate of our approach on the UCKinect Dataset is 98.87%. Our method outperforms the Lie group[5], Grassmann manifold[16], Eigenjoints[29], and learning feature combination[30], which achieved recognition rates of 97.08, 97.91,97.1, and 98.00%, respectively, as shown in Table 4.Table 4
Recognition rate on the UCKinect-Action dataset.
Approach Recognition accuracy Lie group[5] 97.08  Grassmann manifold[16] 97.91  Eigenjoints[29] 97.1  Learning feature combination[30] 98.00  Our approach 98.87The average testing time of one action sequence in the dataset costs 33.6ms, which is lower than that of Lie group (69.2ms) but higher than that of Learning features combination (13.7ms). The reason is that the skeleton feature dimension of our approach (798-dimension) is lower than that of Lie Group (2052-dimension) but higher than that Learning features combination (256-dimension). Unfortunately, the average testing time of Grassmann Manifold and Eigenjoints cannot be obtained without the source code of the approaches.


### 4.3. Experiments on the G3D-Gaming Dataset

The cross-subject test setting, in which half of subjects were used for training and the remaining subjects were used for testing, is used to perform recognition on the data. Table5 compares our approach with other approaches on the dataset (GB-RBM+HMM [21] and LieNet [17] used a deep-learning method to recognize human action). Our approach achieves a higher recognition rate.Table 5
Recognition rate on the Florence3D-Action dataset.
Approach Recognition accuracy SO(3)[14] 87.95  Lie group[5] 91.09  GB-RBM+HMM[31] 86.40  LieNet[22] 89.10  tLDS[32] 90.60  Our approach 95.01The average testing time of one action sequence in the dataset only costs 34.8ms, which is lower than that of Lie group(71.2ms) andSO(3)(58.9ms). The reason is that the skeleton feature dimension of our approach (798-dimension) is lower than that of Lie Group(2052-dimension) and SO(3)(1026-dimension). However, the authors of tLDS do not open the source code of their approach, the average testing time of their approach cannot be obtained. Since the deep learning-based approaches usually use GPU to accelerate their models while the nondeep learning-based approaches usually use CPU to perform their experiments, it is hard to implement a fair comparison between these two classes of approaches (our approach belongs to the nondeep learning-based approaches, and LieNet and GB-RBM+HMM belong to the deep learning-based approaches).


## 4.1. Experiments on the MSRAction3D Dataset

Following the experimental protocol of [4], 20 actions in the MSRAction3D dataset are divided into three subsets AS1, AS2, and AS3, each including eight actions. AS1 and AS2 include actions with similar movement. AS3 groups include more complex actions. A half of the subjects are chosen for training, and the remaining subjects for testing. The experiment is run on ten different combinations of training and testing sets, and the mean performance is reported. Figure 6 shows that our approach outperforms various other representations. Our approach achieves a mean accuracy of 94.58% on the MSRAction3D dataset, outperforming other action recognition approaches, including Bag of 3D Points [4], Eigenjoints [29], and Lie group [5], which achieved accuracies of 74.7, 83.3, and 91.88%, respectively. Our approach performs better than the others both in distinguishing similar actions and in recognizing complex actions. This is mainly because the informative skeleton sequences, represented by the Lie group, are used to train SVM classifiers.Figure 6
Recognition rate on the MSRAction3D dataset based on AS1, AS2, and AS3.Following the experimental protocol of [16], the dataset containing all actions is tested. The experimental setting is more challenging than that of [4]. Our approach achieves an accuracy of 97.4%, outperforming other relative action recognition approaches, including Grassmann manifold [16], graph-based model [20], and Lie group [5], which achieved accuracies of 91.21, 92.2, and 92.46%, respectively, as shown in Table 3.Table 3
Recognition rate on the MSRAction3D dataset based on the experimental protocol of [16].
Approach Grassmann manifold[16] Graph-based model[20] Lie Group[5] Our approach Accuracy 91.21 92.2 92.46 97.4Figure7 shows the classification confusion matrix on the whole MSR-Action3D dataset. Most actions on the dataset can be correctly recognized by our approach, but classification errors occurred if two actions were extremely similar, such asdraw tick anddraw  X.Figure 7
The confusion matrix for action classification in entire the MSR-Action3D dataset.Matlab is used to run the experiments on a 3.60GHz Intel Core i7-4790 CPU machine. The average testing time of one action sequence in the dataset only costs 35.1ms, which is lower than that of Lie group(72.5ms).The reason is that the skeleton feature dimension of our approach(798-dimension) is lower than that of Lie Group(2052-dimension). However, since the authors of Grassmann Manifold and Graph-based model do not open the source code of their approaches, the average testing time of their approaches cannot be obtained.


## 4.2. Experiments on the UCKinect-Action Dataset

The recognition rate of our approach on the UCKinect Dataset is 98.87%. Our method outperforms the Lie group[5], Grassmann manifold[16], Eigenjoints[29], and learning feature combination[30], which achieved recognition rates of 97.08, 97.91,97.1, and 98.00%, respectively, as shown in Table 4.Table 4
Recognition rate on the UCKinect-Action dataset.
Approach Recognition accuracy Lie group[5] 97.08  Grassmann manifold[16] 97.91  Eigenjoints[29] 97.1  Learning feature combination[30] 98.00  Our approach 98.87The average testing time of one action sequence in the dataset costs 33.6ms, which is lower than that of Lie group (69.2ms) but higher than that of Learning features combination (13.7ms). The reason is that the skeleton feature dimension of our approach (798-dimension) is lower than that of Lie Group (2052-dimension) but higher than that Learning features combination (256-dimension). Unfortunately, the average testing time of Grassmann Manifold and Eigenjoints cannot be obtained without the source code of the approaches.


## 4.3. Experiments on the G3D-Gaming Dataset

The cross-subject test setting, in which half of subjects were used for training and the remaining subjects were used for testing, is used to perform recognition on the data. Table5 compares our approach with other approaches on the dataset (GB-RBM+HMM [21] and LieNet [17] used a deep-learning method to recognize human action). Our approach achieves a higher recognition rate.Table 5
Recognition rate on the Florence3D-Action dataset.
Approach Recognition accuracy SO(3)[14] 87.95  Lie group[5] 91.09  GB-RBM+HMM[31] 86.40  LieNet[22] 89.10  tLDS[32] 90.60  Our approach 95.01The average testing time of one action sequence in the dataset only costs 34.8ms, which is lower than that of Lie group(71.2ms) andSO(3)(58.9ms). The reason is that the skeleton feature dimension of our approach (798-dimension) is lower than that of Lie Group(2052-dimension) and SO(3)(1026-dimension). However, the authors of tLDS do not open the source code of their approach, the average testing time of their approach cannot be obtained. Since the deep learning-based approaches usually use GPU to accelerate their models while the nondeep learning-based approaches usually use CPU to perform their experiments, it is hard to implement a fair comparison between these two classes of approaches (our approach belongs to the nondeep learning-based approaches, and LieNet and GB-RBM+HMM belong to the deep learning-based approaches).


## 5. Conclusion and Future Work

This paper proposes a new skeleton-based action representation, which consists of static and dynamic features. First, the orientation of a rigid body is regarded as six rotation matrices, and each rotation matrix is represented as a point inSO(3). The rigid-body orientations in a skeleton are used to construct the static feature in order to avoid dealing with skeletal scale variations. Second, the motions of rigid bodies, represented by SE(3), are used to construct the dynamic features in order to capture the temporal information of the skeleton. Finally, based on the proposed representation, the key-skeleton-patterns are employed to discover the most informative skeleton sequences. The experiment results show that our approach achieves better performance than other state-of-the-art skeleton-based action recognition approaches. Further research should combine the Lie group with a linear dynamical system to model human actions as a tensor time series.



---

*Source: 7952974-2018-12-05.xml*