# Nonnegative Tensor-Based Linear Dynamical Systems for Recognizing Human Action from 3D Skeletons

Authors: Guang Li; Kai Liu; Wenwen Ding; Fei Cheng; Chongyang Ding
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

Recently, skeleton-based action recognition has become a very important topic in the field of computer vision. It is a challenging task to accurately build a human action model and precisely distinguish similar human actions. In this paper, an action (skeleton sequence) is represented as a third-order nonnegative tensor time series to capture the original spatiotemporal information of the action. As a linear dynamical system (LDS) is an efficient tool for encoding the spatiotemporal data in various disciplines, this paper proposes a nonnegative tensor-based LDS (nLDS) to model the third-order nonnegative tensor time series. Nonnegative Tucker decomposition (NTD) is utilized to estimate the parameters of the nLDS model. These parameters are used to build extended observability sequenceO∞T for the action, which implies that O∞T can be considered as the feature descriptor of the action. To avoid the limitations introduced by approximating O∞T with a finite-order matrix, we represent an action as a point on infinite Grassmann manifold comprising the orthonormalized extended observability sequences. The classification task can be performed by dictionary learning and sparse coding on the infinite Grassmann manifold. The experimental results on the MSR-Action3D, UTKinect-Action, and G3D-Gaming datasets demonstrate that the proposed approach achieves a better performance in comparison with the state-of-the-art methods.


---

## Body

## 1. Introduction

Human action recognition based on spatiotemporal data has been one of the most prominent research topics owing to its applications in human-computer interfaces [1], gaming [2], and surveillance systems [3]. Over the past few decades, numerous methods have been proposed for recognizing human actions from monocular RGB videos [4]. However, the monocular RGB data is very sensitive to background clutter, occlusion, variations in the view-point, and illumination changes. Thus, although in the past few decades several significant research studies have been conducted, the method to accurately recognize human actions from RGB videos still remains a challenging problem. As a human skeleton can be viewed as an articulated system of rigid bodies connected by bone joints, a human action can be described as the spatiotemporal evolution of a series of skeletons. Therefore, if human skeleton sequences can be accurately extracted from RGB videos, it is possible to perform action recognition by classifying the skeleton sequences. However, it is an extremely difficult task to reliably extract the human skeleton from monocular video sensors. With the development of cost-effective depth sensors [5], it has become easier to extract the three-dimensional (3D) positions of skeletal joints from depth data. Hence, skeleton-based action recognition has once again become an active area of research.When recognizing human actions, the action representations embodying the temporal dynamics can provide a more relevant description than using static data [6]. A linear dynamical system (LDS) [7] is an effective tool in various disciplines for capturing the spatiotemporal data. Hence, the authors of [6, 8] employed an LDS model to capture the spatiotemporal information of an action (skeleton sequence) and used the singular value decomposition (SVD) [9] or Tucker decomposition [10] to estimate model parameters (A,C). The parameters were used to build finite observability matrix OkT=[CT,(CA)T,(CA2)T,…,(CAk-1)T]. Hence, an action, represented by an LDS, was alternately identified as a point on the finite Grassmann manifold corresponding to the column space of OkT.Motivated by the above methods, this paper proposes a novel approach to model and analyze human actions. The overall approach is shown in Figure1. In this study, an action (a skeleton sequence) is represented as a third-order nonnegative tensor and each skeleton is converted into a second-order nonnegative tensor. To retain the spatiotemporal information of an action to the maximum extent, a nonnegative tensor-based LDS (nLDS) is proposed to model the action. In this work, nonnegative Tucker decomposition (NTD) [11] is used to decompose the third-order nonnegative tensor for improving the accuracy of action recognition. Because the NTD is a powerful tool to extract a part-based representation of high-dimensional tensors, an action can be represented by a linear combination of relevant components.Figure 1
Complete framework of the proposed method.The parameter tuple(A,C) can be learned by the nLDS model and NTD of an action. The parameters C and A represent the appearance and dynamics of the nLDS model, respectively. Thus, it is appropriate to regard extended observability sequence O∞T=[CT,(CA2)T,…,(CAn)T,…] as the feature descriptor of the action. The conventional method approximates O∞T as a finite-order matrix OmT and maps OmT to a point on a finite Grassmann manifold. However, the value of order m can affect the asymptotic behavior of O∞T and computational complexity. To avoid the limitations introduced by choosing the value of order m, an action is represented as a point on infinite Grassmann manifold S∞T consisting of the orthonormalized extended observability sequences. Finally, classification is performed using dictionary learning and sparse coding on the infinite Grassmann manifold.The main contributions of this study are as follows:(1) To retain the spatiotemporal information of an action, an nLDS is used to model the action that is represented by a third-order nonnegative tensor.(2) Compared with the Tucker decomposition, the uniqueness of NTD allows the representation of an action to be more discriminative. Thus, to further improve the accuracy of action recognition, NTD is used to estimate the parameters of the nLDS model. The parameters are utilized to build an extended observability sequenceO∞T=[CT,(CA2)T…,(CAn)T,…] that can be considered as the feature descriptor of the action.(3) To overcome the limitation caused by approximating extended observability sequenceO∞T with finite-order matrix OkT, an action is represented as a point on infinite Grassmann manifold S∞T consisting of the orthonormalized extended observability sequences.The rest of the paper is organized as follows: Section2 reviews related work, Section 3 briefly introduces some fundamental concepts on the Tucker model, NTD, and LDS. Section 4 elaborates the nLDS model and describes how to represent an action as a point on the infinite Grassmann manifold. Section 5 presents our experimental results. Section 6 concludes this paper.


## 2. Related Work

A brief overview of the skeleton-based action recognition approaches is provided in this section. Presently, the existing skeleton-based action recognition approaches can be categorized into two types; the first type represents the human skeleton as a set of skeletal joints. Wang et al. [22] employed pairwise relative positions of the joints to represent a human skeleton and used a hierarchy of Fourier coefficients to model the temporal evolution of this representation. To obtain the discriminative joint combinations, they used a multiple kernel learning approach to characterize the human actions. Li et al. [23] used a graph-based model to represent the relative spatial variations between various skeletal joints. They utilized the relative variance of the joint relative distance (RVJRD) [24] to indicate their activity levels in a human action to obtain the most informative skeletal joint pairs. To derive the spatiotemporal compatibility of the skeletal joints between different actions, Koniusz et al. [25] used a sequence compatibility kernel (SCK) to capture the spatial and temporal similarities between the skeletons in an action. Furthermore, they employed a dynamics compatibility kernel to represent the similarity between a pair of skeletons in a given action to capture the spatiotemporal dynamics of that action. Zhu et al. [26] input the raw 3D skeletal joint locations to an end-to-end fully connected deep long short-term memory (LSTM) network for recognizing skeleton-based human actions. Ding et al. [6] represented a human skeleton as a 3D joint-based tensor, and then a human action was considered as a third-order tensor time series. They proposed a tensor-based LDS (tLDS) to model a tensor time series and estimated the parameters of the LDS model using the Tucker decomposition. To eliminate the noise and occlusion in 3D skeleton data, Liu et al. [17] proposed a spatiotemporal long short-term memory (ST-LSTM) network with trust gates. By analyzing the reliability of the skeleton data, trust gates could dynamically update the long-term context information stored in the memory cell. Lee et al. [14] built an Ensemble Temporal Sliding LSTM (TS-LSTM) network which was composed of short-term, medium-term, and long-term TS-LSTM networks. The subnetworks could capture the temporal dependencies between skeletons and the spatial dependency of each skeleton. The second type of skeleton-based action recognition approaches represents a human skeleton as a set of connected rigid bodies. The authors of [12] represented human actions as curves in the Lie group. For the easier performance of the classification task, the human actions represented by the Lie group were mapped to its Lie algebra in the vector space. In [8], the authors divided a skeleton into smaller body parts and employed certain bioinspired shape features to represent each body part. An LDS was used to learn the temporal evolution of the bioinspired features. Using the motion velocities, direction of the motion, and curvatures of the 3D trajectories, Ding et al. [27] categorized the foremost actions into two types of action-units: dynamic instants and intervals. They utilized self-organizing mapping (SOM) [28] to cluster the action-units with the spatiotemporal feature and employed the sequences of the discrete symbols of each action to build profile hidden Markov models (PHMMs) [29] to obtain the spatiotemporal information between the action-units in the given actions. Huang et al. [21] generated a neural network architecture to learn the most informative Lie group representations. Based on the proposed network structure, they input the Lie group features into rotation mapping layers to obtain the desirable results.


## 3. Briefs of Basic Concepts

A brief overview of the Tucker model, NTD, and LDS is presented here, which will help in understanding the tensor-based LDS. LetP∈RI1×I2⋯×IN denote a n-order tensor, P(n) represent the mode-n matricization of tensor P, P represent a matrix, and W(n) represent a mode-n matrix belonging to the Tucker model. If all elements of tensor P are nonnegative (i.e., Pi1,i2,…,iN≥0, where 1≤n≤N and 1≤in≤In ), P is called a nonnegative tensor. The mode-n matricization of tensor P utilizes the elements of tensor P to structure matrix P(n)∈RInIn+1In+2⋯INI1I2⋯In-1.


### 3.1. Tucker Model

The mode-n product of tensorK∈RJ1×J2⋯×JN by matrix W(n)∈RIn×Jn is defined as (J1×⋯×Jn-1×In×Jn+1×⋯×JN)-tensor given by(1)K×nWj1⋯jn-1injn+1⋯jN≈∑jn=1Jnkj1⋯jn⋯jNainjn,for all the index values. With the help of the mode-n product, (1) is rewritten in a form of the matrix unfoldings by fixing the n-th mode as follows:(2)P=K×nW⇔Pn=WKn,where P(n) and K(n) are the matrix unfolding of P and K, respectively.The Tucker model decomposesN-order tensor P∈RI1×I2⋯×IN into the mode products of core tensor K∈RJ1×J2⋯×JN and N mode matrices W(n) as follows:(3)P≈K×1W1×2W2⋯×NWN,where W(n) are the factor matrices and W(n)TW(n)=I.Mode-n matricization of tensorP in (3) can be represented by the mode-n matricization of the core tensor and mode matrices:(4)P≈WnKnWn-1⊗⋯⊗W2⊗W1⊗WN⊗⋯⊗Wn+2⊗Wn+1T, where ⊗ denotes the Kronecker product.


### 3.2. Nonnegative Tucker Decomposition

Given nonnegativeN-order tensor P∈RI1×I2⋯×IN, NTD of P obtains core tensor K and mode matrix W(n)∈RIn×Jn, which are restricted to having only nonnegative elements, such that(5)P≈P^=K×1W1×2W2⋯×NWN.To search for approximate factorizationP^ of tensor P, the cost function was used to quantify the quality of the approximation. The generalized Kullback–Leibler divergence (or I-divergence) is usually used to construct the cost function as follows:(6)DP∥P^=∑i1i2⋯iNpi1i2⋯iNlog⁡pi1i2⋯iNp^i1i2⋯iN-pi1i2⋯iN+p^i1i2⋯iN.To obtain mode matrices W(n) and core tensor K of tensor P, Kim et al. [11] used multiplicative updating algorithms to minimize cost function (6) as follows.Problem 1.
MinimizeD(P∥P^) with respect to P and P^, subject to constraints K≥0,W(1)≥0,W(2)≥0,…,W(N)≥0.They define(7)W∖n=Wn-1⊗⋯⊗W2⊗W1⊗WN⊗⋯⊗Wn+2⊗Wn+1, which is the Kronecker product structure of mode matrices W(n), and W(∖n) is represented as a backward cyclic form. The mode-n matrices of the NTD can be rewritten in the form:(8)Pn=WnKnW∖n=WnKWn.Kim et al. [11] derived multiplicative updating algorithms for the mode matrices W(n) and core tensor K of NTD as follows:The update rule for mode matricesW(n) is(9)Wn←Wn⊛Pn/WnKWnKWn⊤1y⊤.The update rule for core tensorK is(10)K←K⊛P/P^×1W1⊤⋯×NWN⊤ξ×1W1⊤⋯×NWN⊤,where y∈RJn×1 with yi=∑jKW(n)ij, 1∈RIn×1, / is the element-wise division, ⊛ denotes the Hadamard product, and ξ is a distinct tensor, all of whose elements are 1. I-divergence D(P∥P^) is nonincreasing under the update rules.


### 3.3. Linear Dynamical Systems

An LDS belongs to a multivariate time series (MTS) model. The MTS model utilizes hidden states to indirectly represent the observation sequences. Given MTSY=[y(1),…,y(t),…,y(λ)], an LDS is usually described as follows:(11)zt+1=Azt+vtvt~0,Qyt=Czt+wtwt~0,R, where discrete variable t is the time index, x(t) denotes the d-dimensional hidden state at time t, y(t) represents an n-dimensional observed state at time t, and d is order of the LDS. A∈Rd×d is a transitionmatrix that can transform current hidden state x(t+1) to previous hidden state x(t). ObservationmatrixC∈Rn×d can transform hidden state x(t) to observed state y(t). Noise covariance matrices are Q and R. Noise components w(t) and v(t) represent multivariate normal distributions with a zero-mean and covariance matrices Q∈Rd×d and R∈Rn×n, respectively. In [30], G. Doretto et al. employed singular value decomposition (SVD) Y=UΣV of the observed sequence to obtain the best estimate of C and A as follows:(12)C^=UA^=ΣVTD1VVTD2V-1Σ-1, where D1=[00;Iτ-10] and D1=[Iτ-10;00].The parameters(A,C) of LDS do not lie in a linear space. Transition matrix A is constrained to be stable and its eigenvalues are distributed in the unit circle. Observation matrix C is an orthonormal matrix. In fact, matrix C lies in the Stiefel manifold. Parameters (A,C) can be utilized to describe the intrinsic characteristics of the LDS model [31] because A and C represent the dynamics and spatial appearance, respectively. Therefore, pair A,C can be used to describe a set of joint trajectories of an articulated body model. The extended observability matrix [32] for a tuple (A,C) has the following form:(13)O∞T=CT,CAT,CA2T,…,CAnT,….In the current human action research based on skeletons, a human action is usually described as a finite skeleton sequence. Therefore, a human action can be defined as a k-length extended observability matrix as follows:(14)OkT=CT,CAT,CA2T,…,CAk-1T. Here k is the total number of frames in the skeleton sequence. The size of OkT is kn×d. The column space of OkT is a d-dimensional subspace of Rkn.Grassmann manifoldG(p,q) [33] represents a set of q-dimensional linear subspaces of Rp. Each point on the Grassmann manifold is a particular subspace spanned by the column space of p×q, the orthogonal matrices. To obtain a particular subspace spanned by the columns of OkT, the Gram–Schmidt orthonormalization can be used to compute an orthonormal basis. Thus, a human action can be represented as a point on Grassmann manifold G(kn,d) corresponding to the column space of OkT.


## 3.1. Tucker Model

The mode-n product of tensorK∈RJ1×J2⋯×JN by matrix W(n)∈RIn×Jn is defined as (J1×⋯×Jn-1×In×Jn+1×⋯×JN)-tensor given by(1)K×nWj1⋯jn-1injn+1⋯jN≈∑jn=1Jnkj1⋯jn⋯jNainjn,for all the index values. With the help of the mode-n product, (1) is rewritten in a form of the matrix unfoldings by fixing the n-th mode as follows:(2)P=K×nW⇔Pn=WKn,where P(n) and K(n) are the matrix unfolding of P and K, respectively.The Tucker model decomposesN-order tensor P∈RI1×I2⋯×IN into the mode products of core tensor K∈RJ1×J2⋯×JN and N mode matrices W(n) as follows:(3)P≈K×1W1×2W2⋯×NWN,where W(n) are the factor matrices and W(n)TW(n)=I.Mode-n matricization of tensorP in (3) can be represented by the mode-n matricization of the core tensor and mode matrices:(4)P≈WnKnWn-1⊗⋯⊗W2⊗W1⊗WN⊗⋯⊗Wn+2⊗Wn+1T, where ⊗ denotes the Kronecker product.


## 3.2. Nonnegative Tucker Decomposition

Given nonnegativeN-order tensor P∈RI1×I2⋯×IN, NTD of P obtains core tensor K and mode matrix W(n)∈RIn×Jn, which are restricted to having only nonnegative elements, such that(5)P≈P^=K×1W1×2W2⋯×NWN.To search for approximate factorizationP^ of tensor P, the cost function was used to quantify the quality of the approximation. The generalized Kullback–Leibler divergence (or I-divergence) is usually used to construct the cost function as follows:(6)DP∥P^=∑i1i2⋯iNpi1i2⋯iNlog⁡pi1i2⋯iNp^i1i2⋯iN-pi1i2⋯iN+p^i1i2⋯iN.To obtain mode matrices W(n) and core tensor K of tensor P, Kim et al. [11] used multiplicative updating algorithms to minimize cost function (6) as follows.Problem 1.
MinimizeD(P∥P^) with respect to P and P^, subject to constraints K≥0,W(1)≥0,W(2)≥0,…,W(N)≥0.They define(7)W∖n=Wn-1⊗⋯⊗W2⊗W1⊗WN⊗⋯⊗Wn+2⊗Wn+1, which is the Kronecker product structure of mode matrices W(n), and W(∖n) is represented as a backward cyclic form. The mode-n matrices of the NTD can be rewritten in the form:(8)Pn=WnKnW∖n=WnKWn.Kim et al. [11] derived multiplicative updating algorithms for the mode matrices W(n) and core tensor K of NTD as follows:The update rule for mode matricesW(n) is(9)Wn←Wn⊛Pn/WnKWnKWn⊤1y⊤.The update rule for core tensorK is(10)K←K⊛P/P^×1W1⊤⋯×NWN⊤ξ×1W1⊤⋯×NWN⊤,where y∈RJn×1 with yi=∑jKW(n)ij, 1∈RIn×1, / is the element-wise division, ⊛ denotes the Hadamard product, and ξ is a distinct tensor, all of whose elements are 1. I-divergence D(P∥P^) is nonincreasing under the update rules.


## 3.3. Linear Dynamical Systems

An LDS belongs to a multivariate time series (MTS) model. The MTS model utilizes hidden states to indirectly represent the observation sequences. Given MTSY=[y(1),…,y(t),…,y(λ)], an LDS is usually described as follows:(11)zt+1=Azt+vtvt~0,Qyt=Czt+wtwt~0,R, where discrete variable t is the time index, x(t) denotes the d-dimensional hidden state at time t, y(t) represents an n-dimensional observed state at time t, and d is order of the LDS. A∈Rd×d is a transitionmatrix that can transform current hidden state x(t+1) to previous hidden state x(t). ObservationmatrixC∈Rn×d can transform hidden state x(t) to observed state y(t). Noise covariance matrices are Q and R. Noise components w(t) and v(t) represent multivariate normal distributions with a zero-mean and covariance matrices Q∈Rd×d and R∈Rn×n, respectively. In [30], G. Doretto et al. employed singular value decomposition (SVD) Y=UΣV of the observed sequence to obtain the best estimate of C and A as follows:(12)C^=UA^=ΣVTD1VVTD2V-1Σ-1, where D1=[00;Iτ-10] and D1=[Iτ-10;00].The parameters(A,C) of LDS do not lie in a linear space. Transition matrix A is constrained to be stable and its eigenvalues are distributed in the unit circle. Observation matrix C is an orthonormal matrix. In fact, matrix C lies in the Stiefel manifold. Parameters (A,C) can be utilized to describe the intrinsic characteristics of the LDS model [31] because A and C represent the dynamics and spatial appearance, respectively. Therefore, pair A,C can be used to describe a set of joint trajectories of an articulated body model. The extended observability matrix [32] for a tuple (A,C) has the following form:(13)O∞T=CT,CAT,CA2T,…,CAnT,….In the current human action research based on skeletons, a human action is usually described as a finite skeleton sequence. Therefore, a human action can be defined as a k-length extended observability matrix as follows:(14)OkT=CT,CAT,CA2T,…,CAk-1T. Here k is the total number of frames in the skeleton sequence. The size of OkT is kn×d. The column space of OkT is a d-dimensional subspace of Rkn.Grassmann manifoldG(p,q) [33] represents a set of q-dimensional linear subspaces of Rp. Each point on the Grassmann manifold is a particular subspace spanned by the column space of p×q, the orthogonal matrices. To obtain a particular subspace spanned by the columns of OkT, the Gram–Schmidt orthonormalization can be used to compute an orthonormal basis. Thus, a human action can be represented as a point on Grassmann manifold G(kn,d) corresponding to the column space of OkT.


## 4. Nonnegative Tensor-Based Skeleton Sequence Model

### 4.1. Nonnegative Tensor Representation of Skeleton Sequence

The continuity of human motions decides that a skeleton sequence, which is used to describe a human action, is a combination of interrelated skeletons. Instead of traditional skeleton feature vectorization, a skeleton sequence is represented as a nonnegative tensor based on a time series. By searching the nonnegative tensor components of a human action, the tensor representation can better reflect independence of each skeleton and the variation between different skeletons.Figure2(a) shows a human skeleton with19 rigid bodies and20 joints. We preprocess the complete skeleton in the action datasets, which makes it more accurate and easy to represent a skeleton sequence as a nonnegative tensor. We use the preprocessing method elaborated in Section 5.9 to keep the joints of a skeleton in the first octant of the global coordinate system (refer to Figure 2(b)).Figure 2
(a) A human skeleton including 20 joints and 19 rigid bodies. (b) Using the translation to move a skeleton to the first octant.
(a) (b)LetW=(C,E) be a preprocessed skeleton (the preprocessed skeleton located at the first octant of the global coordinate system). C=c1,c2,…,cN is a set of N joints in the first octant of the global coordinate system. ci=[xi,yi,zi]i=1:N denotes the 3D position of joint i. E=e1,2,e1,3,…,ei,j,…,eM-1,M is a set of M rigid bodies. ei,j=[ci-cj] represents the rigid body connecting adjacent joints ci and cj. αi,j,βi,j,γi,j represent the three angles between rigid body ei,j and the global x, y, and z-axes (0≤α,β,γ≤2π). Therefore, each skeleton can be represented as a second-order nonnegative tensor or matrix as follows:(15)Y=v1,2,v1,3,…,vi,j,…,vM-1,MM-1×9T, where vi,j=ci,cj,αi,j,βi,j,γi,j is a distinct set that includes the joints of ei,j and the angles between ei,j and the three global axis. Then, a skeleton sequence can be represented as a third-order nonnegative tensor as Y=Y(1),Y(2),…,Y(λ), where λ is the number of frames in the skeleton sequence.


### 4.2. Nonnegative Tensor-Based LDS Model

Mathematically, a third-order tensor can be unfolded into a second-order tensor time series. It is well known that a second-order tensor time series can be modeled as the output of an LDS. Therefore, a third-order nonnegative tensor, representing a skeleton sequence, is used to build an LDS model. The parameters(A,C) are estimated using the LDS model and NTD of the action. Then, the LDS model is called the nonnegative tensor-based LDS (nLDS) model, as shown in Figure 3.Figure 3
nLDS model with a third-order nonnegative tensor.A skeleton sequence can be represented as a third-order nonnegative tensor as follows:(16)Y=Y1,Y2,…,Yλ∈Rj1×j2×λ, where 1⩽t⩽λ and Y(t)∈Rj1×j2. The NTD of Y uses the update rules proposed by Kim et al. [11] as follows:(17)Wi←Wi⊛Yi/WiSWiSWi⊤1y⊤S←S⊛Y/Y^×1W1⊤×2W2⊤×3W3⊤ξ×1W1⊤×2W2⊤×3W3⊤, where i=1,2, and 3. Encoding variable matrix S(W)(3) is computed in the following manner:(18)SW3=IS3W∖3T=S×1W1×2W2×3I3=S×1W1×2W23=S×m≠3Wm3. Then, the NTD of Y is given by(19)Y≈S×1W1×2W2×3W3, where the core tensor S∈Rd1×d2×τ and mode matrices (W(1)∈Rj1×d1, W(2)∈Rj2×d2, and W(3)∈Rλ×τ) are restricted to having only nonnegative elements in the factorization, 1≤d1≤j1, 1≤d2≤j2, and 1≤τ≤λ. This is illustrated in Figure 4. Then, the mode-3 matricization of Y is achieved as follows:(20)Y3≈W3S3W2⊗W1T,where S(3)∈Rτ×d1d2 is the mode-3 matricization of core tensor S and Y(3)∈Rλ×j1j2 (W(3)S(3)∈Rλ×d1d2, (W(2)⊗W(1))T∈Rd1d2×j1j2, d1d2=d1×d2 and j1j2=j1×j2).Figure 4
Nonnegative Trucker decomposition of a third-order nonnegative tensor.Let third-order nonnegative tensorX=X(1),X(2),…,X(λ)∈Rd1×d2×λ, where 1⩽t⩽λ and X(t)∈Rd1×d2. The mode-3 matricization of X is X(3)=[vecX(1),vecX(2),…,vecX(λ)]T, where X(3)∈Rλ×d1d2 is a nonnegative matrix and vecX(t)∈Rd1d2 is the vector representation of X(t). The mode-3 matricization of Y is Y(3)=[vecY(1),vecY(2),…,vecY(λ)]T, where Y(3)∈Rλ×j1j2 is a nonnegative matrix and vecY(t)∈Rj1j2 is the vector representation of Y(t).Here, we assume thatvecX(t) represents the hidden state and vecY(t) represents the observation. Then, second-order nonnegative tensor time series Y(3)=[vecY(1),vecY(2),…,vecY(λ)]T can be represented by the LDSs as follows:(21)vecXt+1=AvecXt+vtvecYt=CvecXt+wt,where vecX(t)∈Rd1d2, vecY(t)∈Rj1j2, 0⩽t⩽λ, A∈Rd1d2×d1d2, C∈Rj1j2×d1d2 is a nonnegative matrix, w(t)∈Rj1j2 is a zero-mean Gaussian noise modeling the stochastic relation between the states and observations, and v(t)∈Rd1d2 is a zero-mean Gaussian noise modeling the stochastic component of the transition. (We suppose that C∈Rj1j2×d1d2 is a nonnegative matrix.) Then, Y3 can be rewritten as follows:(22)Y3=vecY1,vecY2,…,vecYλT=CvecX1,CvecX2,…,CvecXλT+w1,w2,…,wλ=X3CT+W3,where W(3)=[w(1),w(2),…,w(λ)]∈Rλ×j1j2.Based on (22) and (20), we have(23)W3S3W2⊗W1T≈X3CT+W3.We transpose the matrix on both sides of (23) as follows:(24)W2⊗W1W3S3T≈CX3T+W3T.Next, we consider the problem of finding the estimates ofC and X(3)T in the sense of Frobenius (C⩾0 and X(3)T⩾0 represent nonnegative matrices C and X(3)T, respectively):(25)argminfC,X3T=W2⊗W1W3S3T-CX3TFs.t.C⩾0,X3T⩾0,C∈Rj1j2×d1d2,X3T∈Rd1d2×λ.Since f(C,X(3)T)⩾0, f(C,X(3)T) can achieve its minimum 0 when C=W(2)⊗W(1) and X(3)T=(W(3)S(3))T ((W(2)⊗W(1)) is a j1j2 by d1d2 matrix, while (W(3)S(3))T is a d1d2 by λ matrix), this means that W3TF is approximately equal to zero when (W(2)⊗W(1))(W(3)S(3))T-CX(3)TF is equal to zero. Therefore, the tuple(C^=W(2)⊗W(1), X^(3)T=(W(3)S(3))T) is one of the solutions to the problem (25). In other words, the tuple (C^=W(2)⊗W(1), X^(3)T=(W(3)S(3))T) is an element in the set represented by (25).Then, transition matrixA^ is obtained by solving the least square problem as follows:(26)A^=argminA⁡AX3T1:λ-1-X3T2:λ=X3T2:λX31:λ-1†, where X(3)(i:λ)=[vecX(i),…,vecX(λ)]T and ·F and † denote the Frobenius norm and Moore–Penrose inverse, respectively. Given the above transition matrix A^ and observation matrix C^, noise covariance matrices Q^ and R^ can be estimated directly from the residuals. According to the parameters (A^,C^,Q^,R^) estimated using NTD, we can rewrite the LDS model expressed in (21) as follows:(27)vecXt+1=A^vecXt+vtvt~0,Q^vecYt=C^(vecXt+wtwt~0,R^,calling itthe nonnegative tensor-based LDS(nLDS).The pseudocode in Algorithm1 shows the processing to build the nLDS model and solve its parameters.Algorithm 1:Learning the nLDS model with a second-order tensor time series.
Input:Third-order non-negative tensor Y Output: Parameters C and A in the nLDS model 1:Y(3)=[vecY(1),vecY(2),…,vecY(λ)]T∗ Unfolding Y along three modes 2: Building the nLDS model ofY 3:Y(3)=X(3)CT+W(3)∗ Rewriting Y(3) using nLDS 4:Y=S×1W(1)×2W(2)×3W(3)∗ Using NTD to factorize Y 5:Y(3)=W(3)S(3)(W(2)⊗W(1))T∗ the 3-mode matricization of Y 6:W(3)S(3)(W(2)⊗W(1))T=X(3)CT+W3∗ 7:C^=W(2)⊗W(1) and X^(3)T=(W(3)S(3))T 8:A^=argminA⁡AX(3)T(1:λ-1)-X(3)T(2:λ)=X(3)T(2:λ)X(3)(1:λ-1)†∗ Finding the best estimate of the parameter A


### 4.3. Representing a Human Action as a Point on Infinite Grassmann Manifold

Starting from the initial statevecX(1), the expected observation sequence of an nLDS model is represented as follows:(28)EvecY1,vecY2,vecY3,…T=C^T,C^A^2T,C^A^3T,…TvecX1s.t.C^TC^;μA^<1. Here the transition matrix A∈RI×I is stable with eigenvalues inside the unit circle, the observation matrix C∈RJ×I is an orthonormal matrix, and μ(A^) is the largest eigenvalue of A^ in magnitude. The expected observation sequence lies in the column space of the extended observability sequence given by O∞T=[C^T,(C^A^2)T,(C^A^3)T,…]. The column space of O∞T can be seen as the descriptor of an LDS due to invariance of the choice of the basis of its state space. In this manner, the nLDS model of an action can be represented by its O∞T, which means that the O∞T can be seen as the feature descriptor of the action.The traditional methods [6, 30] can approximate the extended observability sequence O∞T by taking the finite observability matrix (k-order observability matrix) OkT=[C^T,(C^A^2)T,…,(C^A^k-1)T]. Thus, an LDS can be represented as a point on a Grassmann manifold corresponding to the column space of OkT. The value of the order k can influence the approximation of the extended observability matrix: if the value of k is too small, a k-order observability matrix cannot adequately represent the behavior of the extended observability matrix; conversely, the finite observability matrix can be asymptotical to the extended observability matrix by increasing the value of k, but this also leads to an increase in the computational cost. To avoid these limitations, we use the method proposed in [34] to project infinite-order observability matrices to points on infinite Grassmann manifold (infinite Grassmann manifold has been defined as G(I,∞)=⋃k=I∞G(I,k) in [35]). Let O∞T be an element of O∞T=[C^T,(C^A^2)T,(C^A^3)T,…],C^TC^;|μ(A^)|<1. For any O∞T∈O∞T, the orthonormalization of O∞T is performed by the Cholesky decomposition P=Chol(O∞O∞T). Then, the orthonormalization of O∞T is defined as(29)Z∞T=Z∣Z=O∞TP-T,P=CholO∞O∞T,O∞T∈O∞T.S∞T denotes the quotient space of Z∞T. According to the definition of S∞T, it is infinite Grassmann manifold G(I,∞) with an extra intrinsic structure. Thus, an action, represented by O∞T, can be alternately identified as a point on infinite Grassmann manifold S∞T consisting of the orthonormalized extended observability sequences.


### 4.4. Sparse Coding and Dictionary Learning on Infinite Grassmann Manifold

To classify the actions projected to infinite Grassmann manifoldS∞T, an efficient method [35] is used to perform parse coding and dictionary learning on the infinite Grassmann manifold. Given dictionary D=D1,D2,…,DK, action set X^=X1,X2,…,XN, and coefficients y=y1,y2…,yK ( Dj, Xi∈S∞T and y(j,i)∈yi), a sparse coding objective function on S∞T can be expressed as(30)LX,D=miny⁡∑i=1NXiXiT∑j=1Kyj,iDjDjTF+λyi1. The purpose of dictionary learning is to search a good dictionary that can represent all the actions with a small reconstruction error. Let the tuples of the element Dh∈S∞T and the action Xi∈S∞T be (Ah,Ch) and (A^i,C^i), respectively. Then, the problem of dictionary learning on the infinite Grassmann manifold can be expressed as optimization function minD⁡∑h=1K2Θ(h), where(31)Θh=∑i=1N∑j=1,j≠hKyh,iyj,iPh-1∑t=0∞AhtTChTCTAjtPj-TF2-∑i=1Nyh,iPh-1∑t=0∞AhTChTC^iA^itP^i-TF2, where Ph, Pj, and P^i represent the Cholesky decomposition matrices associated with dictionary elements Dh, Dj, and action Xi, respectively.


## 4.1. Nonnegative Tensor Representation of Skeleton Sequence

The continuity of human motions decides that a skeleton sequence, which is used to describe a human action, is a combination of interrelated skeletons. Instead of traditional skeleton feature vectorization, a skeleton sequence is represented as a nonnegative tensor based on a time series. By searching the nonnegative tensor components of a human action, the tensor representation can better reflect independence of each skeleton and the variation between different skeletons.Figure2(a) shows a human skeleton with19 rigid bodies and20 joints. We preprocess the complete skeleton in the action datasets, which makes it more accurate and easy to represent a skeleton sequence as a nonnegative tensor. We use the preprocessing method elaborated in Section 5.9 to keep the joints of a skeleton in the first octant of the global coordinate system (refer to Figure 2(b)).Figure 2
(a) A human skeleton including 20 joints and 19 rigid bodies. (b) Using the translation to move a skeleton to the first octant.
(a) (b)LetW=(C,E) be a preprocessed skeleton (the preprocessed skeleton located at the first octant of the global coordinate system). C=c1,c2,…,cN is a set of N joints in the first octant of the global coordinate system. ci=[xi,yi,zi]i=1:N denotes the 3D position of joint i. E=e1,2,e1,3,…,ei,j,…,eM-1,M is a set of M rigid bodies. ei,j=[ci-cj] represents the rigid body connecting adjacent joints ci and cj. αi,j,βi,j,γi,j represent the three angles between rigid body ei,j and the global x, y, and z-axes (0≤α,β,γ≤2π). Therefore, each skeleton can be represented as a second-order nonnegative tensor or matrix as follows:(15)Y=v1,2,v1,3,…,vi,j,…,vM-1,MM-1×9T, where vi,j=ci,cj,αi,j,βi,j,γi,j is a distinct set that includes the joints of ei,j and the angles between ei,j and the three global axis. Then, a skeleton sequence can be represented as a third-order nonnegative tensor as Y=Y(1),Y(2),…,Y(λ), where λ is the number of frames in the skeleton sequence.


## 4.2. Nonnegative Tensor-Based LDS Model

Mathematically, a third-order tensor can be unfolded into a second-order tensor time series. It is well known that a second-order tensor time series can be modeled as the output of an LDS. Therefore, a third-order nonnegative tensor, representing a skeleton sequence, is used to build an LDS model. The parameters(A,C) are estimated using the LDS model and NTD of the action. Then, the LDS model is called the nonnegative tensor-based LDS (nLDS) model, as shown in Figure 3.Figure 3
nLDS model with a third-order nonnegative tensor.A skeleton sequence can be represented as a third-order nonnegative tensor as follows:(16)Y=Y1,Y2,…,Yλ∈Rj1×j2×λ, where 1⩽t⩽λ and Y(t)∈Rj1×j2. The NTD of Y uses the update rules proposed by Kim et al. [11] as follows:(17)Wi←Wi⊛Yi/WiSWiSWi⊤1y⊤S←S⊛Y/Y^×1W1⊤×2W2⊤×3W3⊤ξ×1W1⊤×2W2⊤×3W3⊤, where i=1,2, and 3. Encoding variable matrix S(W)(3) is computed in the following manner:(18)SW3=IS3W∖3T=S×1W1×2W2×3I3=S×1W1×2W23=S×m≠3Wm3. Then, the NTD of Y is given by(19)Y≈S×1W1×2W2×3W3, where the core tensor S∈Rd1×d2×τ and mode matrices (W(1)∈Rj1×d1, W(2)∈Rj2×d2, and W(3)∈Rλ×τ) are restricted to having only nonnegative elements in the factorization, 1≤d1≤j1, 1≤d2≤j2, and 1≤τ≤λ. This is illustrated in Figure 4. Then, the mode-3 matricization of Y is achieved as follows:(20)Y3≈W3S3W2⊗W1T,where S(3)∈Rτ×d1d2 is the mode-3 matricization of core tensor S and Y(3)∈Rλ×j1j2 (W(3)S(3)∈Rλ×d1d2, (W(2)⊗W(1))T∈Rd1d2×j1j2, d1d2=d1×d2 and j1j2=j1×j2).Figure 4
Nonnegative Trucker decomposition of a third-order nonnegative tensor.Let third-order nonnegative tensorX=X(1),X(2),…,X(λ)∈Rd1×d2×λ, where 1⩽t⩽λ and X(t)∈Rd1×d2. The mode-3 matricization of X is X(3)=[vecX(1),vecX(2),…,vecX(λ)]T, where X(3)∈Rλ×d1d2 is a nonnegative matrix and vecX(t)∈Rd1d2 is the vector representation of X(t). The mode-3 matricization of Y is Y(3)=[vecY(1),vecY(2),…,vecY(λ)]T, where Y(3)∈Rλ×j1j2 is a nonnegative matrix and vecY(t)∈Rj1j2 is the vector representation of Y(t).Here, we assume thatvecX(t) represents the hidden state and vecY(t) represents the observation. Then, second-order nonnegative tensor time series Y(3)=[vecY(1),vecY(2),…,vecY(λ)]T can be represented by the LDSs as follows:(21)vecXt+1=AvecXt+vtvecYt=CvecXt+wt,where vecX(t)∈Rd1d2, vecY(t)∈Rj1j2, 0⩽t⩽λ, A∈Rd1d2×d1d2, C∈Rj1j2×d1d2 is a nonnegative matrix, w(t)∈Rj1j2 is a zero-mean Gaussian noise modeling the stochastic relation between the states and observations, and v(t)∈Rd1d2 is a zero-mean Gaussian noise modeling the stochastic component of the transition. (We suppose that C∈Rj1j2×d1d2 is a nonnegative matrix.) Then, Y3 can be rewritten as follows:(22)Y3=vecY1,vecY2,…,vecYλT=CvecX1,CvecX2,…,CvecXλT+w1,w2,…,wλ=X3CT+W3,where W(3)=[w(1),w(2),…,w(λ)]∈Rλ×j1j2.Based on (22) and (20), we have(23)W3S3W2⊗W1T≈X3CT+W3.We transpose the matrix on both sides of (23) as follows:(24)W2⊗W1W3S3T≈CX3T+W3T.Next, we consider the problem of finding the estimates ofC and X(3)T in the sense of Frobenius (C⩾0 and X(3)T⩾0 represent nonnegative matrices C and X(3)T, respectively):(25)argminfC,X3T=W2⊗W1W3S3T-CX3TFs.t.C⩾0,X3T⩾0,C∈Rj1j2×d1d2,X3T∈Rd1d2×λ.Since f(C,X(3)T)⩾0, f(C,X(3)T) can achieve its minimum 0 when C=W(2)⊗W(1) and X(3)T=(W(3)S(3))T ((W(2)⊗W(1)) is a j1j2 by d1d2 matrix, while (W(3)S(3))T is a d1d2 by λ matrix), this means that W3TF is approximately equal to zero when (W(2)⊗W(1))(W(3)S(3))T-CX(3)TF is equal to zero. Therefore, the tuple(C^=W(2)⊗W(1), X^(3)T=(W(3)S(3))T) is one of the solutions to the problem (25). In other words, the tuple (C^=W(2)⊗W(1), X^(3)T=(W(3)S(3))T) is an element in the set represented by (25).Then, transition matrixA^ is obtained by solving the least square problem as follows:(26)A^=argminA⁡AX3T1:λ-1-X3T2:λ=X3T2:λX31:λ-1†, where X(3)(i:λ)=[vecX(i),…,vecX(λ)]T and ·F and † denote the Frobenius norm and Moore–Penrose inverse, respectively. Given the above transition matrix A^ and observation matrix C^, noise covariance matrices Q^ and R^ can be estimated directly from the residuals. According to the parameters (A^,C^,Q^,R^) estimated using NTD, we can rewrite the LDS model expressed in (21) as follows:(27)vecXt+1=A^vecXt+vtvt~0,Q^vecYt=C^(vecXt+wtwt~0,R^,calling itthe nonnegative tensor-based LDS(nLDS).The pseudocode in Algorithm1 shows the processing to build the nLDS model and solve its parameters.Algorithm 1:Learning the nLDS model with a second-order tensor time series.
Input:Third-order non-negative tensor Y Output: Parameters C and A in the nLDS model 1:Y(3)=[vecY(1),vecY(2),…,vecY(λ)]T∗ Unfolding Y along three modes 2: Building the nLDS model ofY 3:Y(3)=X(3)CT+W(3)∗ Rewriting Y(3) using nLDS 4:Y=S×1W(1)×2W(2)×3W(3)∗ Using NTD to factorize Y 5:Y(3)=W(3)S(3)(W(2)⊗W(1))T∗ the 3-mode matricization of Y 6:W(3)S(3)(W(2)⊗W(1))T=X(3)CT+W3∗ 7:C^=W(2)⊗W(1) and X^(3)T=(W(3)S(3))T 8:A^=argminA⁡AX(3)T(1:λ-1)-X(3)T(2:λ)=X(3)T(2:λ)X(3)(1:λ-1)†∗ Finding the best estimate of the parameter A


## 4.3. Representing a Human Action as a Point on Infinite Grassmann Manifold

Starting from the initial statevecX(1), the expected observation sequence of an nLDS model is represented as follows:(28)EvecY1,vecY2,vecY3,…T=C^T,C^A^2T,C^A^3T,…TvecX1s.t.C^TC^;μA^<1. Here the transition matrix A∈RI×I is stable with eigenvalues inside the unit circle, the observation matrix C∈RJ×I is an orthonormal matrix, and μ(A^) is the largest eigenvalue of A^ in magnitude. The expected observation sequence lies in the column space of the extended observability sequence given by O∞T=[C^T,(C^A^2)T,(C^A^3)T,…]. The column space of O∞T can be seen as the descriptor of an LDS due to invariance of the choice of the basis of its state space. In this manner, the nLDS model of an action can be represented by its O∞T, which means that the O∞T can be seen as the feature descriptor of the action.The traditional methods [6, 30] can approximate the extended observability sequence O∞T by taking the finite observability matrix (k-order observability matrix) OkT=[C^T,(C^A^2)T,…,(C^A^k-1)T]. Thus, an LDS can be represented as a point on a Grassmann manifold corresponding to the column space of OkT. The value of the order k can influence the approximation of the extended observability matrix: if the value of k is too small, a k-order observability matrix cannot adequately represent the behavior of the extended observability matrix; conversely, the finite observability matrix can be asymptotical to the extended observability matrix by increasing the value of k, but this also leads to an increase in the computational cost. To avoid these limitations, we use the method proposed in [34] to project infinite-order observability matrices to points on infinite Grassmann manifold (infinite Grassmann manifold has been defined as G(I,∞)=⋃k=I∞G(I,k) in [35]). Let O∞T be an element of O∞T=[C^T,(C^A^2)T,(C^A^3)T,…],C^TC^;|μ(A^)|<1. For any O∞T∈O∞T, the orthonormalization of O∞T is performed by the Cholesky decomposition P=Chol(O∞O∞T). Then, the orthonormalization of O∞T is defined as(29)Z∞T=Z∣Z=O∞TP-T,P=CholO∞O∞T,O∞T∈O∞T.S∞T denotes the quotient space of Z∞T. According to the definition of S∞T, it is infinite Grassmann manifold G(I,∞) with an extra intrinsic structure. Thus, an action, represented by O∞T, can be alternately identified as a point on infinite Grassmann manifold S∞T consisting of the orthonormalized extended observability sequences.


## 4.4. Sparse Coding and Dictionary Learning on Infinite Grassmann Manifold

To classify the actions projected to infinite Grassmann manifoldS∞T, an efficient method [35] is used to perform parse coding and dictionary learning on the infinite Grassmann manifold. Given dictionary D=D1,D2,…,DK, action set X^=X1,X2,…,XN, and coefficients y=y1,y2…,yK ( Dj, Xi∈S∞T and y(j,i)∈yi), a sparse coding objective function on S∞T can be expressed as(30)LX,D=miny⁡∑i=1NXiXiT∑j=1Kyj,iDjDjTF+λyi1. The purpose of dictionary learning is to search a good dictionary that can represent all the actions with a small reconstruction error. Let the tuples of the element Dh∈S∞T and the action Xi∈S∞T be (Ah,Ch) and (A^i,C^i), respectively. Then, the problem of dictionary learning on the infinite Grassmann manifold can be expressed as optimization function minD⁡∑h=1K2Θ(h), where(31)Θh=∑i=1N∑j=1,j≠hKyh,iyj,iPh-1∑t=0∞AhtTChTCTAjtPj-TF2-∑i=1Nyh,iPh-1∑t=0∞AhTChTC^iA^itP^i-TF2, where Ph, Pj, and P^i represent the Cholesky decomposition matrices associated with dictionary elements Dh, Dj, and action Xi, respectively.


## 5. Experiments

In this section, three standard 3D human action datasets, i.e., the MSR-Action3D dataset [36], UTKinect-Action dataset [37], and G3D-Gaming dataset [2], are utilized to evaluate the proposed LDS model and nonnegative tensor representation.


### 5.1. Alternative Nonnegative Tensor Representation of Skeleton Sequence

To obtain the nonnegative tensor representation more easily, the whole skeleton is transformed to the first octant. Each skeleton in the first octant contains N joint points, N-1 joint angles, and N-1 rigid bodies, and an action sequence consists ofλ skeletons. To verify the effectiveness of our approach, we compare it with the following four alternative nonnegative tensor representations:Second-Order Nonnegative Joint Positions (2NJP). An action sequence is represented as a second-order nonnegative tensor, 3N×λ, in which each skeleton is seen as a nonnegative vector consisting of the 3D coordinates of all joint points.Second-Order Nonnegative Rigid Body Direction (2NRBD). An action sequence is represented as a second-order nonnegative tensor, 3(N-1)×λ, in which each skeleton is seen as a nonnegative vector consisting of the directions of all rigid bodies (the directions of a rigid body are represented by the three angles between the rigid body and x-, y-, and z-axis, respectively).Third-Order Nonnegative Joint Angle and Joint Positions (3NJAP). Given two adjacent rigid bodies, we can obtain a joint angle between the two rigid bodies and a coordinate tuple comprising the coordinates of the three joints in the two rigid bodies. An action sequence can be represented as a third-order nonnegative tensor, (N-1)×10×λ, where each skeleton is seen as a second-order nonnegative tensor that consists of all the joint angles and coordinate tuples.Third-Order Nonnegative Joint Angle and Direction (3NJAD). Given two adjacent rigid bodies, we can obtain a direction tuple containing only the directions of the two rigid bodies and a joint angle between the two rigid bodies. An action sequence can be represented as a third-order nonnegative tensor, N(N-1)×7×λ, in which each skeleton is seen as a second-order nonnegative tensor consisting of all the joint angles and direction tuples.


### 5.2. Parameter Estimation

In the MSR-Action3D, UTKinect-Action, and G3D-Gaming datasets, all the skeletons contain the same number of joints and rigid bodies (19 rigid bodies and 20 joint points). NTD computes the best rank approximation of nonnegative tensorY∈Rj1×j2×λ by the specified dimensions (d1,d2,τ)(S∈Rd1×d2×τ is the core tensor). Thus, d1,d2, and τ can affect the cost function D(Y∥Y^) between nonnegative tensor Y and its best rank approximation Y^. Each skeleton sequence in the three datasets can be represented as nonnegative tensor Y∈R18×9×λ (λ is the length of the skeleton sequence). To compute the best rank approximation of nonnegative tensor Y∈R18×9×λ, we set d1=9, d2=4, and τ=Range(λ) (τ=23 on the MSR-Action3D dataset).


### 5.3. Experiments on MSR-Action3D Dataset

The MSR-Action3D dataset includes 557 human pose sequences that were captured by 10 subjects performing 20 actions, with each action having 2 or 3 repetitions. Each pose of this dataset provides the 3D locations of 20 joints. Each action sequence includes approximately 50 frames. Experiments on the dataset are challenging because many actions are very similar and the pose sequence of the same action class can have a large intraclass variation owing to the performing style variations.Following the experimental protocol in [36], 20 actions in the dataset are categorized into three different subsets: AS1, AS2, and AS3, and each subset includes 8 actions. Subsets AS1 and AS2 include actions with similar movements. Subset AS3 includes more complex actions. The cross-subject evaluation method, in which half of the subjects are used for training and remaining subjects for testing, is utilized to perform recognition on each subset. The average recognition is reported over 10 different combinations of training and testing sets. Table 1 shows that the proposed approach outperforms various other methods extracting the action feature from 3D joint positions. Our approach achieves an average accuracy of 97.63% for the MSR-Action3D dataset, outperforming the other action recognition approaches. The average accuracy of our approach is 0.41% better than the average accuracy of Ensemble TS-LSTM [14], 2.78% better than the average accuracy of tLDS [6], 5.79% better than the average accuracy of Bi-LSTM [15], 1.81% better than the average accuracy of 3NJAP-nLDS, and 0.17% better than the average accuracy of 3NJAD-nLDS. The superior performance on the three subsets indicates that our approach is better than the other methods, for both distinguishing similar actions and recognizing complex actions.Table 1
Recognition rate on the MSR-Action3D dataset based on AS1, AS2, and AS3.
Approach AS1 AS2 AS3 Overall  SE3[12] 94.34 83.87 97.43 91.88  EigenJoints[13] 74.5 76.1 96.4 83.30  Ensemble TS-LSTM[14] 95.24 96.43 100 97.22  tLDS[6] 96.81 89.14 98.83 94.85  Bi-LSTM[15] 92.72 84.93 97.89 91.84  2NJP-nLDS 94.13 92.02 98.56 94.90  2NRBD-nLDS 96.04 92.86 96.22 95.04  3NJAP-nLDS 95.67 93.15 98.64 95.82  3NJAD-nLDS 96.88 95.72 98.15 96.92  Our approach 97.53 96.5 98.87 97.63Following the experimental protocol in [8], we test all the actions on the MSR-Action3D dataset. The experiment on the entire dataset is more challenging than that in [36]. Our approach achieves an accuracy of 96.97%, as shown in Table 2.Table 2
Recognition rate on the MSR-Action3D dataset based on experimental protocol of [8].
Approach Recognition Accuracy  SE3[12] 89.48  Grassmann Manifold[8] 91.21  Learning features combination [16] 90.36 ± 2.45  ST-LSTM + Trust Gate [17] 94.80  Bi-LSTM [15] 86.18  Our approach 96.97Figure5 shows the classification confusion matrix in the entire MSR-Action3D dataset. By observing the confusion matrix, we find that the recognition rate of most actions achieves 100%. It is obvious that classification errors occur if two actions are extremely similar, such asdraw tick andhorizontal arm wave.Figure 5
nLDS model with the third-order nonnegative tensor.


### 5.4. Experiments on UTKinect-Action Dataset

The UTKinect-Action dataset is used to further evaluate our approach. This dataset comprises 10 types of human actions, which are captured by a single stationary Kinect in indoor settings. The 10 actions are walking, sitting down, standing up, picking up, carrying, throwing, pushing, pulling, waving, and clapping hands. Ten different subjects (9 males, 1 female) performed each action twice. Overall, there are 6220 frames of the 199 action sequences. This dataset is very challenging. Firstly, as the body parts of some actions are out of the field-of-view, parts of the human body are invisible. Secondly, different subjects perform the same action with different limbs, such as left-hand waving and right-hand waving. Thirdly, the action sequences captured from different views cause difficulties for action recognition.To appropriately compare our approach with the state-of-the-art algorithms, the leave-one-sequence-out cross validation (LOOCV) method is applied to perform our experiment on the dataset. For each iteration, we choose an action sequence for testing and use the remaining action sequences for training. Each testing sequence was randomly chosen. The experiment on the dataset was performed ten times. Table3 presents the experiment results achieved by our approach and other state-of-the-art methods. The recognition rate of our approach on the dataset is 98.23%. It is obvious that our approach outperforms SE3 [12], EigenJoints [13], Grassmann manifold [8], Key-Pose-Motifs [18], learning features combination [16], Ensemble TS-LSTM [14], tLDS [6], and Bi-LSTM [15], which achieve recognition rates of 97.08%, 97.10%, 88.5%, 93.47%, 98.00%, 96.97%, 96.48%, and 96.89%, respectively. The main reason for this may be that our approach more accurately reflects the relationships between the skeletons in the action sequence.Table 3
Recognition rate on the UTKinect-Action dataset.
Approach Recognition Accuracy SE3[12] 97.08  EigenJoints[13] 97.10  Grassmann Manifold[8] 88.5  Key-Pose-Motifs[18] 93.47  Learning features combination[16] 98.00  Ensemble TS-LSTM[14] 96.97  tLDS[6] 96.48  Bi-LSTM[15] 96.89  Our approach 98.23


### 5.5. G3D-Gaming Dataset

The G3D-Gaming dataset consists of 663 sequences of the 20 different gaming actions captured by Microsoft Kinect. Ten different subjects performed each gaming action more than twice. The dataset can provide three types of data: synchronized video, depth, and skeleton data. Only the skeleton data is chosen to perform our experiment. Experiments on this dataset are also challenging owing to the following two factors: First, when the body parts are occluded, the Kinect tracker gives the inferred results that influence the recognition rate, such as the TennisSwingBackhand, Golf, and ThrowBowlingBall rates. Second, if the movement range of two different actions is relatively small, the two actions may easily interfere with each other in the action recognition. The cross-subject evaluation method is used to perform our experiment. The average recognition results are reported over ten different combinations of the training sets and testing sets. The proposed approach is compared with the state-of-the-art methods reported for the G3D-Gaming dataset, as listed in Table4. GB-RBM+HMM [19] and LieNet [21] use deep learning method to recognize human action: GB-RBM+HMM incorporates a Gaussian binary-restricted Boltzmann machine (GB-RBM) with a hidden Markov model (HMM) to capture the global and local dynamic features of the joint trajectories. LieNet combines the Lie group structures with a deep network architecture to obtain more appropriate Lie group features used to recognize human action. Our approach achieves a recognition accuracy of 92.56%. It is obvious that our approach outperforms GB-RBM+HMM [19], SE [12], SO [20], LieNet [21], and tLDS [6], which achieve recognition rates of 86.40%, 91.09%, 87.95%, 89.10%, and 90.60%, respectively.Table 4
Recognition rates on the G3D-gaming dataset.
Approach Recognition Accuracy GB-RBM+HMM[19] 86.40  SE[12] 91.09  SO[20] 87.95  LieNet[21] 89.10  tLDS[6] 90.60  Our approach 92.56


### 5.6. Discussion about Nonnegative Tensor Representation and Extended Observability Sequence

In this work, an action can be represented by a third-order nonnegative tensor. Referring to the approaches proposed in [6], the finite extended observability sequence OkT of the third-order nonnegative tensor can be considered as the feature descriptor for the action (OkT=[C^T,(C^A^2)T,…,(C^A^k-1)T] and k is the truncation parameter of the extended observability sequence O∞T). The subspace spanned by columns of the finite extended observability sequence corresponds to a point on a Grassmann manifold. Therefore, in order to verify the effectiveness of infinite Grassmann manifold, we use the dictionary learning and sparse coding on Grassmann manifold to classify the nonnegative tensor-based actions represented as points on a Grassmann manifold (the application of extended observability sequence approximation is that an extended observability sequence can be mapped to a point on infinite Grassmann manifold). The experiment results, shown in Table 5, demonstrate that the application of extended observability sequence approximation is effective in improving the accuracies on the three datasets.Table 5
Discussion about infinite Grassmann manifold.
Dataset Nonnegative Tensor Representation + Grassmann manifold Our approach(Nonnegative Tensor Representation + Infinite Grassmann Manifold) MSR-Action3D 95.01 96.97  UTKinect-Action 97.32 98.23  G3D-Gaming 90.86 92.56In [6], an action can be represented by a third-order tensor. The third-order tensor can be mapped to a point on an infinite Grassmann manifold by using the approach proposed in [34]. Then, in order to verify the effectiveness of the nonnegative tensor-based action representation, we use the dictionary learning and sparse coding on infinite Grassmann manifold to classify the third-order tensors represented as points on an infinite Grassmann manifold. The experiment results, shown in Table 6, demonstrate that the nonnegative tensor-based action representation is effective in improving the accuracies on the three datasets.Table 6
Discussion about nonnegative tensor representation.
Dataset Tensor Representation + infinite Grassmann manifold Our approach(Nonnegative Tensor Representation + Infinite Grassmann Manifold) MSR-Action3D 96.17 96.97  UTKinect-Action 97.83 98.23  G3D-Gaming 91.6 92.56


### 5.7. Evaluating the Effect of Infinite Mapping and NTD on the Accuracy of Action Recognition

The extended observability sequence of a third-order nonnegative tensor is given byO∞T=[C^T,(C^A^2)T,(C^A^3)T,…]. The parameters (C^,A^) are estimated by the nonnegative tensor-based LDS (nLDS) model (nLDS model uses NTD to obtain its parameters). The subspace spanned by columns of the finite extended observability sequence OkT corresponds to a point on a Grassmann manifold (k is the truncation parameter of O∞T). Then, the actions, represented by third-order nonnegative tensors, can be mapped to points on a Grassmann manifold. Therefore, in order to verify the effectiveness of NTD, we use LTBSVM [8] to classify the actions represented as points on a Grassmann manifold. The experiment results, shown in Table 7, demonstrate that NTD is effective in improving the accuracies on the three datasets.Table 7
Comparison between SVD and NTD.
Dataset Grassmann manifold[8] (LTBSVM + SVD) LTBSVM + NTD MSR-Action3D 91.21 93.37  UTKinect-Action 88.5 91.69  G3D-Gaming 85.26 87.01In [8], the extended observability matrix of an action is given by θ∞T=[CT,(CA2)T,(CA3)T,…]. The parameters (C,A) are estimated by the ARMA model of the action sequence. We map the extended observability matrix to a point on an infinite Grassmann manifold using the approach proposed in [34]. Then, the dictionary learning and sparse coding on infinite Grassmann manifold are employed to classify the actions represented as points on an infinite Grassmann manifold in order to verify the effectiveness of infinite mapping (or infinite Grassmann manifold). The experiment results, shown in Table 8, demonstrate that infinite mapping is effective in improving the accuracies on the three datasets.Table 8
Comparison between LTBSVM and infinite Grassmann manifold.
Dataset Grassmann manifold[8] (ARMA Model + LTBSVM) ARMA model + Infinite Grassmann Manifold MSR-Action3D 91.21 94.51  UTKinect-Action 88.5 90.65  G3D-Gaming 85.26 89.13


### 5.8. Computation Complexity and Run Time

Computation complexity comprises time complexity and space complexity. The time complexity of our algorithm isO(M∗N2), where N is the scale of a skeleton sequence (or an action) and M is the total number of skeleton sequences in a dataset. Its space complexity is O(M∗N3).Matlab is used to run our experiments on a 3.60GHz Intel Core i7-4790 CPU machine. Run time comprises the training time and the testing time. Table9 shows the run time on the three datasets.Table 9
The run time on the three datasets.
Dataset Run Time MSR-Action3D 23374s  UTKinect-Action 21157s  G3D-Gaming 25189s


### 5.9. Preprocessing and Skeleton Translation

The Preliminary Preprocessing and Translation of Skeleton. Before all skeletons in an action dataset are translated to the local coordinate system, we preprocess the skeletons using the preliminary preprocessing as follows:Preliminary Preprocessing. For each action dataset, all skeletons were transformed to a global coordinate system whose origin locates at the hip center. This makes the skeletal data invariant to absolute location of the human in the scene. One of the skeletons was chosen as a reference skeleton. All other skeletons were normalized (without changing their joint angles) such that their body-part lengths are equal to the corresponding body-part lengths of the reference skeleton, which makes the skeletons scale-invariant. All skeletons were rotated such that the global x-axis is aligned with the ground plane projection of the vector from the left hip to the right hip, which makes the skeletons view-invariant. Figure 6(a) shows a preprocessed skeleton.Figure 6
(a) a preprocessed skeleton in global coordinate system, (b) the translation of the preprocessed skeleton.
(a) (b)Obviously, the hip center of each preprocessed skeleton is located at the origino of the global coordinate system. This means that whether a subject moves a lot or little, the translation’s amount of each joint in the preprocessed skeleton sequence is not large (the preprocessed skeleton sequence represents the locomotion of the subject). In other words, when the locomotion of subjects was represented by the preprocessed skeleton sequences, the amount of translation, required to make all joint coordinates stay in the first octant, is not large. Therefore, the locomotion of subjects does not influence the amount of translation because all skeletons have been preprocessed by preliminary preprocessing. In fact, the amount of translation is only related to the joints in the preprocessed skeleton sequences, while the translation’s amount of the joints is small. For our approach, the preliminary preprocessing not only has no special limitations, but also helps to improve the accuracy for action recognition.Given an action datasetD, all skeletons in the dataset have been preprocessed by the preliminary preprocessing. Let pti(j)=xti(j),yti(j),zti(j)j=[1:J],t=[1:T],i=[1:I] denote the 3D position of joint j at skeleton t in skeleton sequence i, where J is the total number of joints in skeleton t, T is the total number of skeletons in skeleton sequence i, and I is the total number of skeleton sequences in action dataset D (skeleton sequence i belongs to action dataset D, which means that the skeletons in skeleton sequence i have been preprocessed by the preliminary preprocessing). Let o′=(xo′,yo′,zo′) be the origin of the local coordinate system, where xo′=maxxti(j)∣j∈[1:J],t∈[1:T],i∈[1:I]andxti(j)<0, yo′=maxyti(j)∣j∈[1:J],t∈[1:T],i∈[1:I]andyti(j)<0, and zo′=maxzti(j)∣j∈[1:J],t∈[1:T],i∈[1:I]andzti(j)<0. Next, to make all joints stay in the first octant, all preprocessed skeletons in action dataset D are translated to the local coordinate system with the origin at point o′, and the hip center of the preprocessed skeletons is placed at origin o′ of the local coordinate system. Figure 6(b) shows a preprocessed skeleton translated to the local coordinate system.Discussion about Skeleton Translation. Let dop=op denote the distance between the origin p of the local coordinate system and the origin o of the global coordinate system, where p=(xo′+α,yo′+β,zo′+γ), α⩾0, β⩾0, and γ⩾0. Figure 7 shows the relationship between dop and the recognition rates. We found that the recognition rates gradually decreased with the increase of dop, which meant that the translation of the skeletons affects the accuracy for action recognition. Therefore, aiming to reduce the negative effect as far as possible, we set the origin of the local coordinate system to o′, which can ensure that the joints of translated skeletons are located at the first octant of the global coordinate system while dop can achieve its minimum doo′ as well.Figure 7
The influence of skeletal translation on action recognition.


## 5.1. Alternative Nonnegative Tensor Representation of Skeleton Sequence

To obtain the nonnegative tensor representation more easily, the whole skeleton is transformed to the first octant. Each skeleton in the first octant contains N joint points, N-1 joint angles, and N-1 rigid bodies, and an action sequence consists ofλ skeletons. To verify the effectiveness of our approach, we compare it with the following four alternative nonnegative tensor representations:Second-Order Nonnegative Joint Positions (2NJP). An action sequence is represented as a second-order nonnegative tensor, 3N×λ, in which each skeleton is seen as a nonnegative vector consisting of the 3D coordinates of all joint points.Second-Order Nonnegative Rigid Body Direction (2NRBD). An action sequence is represented as a second-order nonnegative tensor, 3(N-1)×λ, in which each skeleton is seen as a nonnegative vector consisting of the directions of all rigid bodies (the directions of a rigid body are represented by the three angles between the rigid body and x-, y-, and z-axis, respectively).Third-Order Nonnegative Joint Angle and Joint Positions (3NJAP). Given two adjacent rigid bodies, we can obtain a joint angle between the two rigid bodies and a coordinate tuple comprising the coordinates of the three joints in the two rigid bodies. An action sequence can be represented as a third-order nonnegative tensor, (N-1)×10×λ, where each skeleton is seen as a second-order nonnegative tensor that consists of all the joint angles and coordinate tuples.Third-Order Nonnegative Joint Angle and Direction (3NJAD). Given two adjacent rigid bodies, we can obtain a direction tuple containing only the directions of the two rigid bodies and a joint angle between the two rigid bodies. An action sequence can be represented as a third-order nonnegative tensor, N(N-1)×7×λ, in which each skeleton is seen as a second-order nonnegative tensor consisting of all the joint angles and direction tuples.


## 5.2. Parameter Estimation

In the MSR-Action3D, UTKinect-Action, and G3D-Gaming datasets, all the skeletons contain the same number of joints and rigid bodies (19 rigid bodies and 20 joint points). NTD computes the best rank approximation of nonnegative tensorY∈Rj1×j2×λ by the specified dimensions (d1,d2,τ)(S∈Rd1×d2×τ is the core tensor). Thus, d1,d2, and τ can affect the cost function D(Y∥Y^) between nonnegative tensor Y and its best rank approximation Y^. Each skeleton sequence in the three datasets can be represented as nonnegative tensor Y∈R18×9×λ (λ is the length of the skeleton sequence). To compute the best rank approximation of nonnegative tensor Y∈R18×9×λ, we set d1=9, d2=4, and τ=Range(λ) (τ=23 on the MSR-Action3D dataset).


## 5.3. Experiments on MSR-Action3D Dataset

The MSR-Action3D dataset includes 557 human pose sequences that were captured by 10 subjects performing 20 actions, with each action having 2 or 3 repetitions. Each pose of this dataset provides the 3D locations of 20 joints. Each action sequence includes approximately 50 frames. Experiments on the dataset are challenging because many actions are very similar and the pose sequence of the same action class can have a large intraclass variation owing to the performing style variations.Following the experimental protocol in [36], 20 actions in the dataset are categorized into three different subsets: AS1, AS2, and AS3, and each subset includes 8 actions. Subsets AS1 and AS2 include actions with similar movements. Subset AS3 includes more complex actions. The cross-subject evaluation method, in which half of the subjects are used for training and remaining subjects for testing, is utilized to perform recognition on each subset. The average recognition is reported over 10 different combinations of training and testing sets. Table 1 shows that the proposed approach outperforms various other methods extracting the action feature from 3D joint positions. Our approach achieves an average accuracy of 97.63% for the MSR-Action3D dataset, outperforming the other action recognition approaches. The average accuracy of our approach is 0.41% better than the average accuracy of Ensemble TS-LSTM [14], 2.78% better than the average accuracy of tLDS [6], 5.79% better than the average accuracy of Bi-LSTM [15], 1.81% better than the average accuracy of 3NJAP-nLDS, and 0.17% better than the average accuracy of 3NJAD-nLDS. The superior performance on the three subsets indicates that our approach is better than the other methods, for both distinguishing similar actions and recognizing complex actions.Table 1
Recognition rate on the MSR-Action3D dataset based on AS1, AS2, and AS3.
Approach AS1 AS2 AS3 Overall  SE3[12] 94.34 83.87 97.43 91.88  EigenJoints[13] 74.5 76.1 96.4 83.30  Ensemble TS-LSTM[14] 95.24 96.43 100 97.22  tLDS[6] 96.81 89.14 98.83 94.85  Bi-LSTM[15] 92.72 84.93 97.89 91.84  2NJP-nLDS 94.13 92.02 98.56 94.90  2NRBD-nLDS 96.04 92.86 96.22 95.04  3NJAP-nLDS 95.67 93.15 98.64 95.82  3NJAD-nLDS 96.88 95.72 98.15 96.92  Our approach 97.53 96.5 98.87 97.63Following the experimental protocol in [8], we test all the actions on the MSR-Action3D dataset. The experiment on the entire dataset is more challenging than that in [36]. Our approach achieves an accuracy of 96.97%, as shown in Table 2.Table 2
Recognition rate on the MSR-Action3D dataset based on experimental protocol of [8].
Approach Recognition Accuracy  SE3[12] 89.48  Grassmann Manifold[8] 91.21  Learning features combination [16] 90.36 ± 2.45  ST-LSTM + Trust Gate [17] 94.80  Bi-LSTM [15] 86.18  Our approach 96.97Figure5 shows the classification confusion matrix in the entire MSR-Action3D dataset. By observing the confusion matrix, we find that the recognition rate of most actions achieves 100%. It is obvious that classification errors occur if two actions are extremely similar, such asdraw tick andhorizontal arm wave.Figure 5
nLDS model with the third-order nonnegative tensor.


## 5.4. Experiments on UTKinect-Action Dataset

The UTKinect-Action dataset is used to further evaluate our approach. This dataset comprises 10 types of human actions, which are captured by a single stationary Kinect in indoor settings. The 10 actions are walking, sitting down, standing up, picking up, carrying, throwing, pushing, pulling, waving, and clapping hands. Ten different subjects (9 males, 1 female) performed each action twice. Overall, there are 6220 frames of the 199 action sequences. This dataset is very challenging. Firstly, as the body parts of some actions are out of the field-of-view, parts of the human body are invisible. Secondly, different subjects perform the same action with different limbs, such as left-hand waving and right-hand waving. Thirdly, the action sequences captured from different views cause difficulties for action recognition.To appropriately compare our approach with the state-of-the-art algorithms, the leave-one-sequence-out cross validation (LOOCV) method is applied to perform our experiment on the dataset. For each iteration, we choose an action sequence for testing and use the remaining action sequences for training. Each testing sequence was randomly chosen. The experiment on the dataset was performed ten times. Table3 presents the experiment results achieved by our approach and other state-of-the-art methods. The recognition rate of our approach on the dataset is 98.23%. It is obvious that our approach outperforms SE3 [12], EigenJoints [13], Grassmann manifold [8], Key-Pose-Motifs [18], learning features combination [16], Ensemble TS-LSTM [14], tLDS [6], and Bi-LSTM [15], which achieve recognition rates of 97.08%, 97.10%, 88.5%, 93.47%, 98.00%, 96.97%, 96.48%, and 96.89%, respectively. The main reason for this may be that our approach more accurately reflects the relationships between the skeletons in the action sequence.Table 3
Recognition rate on the UTKinect-Action dataset.
Approach Recognition Accuracy SE3[12] 97.08  EigenJoints[13] 97.10  Grassmann Manifold[8] 88.5  Key-Pose-Motifs[18] 93.47  Learning features combination[16] 98.00  Ensemble TS-LSTM[14] 96.97  tLDS[6] 96.48  Bi-LSTM[15] 96.89  Our approach 98.23


## 5.5. G3D-Gaming Dataset

The G3D-Gaming dataset consists of 663 sequences of the 20 different gaming actions captured by Microsoft Kinect. Ten different subjects performed each gaming action more than twice. The dataset can provide three types of data: synchronized video, depth, and skeleton data. Only the skeleton data is chosen to perform our experiment. Experiments on this dataset are also challenging owing to the following two factors: First, when the body parts are occluded, the Kinect tracker gives the inferred results that influence the recognition rate, such as the TennisSwingBackhand, Golf, and ThrowBowlingBall rates. Second, if the movement range of two different actions is relatively small, the two actions may easily interfere with each other in the action recognition. The cross-subject evaluation method is used to perform our experiment. The average recognition results are reported over ten different combinations of the training sets and testing sets. The proposed approach is compared with the state-of-the-art methods reported for the G3D-Gaming dataset, as listed in Table4. GB-RBM+HMM [19] and LieNet [21] use deep learning method to recognize human action: GB-RBM+HMM incorporates a Gaussian binary-restricted Boltzmann machine (GB-RBM) with a hidden Markov model (HMM) to capture the global and local dynamic features of the joint trajectories. LieNet combines the Lie group structures with a deep network architecture to obtain more appropriate Lie group features used to recognize human action. Our approach achieves a recognition accuracy of 92.56%. It is obvious that our approach outperforms GB-RBM+HMM [19], SE [12], SO [20], LieNet [21], and tLDS [6], which achieve recognition rates of 86.40%, 91.09%, 87.95%, 89.10%, and 90.60%, respectively.Table 4
Recognition rates on the G3D-gaming dataset.
Approach Recognition Accuracy GB-RBM+HMM[19] 86.40  SE[12] 91.09  SO[20] 87.95  LieNet[21] 89.10  tLDS[6] 90.60  Our approach 92.56


## 5.6. Discussion about Nonnegative Tensor Representation and Extended Observability Sequence

In this work, an action can be represented by a third-order nonnegative tensor. Referring to the approaches proposed in [6], the finite extended observability sequence OkT of the third-order nonnegative tensor can be considered as the feature descriptor for the action (OkT=[C^T,(C^A^2)T,…,(C^A^k-1)T] and k is the truncation parameter of the extended observability sequence O∞T). The subspace spanned by columns of the finite extended observability sequence corresponds to a point on a Grassmann manifold. Therefore, in order to verify the effectiveness of infinite Grassmann manifold, we use the dictionary learning and sparse coding on Grassmann manifold to classify the nonnegative tensor-based actions represented as points on a Grassmann manifold (the application of extended observability sequence approximation is that an extended observability sequence can be mapped to a point on infinite Grassmann manifold). The experiment results, shown in Table 5, demonstrate that the application of extended observability sequence approximation is effective in improving the accuracies on the three datasets.Table 5
Discussion about infinite Grassmann manifold.
Dataset Nonnegative Tensor Representation + Grassmann manifold Our approach(Nonnegative Tensor Representation + Infinite Grassmann Manifold) MSR-Action3D 95.01 96.97  UTKinect-Action 97.32 98.23  G3D-Gaming 90.86 92.56In [6], an action can be represented by a third-order tensor. The third-order tensor can be mapped to a point on an infinite Grassmann manifold by using the approach proposed in [34]. Then, in order to verify the effectiveness of the nonnegative tensor-based action representation, we use the dictionary learning and sparse coding on infinite Grassmann manifold to classify the third-order tensors represented as points on an infinite Grassmann manifold. The experiment results, shown in Table 6, demonstrate that the nonnegative tensor-based action representation is effective in improving the accuracies on the three datasets.Table 6
Discussion about nonnegative tensor representation.
Dataset Tensor Representation + infinite Grassmann manifold Our approach(Nonnegative Tensor Representation + Infinite Grassmann Manifold) MSR-Action3D 96.17 96.97  UTKinect-Action 97.83 98.23  G3D-Gaming 91.6 92.56


## 5.7. Evaluating the Effect of Infinite Mapping and NTD on the Accuracy of Action Recognition

The extended observability sequence of a third-order nonnegative tensor is given byO∞T=[C^T,(C^A^2)T,(C^A^3)T,…]. The parameters (C^,A^) are estimated by the nonnegative tensor-based LDS (nLDS) model (nLDS model uses NTD to obtain its parameters). The subspace spanned by columns of the finite extended observability sequence OkT corresponds to a point on a Grassmann manifold (k is the truncation parameter of O∞T). Then, the actions, represented by third-order nonnegative tensors, can be mapped to points on a Grassmann manifold. Therefore, in order to verify the effectiveness of NTD, we use LTBSVM [8] to classify the actions represented as points on a Grassmann manifold. The experiment results, shown in Table 7, demonstrate that NTD is effective in improving the accuracies on the three datasets.Table 7
Comparison between SVD and NTD.
Dataset Grassmann manifold[8] (LTBSVM + SVD) LTBSVM + NTD MSR-Action3D 91.21 93.37  UTKinect-Action 88.5 91.69  G3D-Gaming 85.26 87.01In [8], the extended observability matrix of an action is given by θ∞T=[CT,(CA2)T,(CA3)T,…]. The parameters (C,A) are estimated by the ARMA model of the action sequence. We map the extended observability matrix to a point on an infinite Grassmann manifold using the approach proposed in [34]. Then, the dictionary learning and sparse coding on infinite Grassmann manifold are employed to classify the actions represented as points on an infinite Grassmann manifold in order to verify the effectiveness of infinite mapping (or infinite Grassmann manifold). The experiment results, shown in Table 8, demonstrate that infinite mapping is effective in improving the accuracies on the three datasets.Table 8
Comparison between LTBSVM and infinite Grassmann manifold.
Dataset Grassmann manifold[8] (ARMA Model + LTBSVM) ARMA model + Infinite Grassmann Manifold MSR-Action3D 91.21 94.51  UTKinect-Action 88.5 90.65  G3D-Gaming 85.26 89.13


## 5.8. Computation Complexity and Run Time

Computation complexity comprises time complexity and space complexity. The time complexity of our algorithm isO(M∗N2), where N is the scale of a skeleton sequence (or an action) and M is the total number of skeleton sequences in a dataset. Its space complexity is O(M∗N3).Matlab is used to run our experiments on a 3.60GHz Intel Core i7-4790 CPU machine. Run time comprises the training time and the testing time. Table9 shows the run time on the three datasets.Table 9
The run time on the three datasets.
Dataset Run Time MSR-Action3D 23374s  UTKinect-Action 21157s  G3D-Gaming 25189s


## 5.9. Preprocessing and Skeleton Translation

The Preliminary Preprocessing and Translation of Skeleton. Before all skeletons in an action dataset are translated to the local coordinate system, we preprocess the skeletons using the preliminary preprocessing as follows:Preliminary Preprocessing. For each action dataset, all skeletons were transformed to a global coordinate system whose origin locates at the hip center. This makes the skeletal data invariant to absolute location of the human in the scene. One of the skeletons was chosen as a reference skeleton. All other skeletons were normalized (without changing their joint angles) such that their body-part lengths are equal to the corresponding body-part lengths of the reference skeleton, which makes the skeletons scale-invariant. All skeletons were rotated such that the global x-axis is aligned with the ground plane projection of the vector from the left hip to the right hip, which makes the skeletons view-invariant. Figure 6(a) shows a preprocessed skeleton.Figure 6
(a) a preprocessed skeleton in global coordinate system, (b) the translation of the preprocessed skeleton.
(a) (b)Obviously, the hip center of each preprocessed skeleton is located at the origino of the global coordinate system. This means that whether a subject moves a lot or little, the translation’s amount of each joint in the preprocessed skeleton sequence is not large (the preprocessed skeleton sequence represents the locomotion of the subject). In other words, when the locomotion of subjects was represented by the preprocessed skeleton sequences, the amount of translation, required to make all joint coordinates stay in the first octant, is not large. Therefore, the locomotion of subjects does not influence the amount of translation because all skeletons have been preprocessed by preliminary preprocessing. In fact, the amount of translation is only related to the joints in the preprocessed skeleton sequences, while the translation’s amount of the joints is small. For our approach, the preliminary preprocessing not only has no special limitations, but also helps to improve the accuracy for action recognition.Given an action datasetD, all skeletons in the dataset have been preprocessed by the preliminary preprocessing. Let pti(j)=xti(j),yti(j),zti(j)j=[1:J],t=[1:T],i=[1:I] denote the 3D position of joint j at skeleton t in skeleton sequence i, where J is the total number of joints in skeleton t, T is the total number of skeletons in skeleton sequence i, and I is the total number of skeleton sequences in action dataset D (skeleton sequence i belongs to action dataset D, which means that the skeletons in skeleton sequence i have been preprocessed by the preliminary preprocessing). Let o′=(xo′,yo′,zo′) be the origin of the local coordinate system, where xo′=maxxti(j)∣j∈[1:J],t∈[1:T],i∈[1:I]andxti(j)<0, yo′=maxyti(j)∣j∈[1:J],t∈[1:T],i∈[1:I]andyti(j)<0, and zo′=maxzti(j)∣j∈[1:J],t∈[1:T],i∈[1:I]andzti(j)<0. Next, to make all joints stay in the first octant, all preprocessed skeletons in action dataset D are translated to the local coordinate system with the origin at point o′, and the hip center of the preprocessed skeletons is placed at origin o′ of the local coordinate system. Figure 6(b) shows a preprocessed skeleton translated to the local coordinate system.Discussion about Skeleton Translation. Let dop=op denote the distance between the origin p of the local coordinate system and the origin o of the global coordinate system, where p=(xo′+α,yo′+β,zo′+γ), α⩾0, β⩾0, and γ⩾0. Figure 7 shows the relationship between dop and the recognition rates. We found that the recognition rates gradually decreased with the increase of dop, which meant that the translation of the skeletons affects the accuracy for action recognition. Therefore, aiming to reduce the negative effect as far as possible, we set the origin of the local coordinate system to o′, which can ensure that the joints of translated skeletons are located at the first octant of the global coordinate system while dop can achieve its minimum doo′ as well.Figure 7
The influence of skeletal translation on action recognition.


## 6. Conclusion and Future Work

In this paper, an action is represented as a third-order nonnegative tensor. To capture the original spatiotemporal information of the action, an nLDS is used to model the action. NTD is employed to estimate the parameters of the nLDS model. The extended observability sequence of the parameters, which is considered as the feature descriptor of the action, is mapped to a point on infinite Grassmann manifold. The dictionary learning and sparse coding on infinite Grassmann manifold are used to perform classification. The experiment results demonstrate that our approach achieves a better performance than other state-of-the-art skeleton-based action recognition approaches. According to the theories proposed by Zhou et al. [38], future researches will focus on the effect of the unique and sparse NTD on action recognition.



---

*Source: 8940807-2019-03-27.xml*