# Extrinsic Least Squares Regression with Closed-Form Solution on Product Grassmann Manifold for Video-Based Recognition

Authors: Yuping Wang; Lichun Wang; Dehui Kong; Baocai Yin
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

Least squares regression is a fundamental tool in statistical analysis and is more effective than some complicated models with small number of training samples. Representing multidimensional data with product Grassmann manifold has recently led to notable results in various visual recognition tasks. This paper proposes extrinsic least squares regression with Projection Metric on product Grassmann manifold by embedding Grassmann manifold into the space of symmetric matrices via an isometric mapping. The proposed regression has closed-form solution which is more accurate compared with numerical solution of previous least squares regression using geodesic distance. Experiments on several recognition tasks show that the proposed method achieves considerable accuracy in comparison with some state-of-the-art methods.


---

## Body

## 1. Introduction

As an important application of computer vision, video-based recognition such as action recognition [1] attracts more and more attention. For inferring the correct label of a query in a given database of examples, there are mainly two kinds of methods. One kind approach is based on representations with the handcrafted features and the other kind is based on deep learning architectures such as Convolutional Neural Networks (CNN) [2]. Generally speaking, deep learning algorithms have been shown to be successful when large amount of data is available [3, 4]. However, the size of database for many recognition tasks in daily life is small. In this case, deep learning algorithms lose efficacy and it becomes important to analyze the structure of data and represent it with discriminant features.Nowadays, Grassmann manifold has proven a powerful representation for video-based applications like activity classification [5], action recognition [6], age estimation [7], face recognition [8, 9], and so on. In the above applications, Grassmann manifold is used to characterize the intrinsic geometry of data. Taking one representative work as an example, Lui [10] factorized a data tensor using Higher Order Singular Value Decomposition (HOSVD) and imposed each factorized element on a Grassmann manifold. This representation yields a very discriminating structure for action recognition.Inference on manifold spaces can be achieved extrinsically by embedding manifold into Euclidean space, which can be considered as flattening the manifold. In the literature, the most popular choice for embedding manifold is through considering tangent spaces [11, 12]. For example, Lui [10] presented a least squares regression on product Grassmann manifold, in which the weighted average from the training samples was computed in tangent space and was projected back to Grassmann manifold by standard logarithmic and exponential map. The distance between points to the tangent pole is equal to geodesic distance, which is restrictive and may lead to inaccurate modeling. An alternate method considers embedding Grassmann manifold into space of symmetric matrices by a diffeomorphism [13] and uses Projection Metric [14] which is equal to the true Grassmann geodesic distance up to a scale of 2.In this paper, by representing multidimensional data on product Grassmann manifold with same form as Lui [10], we propose an extrinsic least squares regression on product Grassmann manifold using Projection Metric and give a closed-form solution which is more accurate. Least squares regression as a simple statistical model has many advantages such as simple calculation and being more effective than some complicated models with small number of training samples [15]. We experiment with the proposed method on three kinds of small-scale datasets including hand gesture, Ballet, and traffic; the higher recognition rates reveal that our method is competitive to some state-of-the-art methods.The rest of this paper is organized as follows: Section2 introduces mathematical background; Section 3 gives product Grassmann manifold representation for video; Section 4 presents distance on product Grassmann manifold; Section 5 proposes extrinsic least squares regression on product Grassmann manifold; Section 6 gives classification based on extrinsic least squares regression; Section 7 shows experiments on different datasets, and experiment results show that the proposed method achieves considerable accuracy; Section 8 analyzes the time complexity of proposed method and Section 9 gives a conclusion.


## 2. Mathematical Background

In this section, we introduce the mathematical background used in this paper.


### 2.1. Grassmann Manifold

Stiefel manifoldS(p,d) is the set of all d×p matrices with orthonormal columns; that is, (1)Sp,d≜X∈Rd×p:XTX=Ip, where Ip is the p×p identity matrix. Grassmann manifold G(p,d) can be defined as a quotient manifold of S(p,d) with an equivalence relation ~. In fact, for any X,Y∈S(p,d), (2)X~Y⟺span⁡X=span⁡Y,where span(X) is the subspace spanned by columns of X∈S(p,d). In other words, Grassmann manifold G(p,d) is the space of p-dimensional linear subspaces of Rd for 0<p<d [16], which may be specified by arbitrary orthogonal matrix with dimension d×p. Notice it is not unique for the choice of matrix X for a point span(X) on Grassmann manifold; that is, the same point on Grassmann manifold can be spanned by different matrix X and Y.


### 2.2. Higher Order Singular Value Decomposition (HOSVD)

HOSVD is a multilinear SVD operating on tensor. LetA∈RI1×I2×⋯×IM be a tensor with order M. The process of reordering the elements of an M-mode tensor into a matrix is called matricization. The mode-k(k=1,…,M) matricization of a tensor A is denoted by A(k) (see details in [17]). Then each A(k) is factored using SVD as follows:(3)Ak=UkΣkVkT,where Σ(k) is a diagonal matrix, U(k) is an orthogonal matrix which spanned the column space of A(k), and V(k) is an orthogonal matrix which spanned the row space of A(k). By using HOSVD method, an M order tensor A can be decomposed as follows:(4)A=S×1U1×2U2×3⋯×MUM, where S∈RI1×I2×⋯×IM is core tensor, U(1),U(2),…,U(M) are orthogonal matrices given in (3), and ×k denotes mode-k multiplication.


### 2.3. Product Manifold

LetU1,U2,…,UM be manifolds; the product manifold of the manifolds U1,U2,…,UM is defined as (5)U=U1×U2×⋯×UM=x1,x2,…,xM:x1∈U1,x2∈U2,…,xM∈UM, where × denotes Cartesian product and Ui(i=1,2,…,M) is called factor manifold.


## 2.1. Grassmann Manifold

Stiefel manifoldS(p,d) is the set of all d×p matrices with orthonormal columns; that is, (1)Sp,d≜X∈Rd×p:XTX=Ip, where Ip is the p×p identity matrix. Grassmann manifold G(p,d) can be defined as a quotient manifold of S(p,d) with an equivalence relation ~. In fact, for any X,Y∈S(p,d), (2)X~Y⟺span⁡X=span⁡Y,where span(X) is the subspace spanned by columns of X∈S(p,d). In other words, Grassmann manifold G(p,d) is the space of p-dimensional linear subspaces of Rd for 0<p<d [16], which may be specified by arbitrary orthogonal matrix with dimension d×p. Notice it is not unique for the choice of matrix X for a point span(X) on Grassmann manifold; that is, the same point on Grassmann manifold can be spanned by different matrix X and Y.


## 2.2. Higher Order Singular Value Decomposition (HOSVD)

HOSVD is a multilinear SVD operating on tensor. LetA∈RI1×I2×⋯×IM be a tensor with order M. The process of reordering the elements of an M-mode tensor into a matrix is called matricization. The mode-k(k=1,…,M) matricization of a tensor A is denoted by A(k) (see details in [17]). Then each A(k) is factored using SVD as follows:(3)Ak=UkΣkVkT,where Σ(k) is a diagonal matrix, U(k) is an orthogonal matrix which spanned the column space of A(k), and V(k) is an orthogonal matrix which spanned the row space of A(k). By using HOSVD method, an M order tensor A can be decomposed as follows:(4)A=S×1U1×2U2×3⋯×MUM, where S∈RI1×I2×⋯×IM is core tensor, U(1),U(2),…,U(M) are orthogonal matrices given in (3), and ×k denotes mode-k multiplication.


## 2.3. Product Manifold

LetU1,U2,…,UM be manifolds; the product manifold of the manifolds U1,U2,…,UM is defined as (5)U=U1×U2×⋯×UM=x1,x2,…,xM:x1∈U1,x2∈U2,…,xM∈UM, where × denotes Cartesian product and Ui(i=1,2,…,M) is called factor manifold.


## 3. Product Grassmann Manifold Representation for Video

Video is a kind of multidimensional data and can be represented as tensorA∈RJ1×J2×J3, where J1, J2, and J3 represent height, width, and length of video, respectively. The variation of each mode can be captured by HOSVD. Lui et al. [18] found that traditional HOSVD is not appropriate for forming product manifold, so they redefined the traditional definition of HOSVD to factorize tensor using the orthogonal matrices V(1), V(2), and V(3) described in (3). That is, (6)A=S^×1V1×2V2×3V3, where S^∈R(J2×J3)×(J1×J3)×(J1×J2) is core tensor.SinceV(k)(k=1,2,3) is a tall orthogonal matrix, hence it is a point on Stiefel manifold. Then span(V(k)) is a point on Grassmann manifold. Hence, (span(V(1)),span(V(2)),span(V(3))) is a point on product Grassmann manifold. Then (span(V(1)),span(V(2)),span(V(3))) is a representation for videos on product Grassmann manifold.


## 4. Distance on Product Grassmann Manifold

The metric on Grassmann manifold is geodesic distance which is the shortest curve between twop-dimensional subspaces X and Y, that is, dg(X,Y)=∑i=1psin2θi with θi representing the principal angles [16]. Recently, Chikuse [13] introduced a projection embedding Π:G(p,d)→sym(d), Π(X)=XXT, where sym(d) denotes space of symmetric matrices. And Hamm and Lee [19] defined a distance calledProjection Metric on Grassmann manifold as follows.Definition 1.
Given two pointsX and Y on Grassmann manifold G(p,d), the distance between X and Y is defined as (7)dsX,Y=22XXT-YYTF.Remark 2.
In fact, for any matrixX≠Y, there exists a p×p orthogonal matrix Qp such that X=YQp, then element span(X)∈G(p,d) is equal to element span(Y)∈G(p,d). In this case, dsX,Y=2/2YQpQpTYT-YYTF=0. Hence it is feasible to use the matrix X representing span(X). And ds(X,Y) is equal to geodesic distance of two points on Grassmann manifold [14].Based on Definition1, we give a kind of definition of distance on product Grassmann manifold which sums distance of each factor Grassmann manifold.Definition 3.
Given two points(X1,Y1) and (X2,Y2) on product Grassmann manifold G(p1,d1)×G(p2,d2), the distance between (X1,Y1) and (X2,Y2) is defined as(8)DsX1,Y1,X2,Y2=dsX1,X2+dsY1,Y2.


## 5. Extrinsic Least Squares Regression on Product Grassmann Manifold

Least squares regression is a simple and efficient technique in statistical analysis. In Euclidean space, parameterβ∈Rn×1 is estimated by minimizing the residual sum-of-square error (9)Rβ=y-Aβ2, where A∈Rn×k is training set and y∈Rn×1 is regression value. The estimated parameter has closed solution as (10)β^=ATA-1ATy. Hence the corresponding error is (11)y-AATA-1ATy2.In Grassmann manifold space, Lui [10] extended the linear least squares regression to a nonlinear form. In detail, the estimated parameter is equal to (12)A⋆A-1A⋆y, where ⋆ is a nonlinear similarity operator, A is a set of training samples on manifold, and y is an element on manifold. So the corresponding error is (13)dgy,A∘A⋆A-1A⋆y, where ∘ is an operator mapping points from vector space back to manifold. While Grassmann manifold is not closed under normal matrix subtraction and addition, the mapping ∘ is realized by employing exponential mapping and its inverse without closed-form solution. To realize the composition map ∘, an improved Karcher Mean Computation algorithm is employed. To avoid loss of the above iterative algorithm, we introduce an extrinsic least squares regression on Grassmann manifold by embedding its elements to space of symmetric matrices. Due to the distance on product Grassmann manifold in (8) being additive for each factor, the extrinsic least squares regression on product Grassmann manifold equals three independent subregression problems on each factor. Taking one factor as example, we show the details in the following.LetDj∈Gp,dj=1,2,…,N be training set where N is number of samples, and y=(yj)∈RN×1 is fitting parameter. X∈G(p,d) is regression value. Similar to the idea of least squares regression in Euclidean space, we give a regression on Grassmann manifold, which is defined in the embedded space of symmetric matrices. The residual is measured as follows:(14)Ry≜minyXXT-∑j=1NDjDjTyjF2,where yj is the jth element in vector y. Next we show how to solve the optimization. We have (15)XXT-∑j=1NDjDjTyjF2=TrXTXXTX+∑i,j=1NyiyjTrDiTDjDjTDi-2∑j=1NyjTrDjTXXTDjand we define (16)KDi,j=TrDiTDjDjTDi=DiTDjF2,KX,Dj=TrDjTXXTDj=XTDjF2. Hence model (14) becomes (17)miny⁡yTKDy-2yTKX,D.Let derivation of (17) with respect to y equal to 0; we have(18)KD+KDTy-2KX,D=0. So the solution of optimization (14) is(19)y∗=2KD+KDT-1KX,D.Hence the corresponding error becomes (20)XXT-∑j=1NDjDjTyj∗F.


## 6. Recognition Based on Extrinsic Least Squares Regression

In this subsection, we consider 3-order product Grassmann manifold(21)Gp1,d1×Gp2,d2×Gp3,d3 for videos, while the situation for higher order is similar. Suppose M classes are defined for the data. We denote training set corresponding with the kth class as Ujk,Vjk,Wjkj=1,2,…,Nk, where Nk is number of samples. Our objective is inferring to which class the test sample (X,Y,Z)∈G(p1,d1)×G(p2,d2)×G(p3,d3) belongs.The residual error of query sample(X,Y,Z) for class k is defined as(22)εk=XXT-∑j=1NkUjkUjkTαjk∗F+YYT-∑j=1NkVjkVjkTβjk∗F+ZZT-∑j=1NkWjkWjkTγjk∗F, where αjk∗,βjk∗,γjk∗(j=1,…,Nk) are solutions of subregression on each factor Grassmann manifold, respectively. The category of the query sample is determined by (23)k∗=arg⁡mink⁡εk.


## 7. Experiments on Different Datasets

In this section, we show performance of the proposed method against some state-of-the-art methods on two kinds of datasets.


### 7.1. Action Recognition

#### 7.1.1. Cambridge Hand Gesture Dataset

The Cambridge hand gesture dataset [20] contains 900 video sequences with nine kinds of hand gestures, which is divided into 5 sets according to different illuminations. Figure 1 shows some hand gesture samples. Set 5 (normal illumination) is considered for training while the remaining sequences (with different illumination characteristics) are used for testing. The original sequences are converted to grayscale and resized to 24×32×23. We denote our method as ELSR and report the correct recognition rate (CRR) for the four illumination sets in Table 1. Compared with product manifold (PM) [10], Grassmann Sparse Coding (gSC) [14], Grassmann Locality-Constrained Coding (gLC) [14], kernel Grassmann Sparse Coding (kgSC) [14], and kernel Grassmann Locality-Constrained Coding (kgLC) [14], we find that our method is competitive to these state-of-the-art methods.Table 1
Recognition results on the Cambridge hand gesture dataset.
Method Set 1 Set 2 Set 3 Set 4 Overall PM [10] 93 89 91 94 91.7 ± 2.3% gSC [14] 93 92 93 94 93.3 ± 0.9% gLC [14] 96 94 96 97 95.4 ± 1.3% kgSC [14] 96 92 93 97 94.4 ± 2.0% kgLC [14] 96 94 96 98 95.7 ± 1.6%  ELSR 98 94 96 96 95.7 ± 1.6%Figure 1
Cambridge hand gesture samples: (a) from left to right (flat-leftward; flat-rightward; flat-contract; spread-leftward; spread-rightward); (b) from left to right (spread-contract; V-shape-leftward; V-shape-rightward; V-shape-contract).
(a) (b)


#### 7.1.2. Ballet Dataset

44 videos are collected from a Ballet instruction DVD as the Ballet dataset [21]. In fact, 8 complex motion patterns from 3 persons are included in the dataset. In detail, the actions are “right-to-left hand opening,” “left-to-right hand opening,” “standing hand opening,” “jumping,” “leg swinging,” “hopping,” “turning,” and “standing still”. The main challenge of this dataset is large variations among classes such as speed, clothing, and motion paths. Figure 2 shows some examples of the dataset. Table 2 shows ELSR has superior performance compared with gSC-dic, gLC-dic, kgSC-dic, and kgLC-dic [14].Table 2
Correct recognition rate on the Ballet dataset.
Method CRR gSC-dic [14] 79.64 ± 1.1% gLC-dic [14] 81.42 ± 0.8% kgSC-dic [14] 83.53 ± 0.8% kgLC-dic [14] 86.94 ± 1.1%  ELSR 96.31 ± 1.7%Figure 2
Examples from the Ballet dataset.


### 7.2. Scene Analysis

For scene analysis, we use the UCSD traffic dataset [22] which contains 254 videos of highway traffic under different weather conditions. Resolution is 320×240 and number of frames ranges from 42 to 52. The dataset is divided into three classes(“heavy,” “medium,” and “light”) according to traffic congestion level. In total, there are 44 sequences defined as heavy traffic, 45 sequences labeled as medium traffic, and 165 sequences are light traffic. Figure 3 shows some typical examples. In experiment, we use the first 40 frames of each video and they are normalized as grayscale with resolution 48 × 48. We adopt the four pairs of training and testing sets provided in paper [23]. The classification results are shown in Table 3; the average correct recognition rate of ELSR is higher than that of gSC and gLC but lower than kgSC and kgLC.Table 3
Average correct recognition rate on the UCSD video traffic dataset.
Method Exp 1 Exp 2 Exp 3 Exp 4 Overall gSC [14] 93.7 87.5 95.3 95.2 92.9 ± 3.7% gLC [14] 96.8 85.9 92.2 93.7 92.2 ± 4.5% kgSC [14] 96.8 89.1 95.3 98.4 94.9 ± 4.1% kgLC [14] 95.2 92.2 96.9 96.8 95.3 ± 2.2%  ELSR 92.1 92.2 92.2 96.8 93.3 ± 2.3%Figure 3
Examples from the USCD dataset: “light”; “medium”; “heavy.”


### 7.3. Discussion

Through above experiments, we conclude that the proposed method is more effective for action recognition than scene analysis. In fact, the product Grassmann manifold could capture the appearance, horizontal motion, and vertical motion through three factor manifolds. To visualize the product manifold representation, the overlay appearance, horizontal motion, and vertical motion of examples from three dataset are given in Figure4. Note that there are obvious variation features along horizontal motion for hand gesture examples, both horizontal and vertical motion for Ballet examples. These curves in last two columns characterize the motion and are the key factors for recognizing. This can be seen as an explanation of the higher CRR result of ELSR on Ballet dataset. Meanwhile, for samples from UCSD, horizontal and vertical motion features are not clear because of all cars running along the same path, and the critical factor is appearance, characterizing the number of cars. Hence for UCSD dataset, the CRR of ELSR is just little higher than gSC and gLC, but lower than kgSC and kgLC which maps to higher-dimensional manifolds using kernel function to diminish nonlinearity.Figure 4
Examples of the overlay appearance, vertical and horizontal motion on three datasets.


## 7.1. Action Recognition

### 7.1.1. Cambridge Hand Gesture Dataset

The Cambridge hand gesture dataset [20] contains 900 video sequences with nine kinds of hand gestures, which is divided into 5 sets according to different illuminations. Figure 1 shows some hand gesture samples. Set 5 (normal illumination) is considered for training while the remaining sequences (with different illumination characteristics) are used for testing. The original sequences are converted to grayscale and resized to 24×32×23. We denote our method as ELSR and report the correct recognition rate (CRR) for the four illumination sets in Table 1. Compared with product manifold (PM) [10], Grassmann Sparse Coding (gSC) [14], Grassmann Locality-Constrained Coding (gLC) [14], kernel Grassmann Sparse Coding (kgSC) [14], and kernel Grassmann Locality-Constrained Coding (kgLC) [14], we find that our method is competitive to these state-of-the-art methods.Table 1
Recognition results on the Cambridge hand gesture dataset.
Method Set 1 Set 2 Set 3 Set 4 Overall PM [10] 93 89 91 94 91.7 ± 2.3% gSC [14] 93 92 93 94 93.3 ± 0.9% gLC [14] 96 94 96 97 95.4 ± 1.3% kgSC [14] 96 92 93 97 94.4 ± 2.0% kgLC [14] 96 94 96 98 95.7 ± 1.6%  ELSR 98 94 96 96 95.7 ± 1.6%Figure 1
Cambridge hand gesture samples: (a) from left to right (flat-leftward; flat-rightward; flat-contract; spread-leftward; spread-rightward); (b) from left to right (spread-contract; V-shape-leftward; V-shape-rightward; V-shape-contract).
(a) (b)


### 7.1.2. Ballet Dataset

44 videos are collected from a Ballet instruction DVD as the Ballet dataset [21]. In fact, 8 complex motion patterns from 3 persons are included in the dataset. In detail, the actions are “right-to-left hand opening,” “left-to-right hand opening,” “standing hand opening,” “jumping,” “leg swinging,” “hopping,” “turning,” and “standing still”. The main challenge of this dataset is large variations among classes such as speed, clothing, and motion paths. Figure 2 shows some examples of the dataset. Table 2 shows ELSR has superior performance compared with gSC-dic, gLC-dic, kgSC-dic, and kgLC-dic [14].Table 2
Correct recognition rate on the Ballet dataset.
Method CRR gSC-dic [14] 79.64 ± 1.1% gLC-dic [14] 81.42 ± 0.8% kgSC-dic [14] 83.53 ± 0.8% kgLC-dic [14] 86.94 ± 1.1%  ELSR 96.31 ± 1.7%Figure 2
Examples from the Ballet dataset.


## 7.1.1. Cambridge Hand Gesture Dataset

The Cambridge hand gesture dataset [20] contains 900 video sequences with nine kinds of hand gestures, which is divided into 5 sets according to different illuminations. Figure 1 shows some hand gesture samples. Set 5 (normal illumination) is considered for training while the remaining sequences (with different illumination characteristics) are used for testing. The original sequences are converted to grayscale and resized to 24×32×23. We denote our method as ELSR and report the correct recognition rate (CRR) for the four illumination sets in Table 1. Compared with product manifold (PM) [10], Grassmann Sparse Coding (gSC) [14], Grassmann Locality-Constrained Coding (gLC) [14], kernel Grassmann Sparse Coding (kgSC) [14], and kernel Grassmann Locality-Constrained Coding (kgLC) [14], we find that our method is competitive to these state-of-the-art methods.Table 1
Recognition results on the Cambridge hand gesture dataset.
Method Set 1 Set 2 Set 3 Set 4 Overall PM [10] 93 89 91 94 91.7 ± 2.3% gSC [14] 93 92 93 94 93.3 ± 0.9% gLC [14] 96 94 96 97 95.4 ± 1.3% kgSC [14] 96 92 93 97 94.4 ± 2.0% kgLC [14] 96 94 96 98 95.7 ± 1.6%  ELSR 98 94 96 96 95.7 ± 1.6%Figure 1
Cambridge hand gesture samples: (a) from left to right (flat-leftward; flat-rightward; flat-contract; spread-leftward; spread-rightward); (b) from left to right (spread-contract; V-shape-leftward; V-shape-rightward; V-shape-contract).
(a) (b)


## 7.1.2. Ballet Dataset

44 videos are collected from a Ballet instruction DVD as the Ballet dataset [21]. In fact, 8 complex motion patterns from 3 persons are included in the dataset. In detail, the actions are “right-to-left hand opening,” “left-to-right hand opening,” “standing hand opening,” “jumping,” “leg swinging,” “hopping,” “turning,” and “standing still”. The main challenge of this dataset is large variations among classes such as speed, clothing, and motion paths. Figure 2 shows some examples of the dataset. Table 2 shows ELSR has superior performance compared with gSC-dic, gLC-dic, kgSC-dic, and kgLC-dic [14].Table 2
Correct recognition rate on the Ballet dataset.
Method CRR gSC-dic [14] 79.64 ± 1.1% gLC-dic [14] 81.42 ± 0.8% kgSC-dic [14] 83.53 ± 0.8% kgLC-dic [14] 86.94 ± 1.1%  ELSR 96.31 ± 1.7%Figure 2
Examples from the Ballet dataset.


## 7.2. Scene Analysis

For scene analysis, we use the UCSD traffic dataset [22] which contains 254 videos of highway traffic under different weather conditions. Resolution is 320×240 and number of frames ranges from 42 to 52. The dataset is divided into three classes(“heavy,” “medium,” and “light”) according to traffic congestion level. In total, there are 44 sequences defined as heavy traffic, 45 sequences labeled as medium traffic, and 165 sequences are light traffic. Figure 3 shows some typical examples. In experiment, we use the first 40 frames of each video and they are normalized as grayscale with resolution 48 × 48. We adopt the four pairs of training and testing sets provided in paper [23]. The classification results are shown in Table 3; the average correct recognition rate of ELSR is higher than that of gSC and gLC but lower than kgSC and kgLC.Table 3
Average correct recognition rate on the UCSD video traffic dataset.
Method Exp 1 Exp 2 Exp 3 Exp 4 Overall gSC [14] 93.7 87.5 95.3 95.2 92.9 ± 3.7% gLC [14] 96.8 85.9 92.2 93.7 92.2 ± 4.5% kgSC [14] 96.8 89.1 95.3 98.4 94.9 ± 4.1% kgLC [14] 95.2 92.2 96.9 96.8 95.3 ± 2.2%  ELSR 92.1 92.2 92.2 96.8 93.3 ± 2.3%Figure 3
Examples from the USCD dataset: “light”; “medium”; “heavy.”


## 7.3. Discussion

Through above experiments, we conclude that the proposed method is more effective for action recognition than scene analysis. In fact, the product Grassmann manifold could capture the appearance, horizontal motion, and vertical motion through three factor manifolds. To visualize the product manifold representation, the overlay appearance, horizontal motion, and vertical motion of examples from three dataset are given in Figure4. Note that there are obvious variation features along horizontal motion for hand gesture examples, both horizontal and vertical motion for Ballet examples. These curves in last two columns characterize the motion and are the key factors for recognizing. This can be seen as an explanation of the higher CRR result of ELSR on Ballet dataset. Meanwhile, for samples from UCSD, horizontal and vertical motion features are not clear because of all cars running along the same path, and the critical factor is appearance, characterizing the number of cars. Hence for UCSD dataset, the CRR of ELSR is just little higher than gSC and gLC, but lower than kgSC and kgLC which maps to higher-dimensional manifolds using kernel function to diminish nonlinearity.Figure 4
Examples of the overlay appearance, vertical and horizontal motion on three datasets.


## 8. Performance Analysis

We analyze time complexity of inferring the label for a query(X,Y,Z)∈G(p1,d1)×G(p2,d2)×G(p3,d3) with given N training samples Uj,Vj,Wjj=1N∈G(p1,d1)×G(p2,d2)×G(p3,d3). The main computing steps include (19) and (22). We take one factor manifold for example. Some terms only related to Ujj=1N such as K(U) can be computed offline. For computing y∗, the time complexity of computing one element of K(X,U) is O(p12d1)+O(d1p1)+O(p12)=O(p12d1); then the complexity of vector K(X,U) is NO(p12d1), and hence the complexity of solving solution y∗ is O(N2)+NO(p12d1). Computing the error ϵ needs O(d1p1)+O(d12p1)+NO(d12)+O(d12)=NO(d12p1). Therefore the whole time complexity of our approach is ∑i=13ON2+NOpi2di+NOdi2pi. For small-scale dataset, our proposed method is effective and the time complexity will not be too large. For example in experiment of Cambridge hand gesture dataset: N=180, d1=32×23=736, d2=23×24=552, d3=24×32=768, p1=24, p2=32, and p3=23.


## 9. Conclusion

In this paper, we propose extrinsic least squares regression on product Grassmann manifold. Video can be viewed as third order tensor and then transformed to point on product Grassmann manifold factorized through HOSVD. One advantage of this method is the regression has closed-form solution which guides to a more accurate ratio of correct recognition. And when number of training samples is small, the proposed method is efficient. Several experiments on different recognition tasks (hand gesture recognition, action recognition, and scene analysis) show that our method performs very well on three small-scale public datasets.In future work, we would like to devise kernel version of extrinsic least squares regression on product manifold.



---

*Source: 6598025-2018-03-01.xml*