A graph convolutional neural network model with Fisher vector encoding and channel‐wise spatial‐temporal aggregation for skeleton‐based action recognition

IET Image ProcessingVolume 16, Issue 5 p. 1433-1443 ORIGINAL RESEARCH PAPEROpen Access A graph convolutional neural network model with Fisher vector encoding and channel-wise spatial-temporal aggregation for skeleton-based action recognition Jun Tang, Jun Tang College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China College of Animation and Communication, Qingdao Agricultural University, Qingdao, ChinaSearch for more papers by this authorYanjiang Wang, Corresponding Author Yanjiang Wang [email protected] orcid.org/0000-0001-9910-7884 College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China Correspondence College of Control Science and Engineering, China University of Petroleum (East China), Qingdao 266580, China. Email: [email protected]Search for more papers by this authorSichao Fu, Sichao Fu College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, ChinaSearch for more papers by this authorBaodi Liu, Baodi Liu College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, ChinaSearch for more papers by this authorWeifeng Liu, Weifeng Liu orcid.org/0000-0002-5388-9080 College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, ChinaSearch for more papers by this author Jun Tang, Jun Tang College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China College of Animation and Communication, Qingdao Agricultural University, Qingdao, ChinaSearch for more papers by this authorYanjiang Wang, Corresponding Author Yanjiang Wang [email protected] orcid.org/0000-0001-9910-7884 College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China Correspondence College of Control Science and Engineering, China University of Petroleum (East China), Qingdao 266580, China. Email: [email protected]Search for more papers by this authorSichao Fu, Sichao Fu College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, ChinaSearch for more papers by this authorBaodi Liu, Baodi Liu College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, ChinaSearch for more papers by this authorWeifeng Liu, Weifeng Liu orcid.org/0000-0002-5388-9080 College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, ChinaSearch for more papers by this author First published: 17 January 2022 https://doi.org/10.1049/ipr2.12422AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Skeleton-based action recognition is an inspired yet challenging task in computer vision. Recently, the latest graph convolutional network (GCN), which generalises well-established convolutional neural networks to non-Euclidean structures, is proven to be highly successful for action recognition from body skeleton data. However, the GCN architecture has not been fully studied. In this work, a Fisher vector (FV) encoding based GCN architecture (FV-GCN) is proposed, which exceeds the limitations of existing GCN-based methods by combining the GCN model with FV encoding. A channel-wise spatial–temporal aggregation function to preserve spatial–temporal information in the whole action clip and integrate it into the FV-GCN architecture is also presented. Since FV is different from the GCN structure, this hybrid architecture that incorporates the advantages of both algorithms can discover complementary information of feature representation effectively. On two challenging human action datasets, kinetics, and NTU-RGBD, improved performance is demonstrated over the baseline method, and the FV-GCN is better or comparable to some state-of-the-art methods. 1 INTRODUCTION Human action recognition has attracted the academic community since its wide applications in security analysis, multimedia content understanding, human-computer interaction, virtual reality, etc. Multiple modalities, such as RGB images [1-3], optical flows [4-6], depth images [7-9], and human skeleton data [10-12] are employed in action recognition. In contrast to other modalities, body skeleton, which can be obtained easily by the high-precision depth cameras and skeleton estimation algorithms [13, 14], are more robust to the complicated background. Action recognition from the skeleton data has evolved over the years from hand-crafted feature-based methods to the latest approaches based on deep learning. In earlier methods focusing on hand-crafted features, covariance matrices of joint trajectories [15], relative positions of joints [16], rotations and translations between body parts [17] are utilised. Encouraged by the success of deep learning, skeleton-based action recognition has utilised recurrent neural network (RNN) [11], [18-20], convolutional neural network (CNN) [21-23], and graph convolutional network(GCN) [24] to learn features via end-to-end learning, and proven highly successful. Over the past few years, Fisher vector (FV) and its variants [25-27], which aim to construct a high-level visual representation, have drawn much attention and been utilised in image classification [25], and video-based action recognition [28, 29] from optical flows or improved dense trajectories. However, FV has not been used in skeleton-based action recognition yet. Recently, inspired by the success of CNN, several studies [30-32] have started exploring the combination of FV and CNN in image classification. It has proven to be an effective method to boost performance. However, FV encoding has never been combined with the latest GCN or adopted in action recognition from skeleton data yet. The combination of GCN with FV encoding for action recognition from body skeleton data remains to be further studied. In this work, we propose an FV encoding based GCN architecture (FV-GCN) for action recognition from body skeleton data by combining the GCN with FV encoding. As illustrated in Figure 1, this architecture employs the feature maps learned by spatial–temporal graph convolutional networks (ST-GCN) [24], and aggregates them by channel-wise spatial–temporal aggregation function to obtain a compact feature. The distribution of the feature can be modelled by Gaussian mixture model (GMM). Thus we can further encode them to construct a high-level action representation by FV encoding followed by linear support vector machine (SVM). Since the FV structure is different from the GCN model, this hybrid architecture that incorporates the advantages of both algorithms can be effective in discovering complementary information of feature representations. FIGURE 1Open in figure viewerPowerPoint The pipeline of FV encoding based GCN (FV-GCN) architecture for skeleton-based action recognition The contributions of this work include: 1) We propose FV-GCN, a generic architecture to obtain more discriminative skeleton feature representation, which is the first that combines GCN with FV encoding for skeleton-based action recognition. 2) We present a channel-wise spatial–temporal aggregation function and integrate it into our FV-GCN architecture to further improve performance. 3) On two challenging human action datasets, we demonstrate improved performance over some state-of-the-art methods. This paper is an extension of our previous work [33]. The main extension and modifications of this paper include: 1) We present a channel-wise spatial–temporal aggregation function and integrate it into the FV-GCN architecture to further improve performance. 2) We analyse the distribution pattern of the FV features with various normalisation strategies and demonstrate ℓ 2 $\ell _2$ and power normalisation are effective in FV encoding for skeleton-based action recognition. 3) We add more experiments and exploration studies in several aspects. First, we evaluate the performance of FV-GCN with different spatial–temporal aggregation functions. Second, we compare various normalisation strategies individually or when combined. Third, we perform a detailed ablation study to evaluate the proposed components in FV-GCN. Finally, we provide confusion matrix analysis. The rest of this paper is arranged as follows. Section 2.1 reviews related works. We describe our proposed FV-GCN architecture in Section 3. Experimental results are evaluated in Section 4. Finally, conclusions are reported in Section 5. 2 RELATED WORKS GCN-based action recognition from skeleton data GCN, which generalise well-established CNN to arbitrarily structured graphs, have successfully been applied in lots of applications, such as image classification [34], document classification [35], and semi-supervised learning [36]. Inspired by the fact that the human skeleton data can be considered as a graph rather than a sequence vector or pseudo-image that was adopted in RNN-based or CNN-based methods, recently the graph convolution network (GCN) has been utilised in skeleton-based action recognition. ST-GCN [24] is the first to employs GCN for action recognition from skeleton data, which constructs a spatial–temporal graph on skeleton joints and temporal relations. In ref. [24], the body skeleton data are firstly fed into graph convolution layers. Then the extracted high-level features are classified by SoftMax. This ST-GCN model is employed as the front-end part of our FV-GCN architecture. Following this work, Liu et al. [37] aimed to solve the problem of redundant dependencies between different neighbourhood nodes, and proposed a unified spatial–temporal graph convolutional operator for effective feature learning. Cheng et al. [38] introduced the decoupling aggregation mechanism into the skeleton-based action recognition task, and proposed decoupling GCN with less computation cost. Since the graph of GCN is set fixed and the second-order information of the skeleton data was rarely utilised in previous works, Shi et al. [39] proposed a multi-stream attention-enhanced adaptive graph convolutional neural network to address these issues. The combination of CNN and Fisher vector Despite their success, the CNN architecture has not been fully explored. Recently, several studies have started exploring the combination of CNN and FV encoding in image classification, which has proven to be an effective method to improve the performance of CNN in image classification. Simonyan et al. [30] defined a stacked Fisher vector architecture for image classification. Cimpoi et al. [31] presented a texture descriptor by combining FV and CNN. Tang et al. [40] proposed a deep FisherNet architecture for object classification and emotion image classification. Diba et al. [41] proposed an iterative framework to obtained improved FV features for the image classification task. Yongsheng Pan et al. [32] proposed the foreground FV encoding algorithm for the image classification task. However, these methods have never adopted the latest GCN or been utilised in action recognition from skeleton data yet. The combination of GCN with FV encoding for action recognition from body skeleton data remains to be further studied. 3 FV ENCODING BASED GCN ARCHITECTURE Figure 1 shows the pipeline of FV encoding based GCN (FV-GCN) architecture. This architecture is composed of five steps: (i) feature extraction, (ii) temporal aggregation, (iii) GMM generation, (iv) feature encoding, and (v) normalisation. The high-level representation is finally fed into the linear SVM. We provide detailed descriptions of each step in this section. Feature extraction The ST-GCN model [24] is employed to extract features in our FV-GCN architecture. The ST-GCN model consists of nine layers of spatial–temporal graph convolution. The last layer has 256 channels for output. Then a pooling layer is conducted to obtain 256 dimension features. We extract features from the last convolution layer of the ST-GCN. Channel-wise spatial–temporal aggregation Channel-wise spatial–temporal aggregation is an important component in our FV-GCN architecture. Consider the output feature maps generated from the last convolutional layer of ST-GCN for an action clip. The feature maps is a tensor S ∈ R T × N 1 × C $S\in R^{T\times N_1\times C}$ where T , N 1 $T, N_1$ , and C $C$ are the temporal depth, the number of skeleton joints, and the number of channels of the feature maps, respectively. To obtain the spatial–temporal information from the entire action clip, the input feature maps S $S$ are firstly split into N 1 $N_1$ slices { S 1 , S 2 , ⋯ , S N 1 } , S i ∈ R T × C $\lbrace S_1,S_2,\dots ,S_{N_1}\rbrace ,S_i\in R^{T\times C}$ along the given skeleton joints dimension N 1 $N_1$ . The N 1 $N_1$ feature maps are then aggregated to S 1 ∈ R T × N × C $S1\in R^{T\times N\times C}$ by the aggregation operator in the spatial domain. The spatial aggregated feature maps S 1 $S1$ are further split into T $T$ slices { S 1 , S 2 , ⋯ , S T } , S i ∈ R N × C $\lbrace S_1,S_2,\dots ,S_T\rbrace ,S_i\in R^{N\times C}$ along the temporal dimension T $T$ . A temporal aggregation operator finally aggregates the T $T$ temporal feature maps to output a spatial–temporal aggregated feature map Y $Y$ . Algorithm 1 illustrates the channel-wise spatial–temporal aggregation component in our FV-GCN. This channel-wise spatial–temporal aggregation component aims to aggregate spatial information in the whole action clip and preserves the temporal relations among all the slices. ALGORITHM 1. Channel-wise Spatial–temporal aggregation function Input: ST-GCN features S $S$ of an action clip V , S ∈ R T × N 1 × C $V, S\in R^{T\times N_1\times C}$ where T , N 1 $T, N_1$ , and C $C$ are the temporal depth, the number of skeleton joints, and the number of channels of the feature maps, respectively. Output: spatial–temporal aggregated feature maps Y ∈ R N × C $Y\in R^{N\times C}$ . 1: Split S $S$ into N 1 $N_1$ slices { S 1 , S 2 , ⋯ , S N 1 } , S i ∈ R T × C $\lbrace S_1,S_2,\dots ,S_{N_1}\rbrace ,S_i\in R^{T\times C}$ along the spatial dimension N 1 $N_1$ . 2: S 1 = S 1 ⋄ S 2 ⋄ ⋯ ⋄ S N 1 , S 1 ∈ R T × N × C $S1=S_1\diamond S_2\diamond \dots \diamond S_{N_1}, S1\in R^{T\times N\times C}$ , where ⋄ $\diamond$ is an aggregation operator. 3: Split S 1 $S1$ into T $T$ slices { S 1 , S 2 , ⋯ , S T } , S i ∈ R N × C $\lbrace S_1,S_2,\dots ,S_T\rbrace ,S_i\in R^{N\times C}$ along the temporal dimension T $T$ . 4: Y = S 1 ⋄ S 2 ⋄ ⋯ ⋄ S T , Y ∈ R N × C $Y=S_1\diamond S_2\diamond \dots \diamond S_T, Y\in R^{N\times C}$ , where ⋄ $\diamond$ is an aggregation operator. We propose different aggregation operators for the spatial–temporal slices. Element-wise average of slices S i { i = 1 ⋯ M , M = N 1 or M = T } $S_i\lbrace i=1\dots M, M = N_1\text{ or }M = T\rbrace$ : Y = ( S 1 ⊕ S 2 ⊕ ⋯ ⊕ S T ) / M \begin{equation} Y=(S_1\oplus S_2\oplus \dots \oplus S_T)/M \end{equation} (1) Element-wise maximum of slices S i { i = 1 ⋯ M , M = N 1 or M = T } $S_i\lbrace i=1\dots M, M = N_1\text{ or }M = T\rbrace$ : Y = max { S 1 , S 2 , ⋯ , S M } \begin{equation} Y=\text{max}\lbrace S_1, S_2, \dots , S_M\rbrace \end{equation} (2) Element-wise multiplication of slices S i { i = 1 ⋯ M , M = N 1 or M = T } $S_i\lbrace i=1\dots M, M = N_1\text{ or }M = T\rbrace$ : Y = S 1 ∘ S 2 ∘ ⋯ ∘ S M \begin{equation} Y= S_1\circ S_2\circ \dots \circ S_M \end{equation} (3) Our channel-wise spatial–temporal aggregation is inspired by ref. [42]. But there are significant differences. The key distinction between our approach and ref. [42] is that our aggregation functions are applied in both spatial and temporal domain. Additionally, our approach is applied in channels of feature maps rather than CNN features for K $K$ segments from a video in ref. [42]. GMM generation Gaussian Mixture Model (GMM) [43] μ λ $\mu _\lambda$ is a probabilistic model, which aims to model the feature space as a mixture of Gaussian probability distributions. It contains not only the average information of feature space but also the covariance of their distribution: u λ ( x ) = ∑ i = 1 K ω i u i ( x ) \begin{equation} u_\lambda (x)= \sum _{i=1}^{K}\omega _iu_i(x) \end{equation} (4)where K $K$ is the mixture number of Gaussians, and λ = { ω i , μ i , δ i , i = 1 , ⋯ , K } $\lambda =\lbrace \omega _i, \mu _i,\delta _i, i=1, \dots , K \rbrace$ where ω i ∈ [ 0 , 1 ] , μ i ∈ R D $\omega _i \in [0,1], \mu _i\in R^D$ , and δ i ∈ R D $\delta _i\in R^D$ are the mixture weight, mean vector and diagonal covariance vectors of Gaussian component u i $u_i$ respectively. The spatial–temporal aggregated feature map Y ∈ R N × C $Y\in R^{N\times C}$ is further split into N $N$ slices X = { x i ∈ R C , i = 1 ⋯ N } $X=\lbrace x_i \in R^C, i=1 \dots N\rbrace$ along the given dimension N $N$ . Given the training set of X $X$ , the expectation–maximisation (EM) algorithm [44] is adopted to estimate the parameters of GMM. Then we can describe the action clip V $V$ as gradient vectors for the model parameters [45]: G λ x = 1 N ∑ n = 1 N ▽ λ log u λ ( x n ) . \begin{equation} G^x _\lambda =\frac{1}{N}\sum _{n=1}^N \bigtriangledown _\lambda \log u_\lambda (x_n). \end{equation} (5) The number of parameters λ $\lambda$ determines the dimension of G λ x $G^x _\lambda$ . Feature encoding Fisher vector (FV) is a powerful encoding algorithm inspired by the Fisher kernel [45]. Since it can provide high-level visual representation for better performance, Fisher vector has been successfully utilised in image classification and video-based action recognition tasks. Let G μ , i x $\mathcal {G}_{\mu ,i} ^x$ be the D $D$ -dimensional gradient of μ i $\mu _i$ and G δ , i x $\mathcal {G}_{\delta ,i} ^x$ be the D $D$ -dimensional gradient of the standard deviation δ i $\delta _i$ of Gaussian i $i$ , respectively. G μ , i x = 1 N ω i ∑ n = 1 N γ n ( i ) ( x n − μ i δ i ) \begin{equation} \mathcal {G}^x _{\mu ,i} =\frac{1}{N\sqrt {\omega _i}}\sum _{n=1}^N \gamma _n(i) (\frac{x_n-\mu _i}{\delta _i}) \end{equation} (6) G δ , i x = 1 N 2 ω i ∑ n = 1 N γ n ( i ) [ ( x n − μ i ) 2 δ i 2 − 1 ] \begin{equation} \mathcal {G}^x _{\delta ,i} =\frac{1}{N\sqrt {2\omega _i}}\sum _{n=1}^N \gamma _n(i) [\frac{(x_n-\mu _i)^2}{\delta _i^2}-1] \end{equation} (7)where γ n ( i ) $\gamma _n(i)$ is the soft assignment of descriptor x n $x_n$ to Gaussian i $i$ : γ n ( i ) = ω i μ i ( x n ) ∑ j = 1 K ω j μ j ( x n ) . \begin{equation} \gamma _n(i) =\frac{\omega _i\mu _i(x_n)}{\sum _{j=1}^K \omega _j\mu _j(x_n)}. \end{equation} (8) We can concatenate the G μ , i x $\mathcal {G} ^x _{\mu ,i}$ and G δ , i x ( i = 1 ⋯ K ) $\mathcal {G}^x _{\delta ,i} (i=1\dots K)$ gradients vectors to construct the final Fisher vector: F V : G λ x = [ G μ , 1 x , G δ , 1 x , ⋯ , G μ , k x , G δ , k x ] . \begin{equation} FV: \mathcal {G}^x _{\lambda } =[\mathcal {G}^x _{\mu ,1},\mathcal {G}^x _{\delta ,1},\dots ,\mathcal {G}^x _{\mu ,k},\mathcal {G}^x _{\delta ,k} ]. \end{equation} (9) Normalisation The result G λ x $\mathcal {G}^x _{\lambda }$ is further normalised by two normalisation steps [46], which were necessary to boost the classification accuracy. 3.5.1 ℓ 2 $\ell _2$ normalisation Since the descriptors of images can be decomposed into a background part and an image-specific part, [46] ℓ 2 $\ell _2$ -normalise the vector G λ x $\mathcal {G}^x _{\lambda }$ to eliminate the impact of the background part contained in the image. Note that the skeleton feature is different from image data, which cannot be decomposed into background part and image-specific part. Since the skeleton joints of different body parts have different importance concerning activities, we follow ref. [46] and decompose the skeleton features G λ x $\mathcal {G}^x _{\lambda }$ into a minor part and an important part. We assume that the features X = { x i ∈ R C , i = 1 ⋯ N } $X=\lbrace x_i \in R^C, i=1 \dots N\rbrace$ of the given action clip V $V$ follow a distribution p $p$ . By the law of large numbers, Equation (5) can be rewritten as: G λ x ≈ ▽ λ E x ∼ p log u λ ( x ) = ▽ λ ∫ x p ( x ) log u λ ( x ) d x . \begin{equation} G^x _\lambda \approx \bigtriangledown _\lambda \mathbb {E}_{x\sim p} \log u_\lambda (x) = \bigtriangledown _\lambda \int _x p(x) \log u_\lambda (x) \, \text{d}x. \end{equation} (10) Assuming that the distribution p $p$ consist of two parts: a minor part and an important part. These parts follow distribution μ λ $\mu _\lambda$ and q $q$ , respectively. Let 0 ≤ ω ≤ 1 $0 \le \omega \le 1$ be the proportion of the critical information contained in the action clip: p ( x ) = ω q ( x ) + ( 1 − ω ) μ λ ( x ) . \begin{equation} p(x) = \omega q(x) + (1- \omega ) \mu _\lambda (x). \end{equation} (11) We can rewrite Equation (10) as: G λ x ≈ ω ▽ λ ∫ x q ( x ) log u λ ( x ) d x + ( 1 − ω ) ▽ λ ∫ x μ λ ( x ) log u λ ( x ) d x . \begin{eqnarray} G^x _\lambda && \approx \omega \bigtriangledown _\lambda \int _x q(x) \log u_\lambda (x) \, \text{d}x\nonumber \\ && + (1- \omega ) \bigtriangledown _\lambda \int _x \mu _\lambda (x)\log u_\lambda (x) \, \text{d}x. \end{eqnarray} (12) Since the parameters λ $\lambda$ were estimated through maximum likelihood estimation, which aims to maximise E x ∼ p log u λ ( x ) $\mathbb {E}_{x\sim p} \log u_\lambda (x)$ , we have: ▽ λ ∫ x μ λ ( x ) log u λ ( x ) d x = ▽ λ E x ∼ μ λ log u λ ( x ) ≈ 0 . \begin{equation} \bigtriangledown _\lambda \int _x \mu _\lambda (x)\log u_\lambda (x) \, \text{d}x = \bigtriangledown _\lambda \mathbb {E}_{x\sim \mu _\lambda }\log u_\lambda (x) \approx 0. \end{equation} (13) Then Equation (12) can be rewritten as: G λ x ≈ ω ▽ λ ∫ x q ( x ) log u λ ( x ) d x = ω ▽ λ E x ∼ μ λ log u λ ( x ) . \begin{eqnarray} G^x _\lambda && \approx \omega \bigtriangledown _\lambda \int _x q(x) \log u_\lambda (x) \, \text{d}x \nonumber \\ && = \omega \bigtriangledown _\lambda \mathbb {E}_{x\sim \mu _\lambda }\log u_\lambda (x). \end{eqnarray} (14) According to Equation (14), the results still depend on the proportion of the important information ω $\omega$ contained in the action clip. Therefore two action clips of the same class with different amounts of minor information will have different results. To eliminate the impact of ω $\omega$ , the feature G λ x $\mathcal {G}^x _{\lambda }$ is divided by its ℓ 2 $\ell _2$ normalisation: G λ x = G λ x ∥ G λ x ∥ 2 . \begin{equation} \mathcal {G}^x _{\lambda } = \frac{\mathcal {G}^x _{\lambda }}{\Vert \mathcal {G}^x _{\lambda } \Vert _2 }. \end{equation} (15) 3.5.2 Power normalisation Perronnin et al. [46] demonstrated that, as the number of Gaussians increases, the Fisher vectors become sparser, negatively impacting the dot-product. Unlike ref. [46], we observe that our best results are achieved when the number of Gaussians K $K$ equals 1 for all the datasets in Section 4.3. The dot-product of the ℓ 2 $\ell _2$ normalised vectors is equivalent to the ℓ 2 $\ell _2$ distance. Furthermore, the Fisher vectors are sparse even with the number of Gaussians K $K$ equals 1 shown in Figure 2. Since the ℓ 2 $\ell _2$ distance is an inadequate method for measuring sparse vectors, following [46], power normalisation are applied in each dimension for i = 1 ⋯ 2 K D $i=1\dots 2KD$ : [ G λ x ] i ← sign ( [ G λ x ] i ) | [ G λ x ] i | . \begin{equation} [\mathcal {G}^x _{\lambda } ]_i \leftarrow \text{sign}([\mathcal {G}^x _{\lambda } ]_i) \sqrt { \vert [\mathcal {G}^x _{\lambda } ] _i \vert }. \end{equation} (16) FIGURE 2Open in figure viewerPowerPoint Distribution of the values in some dimensions of the normalised Fisher vector with the number of Gaussians K $K$ equals 1. (a) the 29th dimension, (b) the 78th dimension, and (c) the 232th dimension with only ℓ 2 $\ell _2$ normalisation. (d) The 29th dimension, (e) the 78th dimension, and (f) the 232th dimension with ℓ 2 $\ell _2$ and power normalisation. We estimate these histograms on the 40,320 training skeleton samples of the NTU-RGB+D dataset in cross-subject evaluation The power normalisation is performed first and followed by the ℓ 2 $\ell _2$ normalisation when these two normalisation methods are combined. The effect of power normalisation is shown in Figure 2. We observe that the distribution of ℓ 2 $\ell _2$ normalised FV features in lots of dimensions is crowded around zero. The power normalisation effectively alleviates the sparse trouble of the ℓ 2 $\ell _2$ normalised skeleton features, making it more suitable for dot-product operation. 4 EXPERIMENTS This section evaluates our FV-GCN on two challenging datasets: Kinetics [47] and NTU-RGB+D [18]. More specifically, we first investigate different channel-wise spatial–temporal aggregation functions. We next evaluate the number of Gaussians k and select a proper GMM size. Then we compare different normalisation strategies individually or when combined. To evaluate the proposed components in FV-GCN, we perform a detailed ablation study. We also provide confusion matrix analysis in this section. Finally, we compare our FV-GCN with state-of-the-art methods. In our experiments, FV-GCN uses one-vs-rest linear SVM classifiers with squared hinge loss. Dataset and evaluation metrics The DeepMind Kinetics dataset consists of more than 300,000 realistic action video samples, covering 400 human action classes. Each action class has more than 400 samples. And each sample lasts around 10 s. Since there are only raw videos in the kinetics dataset, [24] estimates the 2D coordinates of joints on frames in the video samples by the OpenPose tool [14]. Each joint located by OpenPose is in the form of ( X , Y , C ) $(X,Y,C)$ , where X $X$ and Y $Y$ are 2 D $2D$ coordinates in the pixel coordinate system, and C $C$ is the confidence value, respectively. The training and validation sets contain 240,000 and 20,000 clips, respectively. The NTU-RGB+D dataset consists of 56,880 action samples and 4 million frames. These samples are collected from 40 volunteers and captured by Microsoft Kinect v2 in different camera settings. The dataset contains 60 different action classes, which are composed of three groups: 40 daily activities (e.g. brushing teeth, sitting down, and clapping), nine health-related activities (e.g. touching head, nausea or vomiting condition, and touching chest), and 11 mutual activities (e.g. patting on back of another person, giving something to another person, and touching other person's pocket). The dataset provides 3D coordinates of 25 joints in the form of ( X , Y , Z ) $(X, Y, Z)$ for each subject. The authors of this dataset recommend two benchmarks: 1) cross-subject (X-sub): the 40 volunteers are divided equally into training and validation sets. The training set consists of 40,320 clips acted by the former 20 volunteers. The validation set consists of 16,560 clips acted by the latter 20 volunteers. 2) cross-view (X-view): the training set contains 37,920 clips captured by cameras 2 and 3 with front and side views of the activities. The testing set contains 18,960 clips captured by camera 1 with 45 degrees views of the action performances. Evaluation on channel-wise spatial–temporal aggregation functions In this work, we evaluate three channel-wise spatial–temporal aggregation functions (i) element-wise average, (ii) element-wise maximum, and (iii) element-wise multiplication. We evaluate the performance of FV-GCN with these aggregations functions. The results are summarised in Table 1. Since the element-wise average function outperforms the other two functions, we set it as the default option. This channel-wise spatial–temporal aggregation preserves the spatial–temporal information among all the slices. TABLE 1. Performance comparison of different channel-wise spatial–temporal aggregation functions of our FV-GCN on Kinetics and NTU-RGB+D datasets. We report the accuracies of both the cross-subject (X-sub) and cross-view (X-view) benchmarks of the NTU-RGB+D dataset Spatial