# 3D Skeletal Human Action Recognition Using a CNN Fusion Model

Authors: Meng Li; Qiumei Sun
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Smart homes have become central in the sustainability of buildings. Recognizing human activity in smart homes is the key tool to achieve home automation. Recently, two-stream Convolutional Neural Networks (CNNs) have shown promising performance for video-based human action recognition. However, such models cannot act directly on the 3D skeletal sequences due to its limitation to the 2D image video inputs. Considering the powerful effect of 3D skeletal data for describing human activity, in this study, we present a novel method to recognize the skeletal human activity in sustainable smart homes using a CNN fusion model. Our proposed method can represent the spatiotemporal information of each 3D skeletal sequence into three images and three image sequences through gray value encoding, referred to as skeletal trajectory shape images (STSIs) and skeletal pose image (SPI) sequences, and build a CNNs’ fusion model with three STSIs and three SPI sequences as input for skeletal activity recognition. Such three STSIs and three SPI sequences are, respectively, generated in three orthogonal planes as complementary to each other. The proposed CNN fusion model allows the hierarchical learning of spatiotemporal features, offering better action recognition performance. Experimental results on three public datasets show that our method outperforms the state-of-the-art methods.


---

## Body

## 1. Introduction

Smart homes are the most important technology of sustainable building. In the development of the next generation of smart homes, vision-based action analyzing methods are of key importance since they can make it possible for human occupants to interface with household appliances using just their physical actions instead of mouse, keyboard, touchscreen, or remote control devices. With the rapid development of the RGB-D sensors (e.g., Microsoft Kinect) and real-time full-body tracking with low cost [1, 2], 3D skeletal action analysis has drawn great attentions [3–5]. Compared to the conventional RGB cameras, the RGB-D sensors have the advantage of being reliable to estimate 3D skeletal joint positions of the human body. Different from RGB data, skeletal data is robust to different indoor environment settings. Hence, 3D skeletal action analysis methods are more suited to smart home technology than RGB-based methods. However, it is still a challenging task to efficiently learn proper features of 3D skeletal sequences in 3D human action analysis due to the large spatiotemporal variation in human actions. Recently, two-stream Convolutional Neural Networks (CNNs) have arisen as an advanced technique to recognize human actions in videos and achieve promising recognition performance [6]. Nevertheless, CNNs were proposed for 2D image video input, and it cannot work directly on the 3D skeletal action sequences.At the test stage, a skeletal sequence is represented by three STSIs and three SPI sequences (multiframe optical flow) that are passed through CNNF to extract spatial features and temporal features, respectively. These are then pooled across spatial streams and temporal streams and passed through CNNR to get compact descriptors. Finally, the skeletal action recognition accuracy is obtained by a class score fusion.In this paper, we propose a novel method consisting of 3D skeletal sequence mapping and CNN fusion model for skeletal action recognition. Using gray value encoding, the proposed skeletal sequence mapping projects each 3D skeletal action sequence into three image sequences and three images, referred to as skeletal trajectory shape images (STSIs) and skeletal pose image (SPI) sequences. Specifically, given a skeletal sequence, the 3D joint positions of each frame of the skeletal action sequence and the whole skeletal action sequence are encoded into sparse scatter points with gray values and specific size in each of the three orthogonal planes to generate three STSIs and three SPI sequences, respectively. Then, the three STSIs and three SPI sequences are fed into the proposed CNN fusion model for skeletal action recognition.Our proposed CNN fusion model is built based on the two-stream CNNs [6]. The two-stream CNNs employ two separate CNN streams, a spatial CNN stream, which extracts spatial features from 2D images, and a temporal CNN stream, which operates on optical flow information. Final action recognition accuracy is obtained by a late class score fusion. We extend the two-stream CNNs and introduce a new CNN fusion model that allows three STSIs and three SPI sequences as input to get spatiotemporal features of skeletal trajectory shape and skeletal pose sequence for skeletal action recognition. The overall flowchart of our approach is illustrated in Figure 1. Our main contributions include three aspects.(1)
We propose a novel CNN fusion model built based on a two-stream architecture with three SPI sequences and three STSIs as input to allow the hierarchical learning of spatiotemporal features of skeletal trajectory shape and skeletal pose sequence for skeletal human action recognition.(2)
We propose a 3D skeletal sequence mapping method to represent the spatiotemporal information carried in 3D skeletal sequences into three SPI sequences and three STSIs on three orthogonal planes by gray value encoding. The gray value encoding is introduced to project the 3D joint positions of each frame of the skeletal action sequence and the whole skeletal action sequence into sparse scatter points with gray value and specific size in each of the three orthogonal planes.(3)
The spatial pooling layer and temporal pooling layer are designed to obtain compact spatial features and temporal features in the two-stream architecture.Figure 1
CNN fusion model for skeletal action classification.


## 2. Background

Recently, deep learning methods have been applied on skeletal action recognition [7–9]. In the work by Du et al. [10], the skeletal joints are divided into five sets corresponding to five body parts. They are fed into five LSTMs for feature fusion and classification. In the Zhu et al. [11], the skeletal joints are fed to a deep LSTM at each time slot to learn the inherent co-occurrence features of skeletal joints. In the study by Shahroudy et al. [12], the long-term context representations of body parts are learned with a part-aware LSTM. In the study by Song et al. [13], both the spatial and temporal information of skeletal sequences are learned with a spatial-temporal LSTM. A Trust Gate is also proposed to remove noisy joints. However, RNNs tend to overemphasize the temporal information especially when the training data is insufficient, thus leading to overfitting [14].CNNs have also been applied to this problem [14, 15]. Hou et al. [14] proposed the Joint Trajectory Map (JTM), which represents both the spatial configuration and dynamics of joint trajectories into three texture images through color encoding, and then fed these texture images to CNNs for classification. However, each JTM just remains the 2D position information of each frame of the 3D skeletal sequence, which may lose some important information. Li et al. [15] proposed to encode the spatiotemporal information of skeletal sequences into joint distance maps (JDMs), and CNNs are employed to exploit the features from the JDMs for human action recognition. However, this method destroys the spatial structure of each skeletal pose. Pham et al. [16] proposed an Enhanced-SPMF map from skeletal data for the action recognition method; however, this method also partly lost the skeletal structure information of human poses.Different from all above methods, we propose a novel CNN fusion model that extends two-stream architecture [6] for 3D skeletal action recognition. Our CNN fusion model allows three STSIs and three SPI sequences as input to obtain spatiotemporal features of skeletal trajectory shape and skeletal pose sequence through hierarchical learning. Besides, our gray value encoding can capture the depth information of 3D skeletal joint positions. Hence, our model retains the 3D spatial structure of each human action instead of 2D position information.


## 3. Proposed Approach

Our proposed approach consists of 2 parts. (1) 3D skeletal sequence mapping. (2) A CNN fusion model for skeletal action classification.


### 3.1. 3D Skeletal Sequence Mapping

Suppose that each skeleton contains joints, and the illustration of an example skeleton with 15 joints is shown in Figure2. The 3D skeletal action A=A1,A2,...,AT is performed over T frames, where Ai=P1,P2,...,PN indicates the skeleton of the ith frame and pji represents the 3D coordinates of the jth joint in Ai.Figure 2
Skeletal joints captured by the depth sensor.For each skeletal actionA, the 3D joint positions of the whole skeletal action sequence A are encoded into sparse scatter points with specific size in each of three orthogonal planes to generate three STSIs, where the three orthogonal planes are defined by the real-world coordinate of the depth sensor. In addition, we propose to use the gray value to represent the depth information of sparse scatter points so that the sparse scatter points can reflect the 3D position information of the skeletal joint. For each skeletal joint pji, the gray value Gi,j according to its depth information is calculated as follows:(1)Gi,j=255Di,jmaxD,where Di,j denotes the depth value of pji and maxD denotes the maximum depth value of A. The depth value represents the distance from the skeletal joint to the corresponding projection plane. Through this process, the 3D position information of each pji is encoded, and the STSIs capture the 3D spatial distribution of joints of the skeletal action which are illustrated in Figure 3.Figure 3
STSIs in the three Cartesian planes for action “wave” in Florence3D-actions’ dataset: (a) the front (xy) plane; (b) the top (xz) plane; (c) the side (yz) plane.
(a)(b)(c)Similarly, for each frame of the skeletal action, the 3D joint positions ofAi are represented by SPI using gray value encoding. The multiframe optical flows of SPI sequences are illustrated in Figure 4.Figure 4
Multiframe optical flows of SPI sequences in the three Cartesian planes for action “wave” in Florence3D-actions’ dataset: (a) the front (xy) plane; (b) the top (xz) plane; (c) the side (yz) plane.
(a)(b)(c)


### 3.2. CNN Fusion Model

We design the CNN fusion model (Figure1) for skeletal action recognition. Each three STSIs are passed through the first part of the spatial stream CNNF separately, aggregated at a spatial-pooling layer, and then sent through the remaining part of the network CNNR. Similarly, the multiframe optical flows of each three SPI sequences are passed through the first part of the temporal stream CNNF separately, aggregated at a temporal-pooling layer, and then sent through the remaining part of the network CNNR. All branches in CNNR share the same parameters. For the spatial CNN stream and temporal CNN stream, we use element-wise maximum operation across the spatial CNN stream in the spatial-pooling layer and temporal CNN stream in the temporal-pooling layer. These two pooling layers should be placed close to the last convolutional layer for optimal action classification. For each branch, our CNN fusion model consists of mainly five convolutional layers followed by two fully connected layers and a softmax classification layer. The model is pretrained on ImageNet images and then fine-tuned on all skeletal action sequences in the training dataset.


## 3.1. 3D Skeletal Sequence Mapping

Suppose that each skeleton contains joints, and the illustration of an example skeleton with 15 joints is shown in Figure2. The 3D skeletal action A=A1,A2,...,AT is performed over T frames, where Ai=P1,P2,...,PN indicates the skeleton of the ith frame and pji represents the 3D coordinates of the jth joint in Ai.Figure 2
Skeletal joints captured by the depth sensor.For each skeletal actionA, the 3D joint positions of the whole skeletal action sequence A are encoded into sparse scatter points with specific size in each of three orthogonal planes to generate three STSIs, where the three orthogonal planes are defined by the real-world coordinate of the depth sensor. In addition, we propose to use the gray value to represent the depth information of sparse scatter points so that the sparse scatter points can reflect the 3D position information of the skeletal joint. For each skeletal joint pji, the gray value Gi,j according to its depth information is calculated as follows:(1)Gi,j=255Di,jmaxD,where Di,j denotes the depth value of pji and maxD denotes the maximum depth value of A. The depth value represents the distance from the skeletal joint to the corresponding projection plane. Through this process, the 3D position information of each pji is encoded, and the STSIs capture the 3D spatial distribution of joints of the skeletal action which are illustrated in Figure 3.Figure 3
STSIs in the three Cartesian planes for action “wave” in Florence3D-actions’ dataset: (a) the front (xy) plane; (b) the top (xz) plane; (c) the side (yz) plane.
(a)(b)(c)Similarly, for each frame of the skeletal action, the 3D joint positions ofAi are represented by SPI using gray value encoding. The multiframe optical flows of SPI sequences are illustrated in Figure 4.Figure 4
Multiframe optical flows of SPI sequences in the three Cartesian planes for action “wave” in Florence3D-actions’ dataset: (a) the front (xy) plane; (b) the top (xz) plane; (c) the side (yz) plane.
(a)(b)(c)


## 3.2. CNN Fusion Model

We design the CNN fusion model (Figure1) for skeletal action recognition. Each three STSIs are passed through the first part of the spatial stream CNNF separately, aggregated at a spatial-pooling layer, and then sent through the remaining part of the network CNNR. Similarly, the multiframe optical flows of each three SPI sequences are passed through the first part of the temporal stream CNNF separately, aggregated at a temporal-pooling layer, and then sent through the remaining part of the network CNNR. All branches in CNNR share the same parameters. For the spatial CNN stream and temporal CNN stream, we use element-wise maximum operation across the spatial CNN stream in the spatial-pooling layer and temporal CNN stream in the temporal-pooling layer. These two pooling layers should be placed close to the last convolutional layer for optimal action classification. For each branch, our CNN fusion model consists of mainly five convolutional layers followed by two fully connected layers and a softmax classification layer. The model is pretrained on ImageNet images and then fine-tuned on all skeletal action sequences in the training dataset.


## 4. Experiments

### 4.1. Experimental Datasets

Experiments were conducted on three public benchmark datasets: Florence3D-Action dataset [17], Toyota Smarthome dataset [18], and NTU RGB + D dataset [12].Florence3D-Action dataset has been captured using a Kinect camera. It includes 9 actions: wave, drink from a bottle, answer phone, clap, tight lace, sit down, stand up, read watch, and bow. During acquisition, 10 subjects were asked to perform the above actions. This resulted in a total of 215 action samples. The 3D positions of 15 skeletal joints are provided with the dataset. The 3D skeletal joints were obtained using the method of [1]. Sample frames of the Florence3D-Action dataset are shown in Figure 5.Figure 5
Sample frames of the Florence3D-Action dataset. The red points in each frame were plotted according to the 3D positions of skeletal joints obtained using the method of [1]. (a) Wave. (b) Drink from a bottle. (c) Bow. (d) Read watch. (e) Sit down. (f) Answer phone.
(a)(b)(c)(d)(e)(f)Toyota Smarthome dataset is a very challenging large dataset. It has been captured using 7 Kinect cameras. It includes 31 daily living actions: walk, drink from cup, sit down, read book, get up, watch TV, eat at table, stir, use telephone, enter, leave, use laptop, clean up, clean dishes, take pills, drink from bottle, pour from bottle, eat snack, lay down, cut, pour from kettle, use stove/oven, pour water, drink from glass, pour grains, boil water, pour from can, from can, insert teabag, use tablet, and cut bread. The subjects are senior people in the age range 60–80 years old. During acquisition, 18 subjects were aware of the recording but they were unaware of the purpose of the study. The videos were clipped per action, resulting in a total of 16,115 action samples. The 3D positions of 13 skeletal joints are provided with the dataset. The 3D skeletal joints were obtained using the method of [18]. Sample frames of the Toyota Smarthome dataset are shown in Figure 6. The NTU RGB + D dataset is a large-scale dataset for human action recognition (Figure 7). It includes 60 actions: drinking, eating, brushing teeth, brushing hair, dropping, picking up, throwing, sitting down, standing up, clapping, reading, writing, tearing up paper, wearing jacket, taking off jacket, wearing a shoe, taking off a shoe, wearing on glasses, taking off glasses, putting on a hat/cap, taking off a hat/cap, cheering up, hand waving, kicking something, reaching into self pocket, hopping, jumping up, making/answering a phone call, playing with phone, typing, pointing to something, taking selfie, checking time, rubbing two hands together, bowing, shaking head, wiping face, saluting, putting palms together, crossing hands in front, sneezing/coughing, staggering, falling down, touching head, touching chest, touching back, touching neck, vomiting, fanning self, punching/slapping other person, kicking other person, pushing other person, patting others back, pointing to the other person, hugging, giving something to other person, touching other persons pocket, handshaking, walking towards each other, and walking apart from each other. There are two protocols in this action dataset, i.e., Cross-Subject (CS) and Cross-View (CV). The 3D skeletal joints were obtained using the method of [12].Figure 6
Sample frames of the Toyota Smarthome dataset. The red points in each frame were plotted according to the 3D positions of skeletal joints obtained using the method of [18]. (a) Walk. (b) Lay down. (c) Read book. (d) Drink from cup. (e) Stir. (f) Get up.
(a)(b)(c)(d)(e)(f)Figure 7
Sample frames of the NTU RGB + D dataset. The red points in each frame were plotted according to the 3D positions of skeletal joints obtained using the method of [12]. We just select 13 skeletal joints of each pose for convenient viewing. (a) Brushing hair. (b) Dropping. (c) Wearing jacket. (d) Wearing jacket. (e) Patting others’ back. (f) Handshaking.
(a)(b)(c)(d)(e)(f)


### 4.2. Implementation Details

In our CNN fusion model, the locations of spatial and temporal pooling layers are placed to the fourth convolutional layers of both spatial and temporal streams. As in the study by Wang et al. [19], both the spatial stream and temporal streams of our CNN fusion model are pretrained on ILSVRC-2012. After obtaining the STSIs and SPI sequences, our CNN fusion model is fine-tuned on all skeletal human activity training sets. The images according to the input STSI and SPI sequences are resized to 256 × 256. The weights of our CNN fusion model are learned using the mini-batch stochastic gradient descent with the momentum being set to 0.7 and weight decay being set to 0.0003. The learning rate is initially set to 0.001, and then, it is set to 0.0001 for every 5,000 iterations and stops at 20,000 iterations.


### 4.3. Experimental Results

As shown in Tables1 and 2, for the Florence3D-Action dataset and the Toyota Smarthome dataset, the proposed approach achieves higher recognition accuracies than other existing methods since our proposed CNN fusion model can allow hierarchical learning to get spatiotemporal features of skeletal trajectory shape and skeletal pose sequence for skeletal action recognition. To allow a fair comparison, for the Florence3D-Action dataset, we followed the cross-subject test setting of [18], for the Toyota Smarthome dataset, we followed the cross-subject (CS) and two cross-view (CV1 and CV2) protocols of [18], and for the NTU RGB + D dataset, we followed the cross-subject (CS) and cross-view (CV) protocols of [12]. On the NTU RGB + D dataset, our method works very well although our method performs less well than DGNN for cross-view protocol. However, DGNN needs intensive computation, which may restrict its real application (Table 3).Table 1
Comparisons with other skeletal action recognition methods on the Florence 3D-Action dataset.
ApproachAccuracy (%)Multipart bag-of-poses [17] (2013)82.0Motion trajectories [20] (2015)87.0Lie group [21] (2014)90.9Bi-LSTM [22] (2018)93.0MIMTL [23] (2017)95.3Optical spectra [14] (2018)95.6JDM [15] (2017)96.1Proposed (spatial)97.3Proposed (temporal)95.1Proposed (fusion)98.2Table 2
Comparisons with other skeletal action recognition methods on the Toyota Smarthome dataset.
ApproachCSCV1CV2HMM [24] (2016)25.710.213.4DT [25] (2011)41.920.923.7LSTM [26] (2016)42.513.417.2I3D [27] (2017)53.434.945.1I3D + NL [28] (2018)53.634.343.9Optical spectra [14] (2018)53.731.243.5JDM [15] (2017)54.034.544.6Separable STA [18] (2019)54.235.250.3DPI [29] (2019)54.536.052.8Proposed (spatial)54.435.752.5Proposed (temporal)51.130.247.9Proposed (fusion)55.036.353.2Table 3
Comparisons with other skeletal action recognition methods on the NTU RGB + D Dataset.
ApproachCSCVLie group [21] (2014)50.152.8Part-aware LSTM [12] (2016)62.970.3ST-LSTM + Trust Gate [30] (2016)69.277.7STA-LSTM [31] (2017)73.481.2GCA-LSTM [32] (2017)74.482.8VA-LSTM [33] (2017)79.487.6SR-TSL [33] (2017)84.892.4HCN [34] (2018)86.591.12s-AGCN [35] (2019)88.595.1AGC-LSTM [36] (2019)89.295.0DGNN [37] (2019)89.996.1NAS-GCN [38] (2020)89.495.7Proposed (spatial)88.393.6Proposed (temporal)83.186.8Proposed (fusion)90.095.2Figure8 shows the confusion matrix for the Florence3D-Action dataset. We can see that our approach works very well. The classification confusions occur when the two actions are highly similar to each other like “drink from a bottle” and “answer phone” in the case of the Florence3D-Action dataset.Figure 8
Confusion matrix for the Florence3D-Action dataset.Figures9–11 show the confusion matrices for the Toyota Smarthome dataset. The Toyota Smarthome dataset contains 31 daily living actions. According to [18], we use 31 daily living actions for CS protocol and 19 daily living actions for CV1 and CV2 protocols. For each action sample in the Toyota Smarthome dataset, the subject performs actions without any information about how to perform it. Hence, the dataset consists of a rich variety of actions with huge intraclass differences. Although the dataset is very challenging, the confusion matrices show that our method can still get good effect for many types of actions.Figure 9
Confusion matrix for the Toyota Smarthome dataset in cross-subject (CS) protocol.Figure 10
Confusion matrix for the Toyota Smarthome dataset in cross-view 1 (CV1) protocol.Figure 11
Confusion matrix for the Toyota Smarthome dataset in cross-view 2 (CV2) protocol.


## 4.1. Experimental Datasets

Experiments were conducted on three public benchmark datasets: Florence3D-Action dataset [17], Toyota Smarthome dataset [18], and NTU RGB + D dataset [12].Florence3D-Action dataset has been captured using a Kinect camera. It includes 9 actions: wave, drink from a bottle, answer phone, clap, tight lace, sit down, stand up, read watch, and bow. During acquisition, 10 subjects were asked to perform the above actions. This resulted in a total of 215 action samples. The 3D positions of 15 skeletal joints are provided with the dataset. The 3D skeletal joints were obtained using the method of [1]. Sample frames of the Florence3D-Action dataset are shown in Figure 5.Figure 5
Sample frames of the Florence3D-Action dataset. The red points in each frame were plotted according to the 3D positions of skeletal joints obtained using the method of [1]. (a) Wave. (b) Drink from a bottle. (c) Bow. (d) Read watch. (e) Sit down. (f) Answer phone.
(a)(b)(c)(d)(e)(f)Toyota Smarthome dataset is a very challenging large dataset. It has been captured using 7 Kinect cameras. It includes 31 daily living actions: walk, drink from cup, sit down, read book, get up, watch TV, eat at table, stir, use telephone, enter, leave, use laptop, clean up, clean dishes, take pills, drink from bottle, pour from bottle, eat snack, lay down, cut, pour from kettle, use stove/oven, pour water, drink from glass, pour grains, boil water, pour from can, from can, insert teabag, use tablet, and cut bread. The subjects are senior people in the age range 60–80 years old. During acquisition, 18 subjects were aware of the recording but they were unaware of the purpose of the study. The videos were clipped per action, resulting in a total of 16,115 action samples. The 3D positions of 13 skeletal joints are provided with the dataset. The 3D skeletal joints were obtained using the method of [18]. Sample frames of the Toyota Smarthome dataset are shown in Figure 6. The NTU RGB + D dataset is a large-scale dataset for human action recognition (Figure 7). It includes 60 actions: drinking, eating, brushing teeth, brushing hair, dropping, picking up, throwing, sitting down, standing up, clapping, reading, writing, tearing up paper, wearing jacket, taking off jacket, wearing a shoe, taking off a shoe, wearing on glasses, taking off glasses, putting on a hat/cap, taking off a hat/cap, cheering up, hand waving, kicking something, reaching into self pocket, hopping, jumping up, making/answering a phone call, playing with phone, typing, pointing to something, taking selfie, checking time, rubbing two hands together, bowing, shaking head, wiping face, saluting, putting palms together, crossing hands in front, sneezing/coughing, staggering, falling down, touching head, touching chest, touching back, touching neck, vomiting, fanning self, punching/slapping other person, kicking other person, pushing other person, patting others back, pointing to the other person, hugging, giving something to other person, touching other persons pocket, handshaking, walking towards each other, and walking apart from each other. There are two protocols in this action dataset, i.e., Cross-Subject (CS) and Cross-View (CV). The 3D skeletal joints were obtained using the method of [12].Figure 6
Sample frames of the Toyota Smarthome dataset. The red points in each frame were plotted according to the 3D positions of skeletal joints obtained using the method of [18]. (a) Walk. (b) Lay down. (c) Read book. (d) Drink from cup. (e) Stir. (f) Get up.
(a)(b)(c)(d)(e)(f)Figure 7
Sample frames of the NTU RGB + D dataset. The red points in each frame were plotted according to the 3D positions of skeletal joints obtained using the method of [12]. We just select 13 skeletal joints of each pose for convenient viewing. (a) Brushing hair. (b) Dropping. (c) Wearing jacket. (d) Wearing jacket. (e) Patting others’ back. (f) Handshaking.
(a)(b)(c)(d)(e)(f)


## 4.2. Implementation Details

In our CNN fusion model, the locations of spatial and temporal pooling layers are placed to the fourth convolutional layers of both spatial and temporal streams. As in the study by Wang et al. [19], both the spatial stream and temporal streams of our CNN fusion model are pretrained on ILSVRC-2012. After obtaining the STSIs and SPI sequences, our CNN fusion model is fine-tuned on all skeletal human activity training sets. The images according to the input STSI and SPI sequences are resized to 256 × 256. The weights of our CNN fusion model are learned using the mini-batch stochastic gradient descent with the momentum being set to 0.7 and weight decay being set to 0.0003. The learning rate is initially set to 0.001, and then, it is set to 0.0001 for every 5,000 iterations and stops at 20,000 iterations.


## 4.3. Experimental Results

As shown in Tables1 and 2, for the Florence3D-Action dataset and the Toyota Smarthome dataset, the proposed approach achieves higher recognition accuracies than other existing methods since our proposed CNN fusion model can allow hierarchical learning to get spatiotemporal features of skeletal trajectory shape and skeletal pose sequence for skeletal action recognition. To allow a fair comparison, for the Florence3D-Action dataset, we followed the cross-subject test setting of [18], for the Toyota Smarthome dataset, we followed the cross-subject (CS) and two cross-view (CV1 and CV2) protocols of [18], and for the NTU RGB + D dataset, we followed the cross-subject (CS) and cross-view (CV) protocols of [12]. On the NTU RGB + D dataset, our method works very well although our method performs less well than DGNN for cross-view protocol. However, DGNN needs intensive computation, which may restrict its real application (Table 3).Table 1
Comparisons with other skeletal action recognition methods on the Florence 3D-Action dataset.
ApproachAccuracy (%)Multipart bag-of-poses [17] (2013)82.0Motion trajectories [20] (2015)87.0Lie group [21] (2014)90.9Bi-LSTM [22] (2018)93.0MIMTL [23] (2017)95.3Optical spectra [14] (2018)95.6JDM [15] (2017)96.1Proposed (spatial)97.3Proposed (temporal)95.1Proposed (fusion)98.2Table 2
Comparisons with other skeletal action recognition methods on the Toyota Smarthome dataset.
ApproachCSCV1CV2HMM [24] (2016)25.710.213.4DT [25] (2011)41.920.923.7LSTM [26] (2016)42.513.417.2I3D [27] (2017)53.434.945.1I3D + NL [28] (2018)53.634.343.9Optical spectra [14] (2018)53.731.243.5JDM [15] (2017)54.034.544.6Separable STA [18] (2019)54.235.250.3DPI [29] (2019)54.536.052.8Proposed (spatial)54.435.752.5Proposed (temporal)51.130.247.9Proposed (fusion)55.036.353.2Table 3
Comparisons with other skeletal action recognition methods on the NTU RGB + D Dataset.
ApproachCSCVLie group [21] (2014)50.152.8Part-aware LSTM [12] (2016)62.970.3ST-LSTM + Trust Gate [30] (2016)69.277.7STA-LSTM [31] (2017)73.481.2GCA-LSTM [32] (2017)74.482.8VA-LSTM [33] (2017)79.487.6SR-TSL [33] (2017)84.892.4HCN [34] (2018)86.591.12s-AGCN [35] (2019)88.595.1AGC-LSTM [36] (2019)89.295.0DGNN [37] (2019)89.996.1NAS-GCN [38] (2020)89.495.7Proposed (spatial)88.393.6Proposed (temporal)83.186.8Proposed (fusion)90.095.2Figure8 shows the confusion matrix for the Florence3D-Action dataset. We can see that our approach works very well. The classification confusions occur when the two actions are highly similar to each other like “drink from a bottle” and “answer phone” in the case of the Florence3D-Action dataset.Figure 8
Confusion matrix for the Florence3D-Action dataset.Figures9–11 show the confusion matrices for the Toyota Smarthome dataset. The Toyota Smarthome dataset contains 31 daily living actions. According to [18], we use 31 daily living actions for CS protocol and 19 daily living actions for CV1 and CV2 protocols. For each action sample in the Toyota Smarthome dataset, the subject performs actions without any information about how to perform it. Hence, the dataset consists of a rich variety of actions with huge intraclass differences. Although the dataset is very challenging, the confusion matrices show that our method can still get good effect for many types of actions.Figure 9
Confusion matrix for the Toyota Smarthome dataset in cross-subject (CS) protocol.Figure 10
Confusion matrix for the Toyota Smarthome dataset in cross-view 1 (CV1) protocol.Figure 11
Confusion matrix for the Toyota Smarthome dataset in cross-view 2 (CV2) protocol.


## 5. Conclusions and Future Work

In this paper, we have proposed a novel CNN fusion model for 3D human skeletal action recognition in smart homes. First, we propose the 3D skeletal sequence mapping to represent the spatiotemporal information of each 3D skeletal sequence into three images and three image sequences through gray value encoding, referred to as skeletal trajectory shape images (STSIs) and skeletal pose image (SPI) sequences. Then, we construct a CNNs fusion model with three STSIs and three SPI sequences as input for skeletal action recognition. Our experimental results have shown that our proposed approach has superior performance in comparison with several state-of-the-art methods on two public action datasets.In future work, we will consider introducing a new mechanism to deeply mine the human-object patterns of human-object actions in smart homes. It can improve the effectiveness of this model in real-world applications. In addition, the features surrounding 3D skeletal data should be captured for recognizing human-object actions. This will be another focus of our future research.



---

*Source: 6650632-2021-04-22.xml*