A 3D graph convolutional networks model for 2D skeleton‐based human action recognition

IET Image ProcessingVolume 17, Issue 3 p. 773-783 ORIGINAL RESEARCHOpen Access A 3D graph convolutional networks model for 2D skeleton-based human action recognition Libo Weng, Libo Weng College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Contribution: ​Investigation, Methodology, Software, Writing - original draftSearch for more papers by this authorWeidong Lou, Weidong Lou College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Contribution: Software, Validation, Writing - review & editingSearch for more papers by this authorXin Shen, Xin Shen College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Contribution: Data curation, Software, Writing - review & editingSearch for more papers by this authorFei Gao, Corresponding Author Fei Gao [email protected] orcid.org/0000-0003-1209-0608 College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Correspondence Fei Gao, College of Computer and Science, Zhejiang University of Technology, Hangzhou 310023, China. Email: [email protected] Contribution: Funding acquisition, Project administration, SupervisionSearch for more papers by this author Libo Weng, Libo Weng College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Contribution: ​Investigation, Methodology, Software, Writing - original draftSearch for more papers by this authorWeidong Lou, Weidong Lou College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Contribution: Software, Validation, Writing - review & editingSearch for more papers by this authorXin Shen, Xin Shen College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Contribution: Data curation, Software, Writing - review & editingSearch for more papers by this authorFei Gao, Corresponding Author Fei Gao [email protected] orcid.org/0000-0003-1209-0608 College of Computer and Science, Zhejiang University of Technology, Hangzhou, China Correspondence Fei Gao, College of Computer and Science, Zhejiang University of Technology, Hangzhou 310023, China. Email: [email protected] Contribution: Funding acquisition, Project administration, SupervisionSearch for more papers by this author First published: 25 October 2022 https://doi.org/10.1049/ipr2.12671AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract With the popularity of cameras, the application of action recognition is more and more extensive. After the emergence of RGB-D cameras and human pose estimation algorithms, human actions can be represented by a sequence of skeleton joints. Therefore, skeleton-based action recognition has been a research hotspot. In this paper, a novel 3D Graph Convolutional Network model (3D-GCN) with space-time attention mechanism for 2D skeleton data is proposed. Three-dimensional graph convolution is employed to extract spatiotemporal features of skeleton descriptor that is composed of joint coordinates, frame differences and angles. Meanwhile, different joints and different frames are given different attention to achieve action classification. A zebra crossing pedestrian dataset named ZCP is also provided, which simulates possible pedestrian actions on the zebra crossing in real scenes. Experimental evaluation is carried out on ZCP dataset and NTU RGB+D dataset. Experimental results show that our method is better than current 2D-based methods and is comparable with 3D methods. 1 INTRODUCTION The task of human action recognition is to correctly classify human action sequences into corresponding actions. It is one of the research hotspots in the field of artificial intelligence and is widely used in intelligent video surveillance, network video description, human–computer interaction, virtual reality games, sports and rehabilitation guidance [1, 2]. In the past decades, human action recognition based on 2D RGB video sequences has been extensively studied [3], and those methods mainly use appearance or motion analysis. With the development of low-cost RGB-D sensors, depth cameras (e.g. Microsoft Kinect [4]) began to appear, so depth information and 3D skeleton data of human body can be readily obtained. Compared with traditional 2D RGB image data, RGB-D human skeleton data is not easily affected by illumination changes, and it is robust for occlusion and background confusion. Although RGB-D camera can run smoothly in a controllable indoor environment, it may be unstable in outdoor. Due to many uncontrollable factors (such as solar radiation and ferromagnetic radiation) in the complex outdoor environment, depth sensor (such as structured light, flying time or stereo vision) will produce a lot of noises [5, 6]. Moreover, its shooting distance is limited to 10 m, which will result in greater limitations for 3D skeleton. In contrast, 2D cameras develop so far and are more stable and reliable. At the same time, reliable 2D skeleton information can be obtained only by monocular RGB camera [7]. Therefore, applicability of 2D skeleton data based on human pose estimation algorithm is more extensive. In this paper, identifying human action from 2D skeleton is focused on. Current 2D skeleton-based human action recognition methods have disadvantages such as insufficient experiments, few action categories, lack of experimental evaluation on public datasets, or insufficient utilization of skeleton space structure. Most of the 2D skeleton-based methods are used for specific scenes and normally with very few action categories, resulting in a lack of experiments on the public datasets. Therefore, a new dataset focuses on the pedestrian passing zebra crossings in the traffic scene is proposed and a novel model named 3D graph convolution (3D-GCN) for 2D skeleton-based human action recognition is proposed, which combines 3D CNN and GCN to classify 2D skeleton action sequences. The main contributions of this paper are as follows: As for skeleton data representation, normalized coordinates, frame difference and joint angle are selected to construct skeleton descriptor. Normalized coordinates capture the relationship between body parts, frame difference represents the motion of skeleton and joint angle calculation can better describe the static pose of adjacent body parts. A novel 3D-GCN model is proposed to better learn spatiotemporal features simultaneously. Through combining graph convolution and 3D convolution, 3D-GCN not only has the exploration ability of graph convolution in space but also has the exploration ability of 3D convolution in time. A temporal and spatial attention model is presented. Different importance degrees are assigned to different joints of the same frame through a trainable space attention matrix. Meanwhile, importance degrees of different frames are obtained through the convolution after maximum pooling and average pooling, and finally the network recognition effect is further improved. 2 RELATED WORK According to the input data form, human action recognition methods can be divided into RGB video-based method and skeleton-based method. Among them, skeleton data can be obtained by depth camera (e.g. Microsoft Kinect [4]) or human pose estimation algorithm [7]. RGB video-based methods Most of these methods are based on appearance and motion information. From image classification to video classification, more attentions are paid to study how to process the temporal information of video, such as optical flow field. Simonyan and Zisserman [8] proposed a spatiotemporal two-stream network to make up for the deficiency of 2D CNN in time dimension. According to process stacked optical flow fields that are composed of continuous frames of dense optical flow images, high-quality motion information is obtained by temporal branch. However, temporal branch that simply stacks optical flow information ignores the time sequence information. Hence, on the basis of long-range temporal structure modelling, Wang et al. [9] presented a temporal segment network (TSN) through combining a sparse temporal sampling strategy and video-level supervision. Each short segment of sparse sampling produces a preliminary prediction. Finally, a video-level prediction is formed based on the consensus of segments. In addition to obtaining motion information through optical flow field, long short-term memory (LSTM) or 3D CNN is also used to explore time relationship. Donahue et al. [10] raised long-term recurrent convolutional networks (LRCNs). Firstly, CNN was used to extract the visual features of each frame. Then, the features were fed into the LSTM network to obtain the temporal dynamic features. Finally, the prediction results of variable length are generated. Tran et al. [11] proposed a convolutional 3D (C3D) network that used 3D convolution to extract spatial and temporal features, which could overcome the lack of 2D CNN in the temporal dimension. On the basis of 2D convolutional networks, Carreira and Zisserman [12] designed a I3D network by expanding all their filters and pool kernels to 3D. Skeleton-based methods In 1973, Johansson's light-spots experiment [13] laid a foundation for action recognition research based on key points. In 2011, the pioneering work of Shotton et al. [14] estimated the joint position of human body based on the depth map and stimulated the research of human action recognition based on the joints. Since then, skeleton-based action recognition methods are constantly emerging. Similar to RGB video-based methods, how to deal with time information is a question worth considering. Furthermore, how to make better use of spatial information of skeleton is also very important. In terms of mining temporal features, LSTM and 3D convolution are also used. By expanding the LSTM network into a two-dimensional network, Liu et al. [15] proposed a spatiotemporal long short-term memory (ST-LSTM) network with a trust gate so that the network could simultaneously simulate the time dependence between different frames and the spatial dependence between different joints in the same frame. Tu et al. [16] presented a two-stream 3D CNN network that encoded spatial and temporal information of skeleton joint 3D coordinates respectively, and then 3D CNN was used to extract depth features from the spatial and temporal streams. Besides, Kim et al. [17] used temporal convolutional networks (TCN) to recognize actions. Spatially, to make better use of inherent structure of skeleton, Liu et al. [15] proposed a bidirectional tree traversal method to visit joints and maintain the adjacency information of skeleton. Yan et al. [18] presented spatial-temporal graph convolutional networks (ST-GCN), which was the first time to apply GCN to skeleton-based action recognition. According to the natural connection structure of the human body, they proposed a space division rule to obtain the adjacency matrix and add time edges between the corresponding joints of consecutive frames. Shi et al. [19] added length and direction information of skeleton on the basis of traditional three-dimensional coordinates. At the end, a two-stream adaptive graph convolutional network (2s-AGCN) was proposed to use the first-order and second-order information of skeleton data together. Kalpit et al. [20] proposed a part-based graph convolutional network (PB-GCN), which divided the skeleton into four regions: head, hand, torso and legs, and used cross-edge parts to share information. Other researchers combined skeletal and temporal information in a single image in different ways. For example, Pham et al. [21] transformed spatiotemporal information of 3D skeleton into RGB images, and then designed deep CNNs (D-CNNs) based on ResNet to learn and classify the images. Besides, Zhang et al. [22] proposed a context-aware graph convolutional network (CA-GCN). CA-GCN obtains the context information of each vertex about other vertices by integrating the information of all vertices. Liu et al. [23] proposed a simple method to disentangle multi-scale graph convolutions and a unified ST graph convolutional operator named G3D to disentangle the importance of nodes in different neighbour hoods for effective long-range modelling. In addition, Zhang et al. [24] explicitly introduced the high-level semantics of joints (joint type and frame index) into the network to enhance the feature representation capability. Cheng et al. [25] proposed a novel shift graph convolutional network (Shift-GCN) to overcome the shortcomings of high computational complexity and inflexible perception in GCN. Chen et al. [26] proposed a novel Channel-wise Topology Refinement Graph Convolution (CTR-GC) to dynamically learn different topologies and effectively aggregate joint features in different channels for skeleton-based action recognition. Yang et al. [27] designed an auto-encoder framework, which can learn ST features from the artificial colour labels of skeleton joints effectively by leveraging the colourized skeleton point cloud. Li et al. [28] proposed Else-Net, a novel Elastic Semantic Network with multiple learning blocks to learn diversified human actions over time. Chi et al. [29] proposed InfoGCN, a learning framework for action recognition combining a novel learning objective and an encoding method. More related work can be found in paper [30]. As for the action recognition of 2D skeletons, Ghazal et al. [31] proposed supervised machine learning to classify the four actions of sitting, standing, walking and falling and used dataset MSR and dataset Le2i for action classification. Among them, K-nearest neighbours (KNNs) classifier is superior to support vector machine (SVM), Naive Bayes, linear discriminant and feed-forward back-propagation neural network. Silva et al. [32] used Fisher vector (FV) to encode joint angle and cross frame key point trajectory, and then used SVM for the classification. Their experiments were carried out on the KTH dataset and the Weizmann dataset for 10 actions classification in human daily activities. Lie et al. [33] realized human fall detection by using LSTM to classify the five actions of standing, walking, falling, lying down and rising. Their experiments were on their unpublished data for the falling detection. The above methods are used for recognition in specific scenarios or have few action categories, resulting in the lack of experiments in public dataset. Avola et al. [34] proposed a two-branch stacked recurrent neural network (RNN) with LSTM units. They divided the skeleton into upper and lower parts and sent them into different branches for prediction, and then fused the predictions of the two branches to get the final classification results. Although the LSTM network performs well in the exploration of temporal relationship, it is well known that its ability of exploring spatial dependence is insufficient. 3 METHOD Effectiveness of 3D convolutional network in processing video data has been verified in the current methods. In the field of action recognition based on RGB video, emergence of 3D convolution methods such as C3D [11] and I3D [12] indicates that 3D convolution can simultaneously learn spatiotemporal characteristics and has strong competitiveness. In recent years, GCN has also shown its own advantages. It is an effective way to learn spatial dependence between joints. Therefore, we come up with the idea of combining 3D CNN and GCN for action recognition. In this paper, GCN is introduced into 3D CNN and 3D-GCN is proposed finally. The overall framework is shown in Figure 1. The video stream is obtained from the monitoring camera for real-time pedestrian detection. Then, the target character is tracked to get the sequence image. After skeleton detection, the skeleton sequence data of the target character is obtained. After calculating, the descriptor is sent to the 3D-GCN model to get the corresponding action tag. FIGURE 1Open in figure viewerPowerPoint Overall frame diagram. φ represents feature extractor, * represents feature comparator Skeleton graph Skeleton data usually consists of a series of continuous skeleton frames, and each frame contains a set of joint coordinates. Given skeleton data, taking human joints as nodes and human bones as edges, a natural human connection graph can be formed as G = (V, E), and V = {vti | t = 1, 2, …,T, i = 1, 2, …, N} is the set that contains all joint nodes. Here, N represents the number of joints, T represents the total number of frames, the edge set E is composed of spatial edge set Es = {< vti, vtj > | (i, j)∈H} and temporal edge set Et = {< vti, v(t+1)i >}, H represents the joint connection set of the natural structure of the human body and Et contains the connection of the same joint in the previous and subsequent frames. The composition of the skeleton descriptor is illustrated in Figure 2. FIGURE 2Open in figure viewerPowerPoint Composition of the skeleton descriptor 3D Graph convolution Calculation of 2D convolution can be understood as sampling the image and multiplying the corresponding weight. Let the convolution kernel size be K*K, the output of single-channel image at position x can be represented according to Equation (1). f o u t ( x ) = ∑ h = 1 K ∑ w = 1 K f i n ( p ( x , h , w ) ) · w ( h , w ) $$\begin{equation}{f_{out}}(x) = \sum_{h = 1}^K {\sum_{w = 1}^K {{f_{in}}(p(x,h,w)) \cdot w(h,w)} } \end{equation}$$ (1)Similarly, 3D convolution is the extension of 2D convolution on the time axis, that is, 2D convolution is a rectangular data block to get a new value, while 3D convolution is a cuboid data block to get a new value. Therefore, Equation (1) is rewritten. Let the convolution kernel be K*K*K, the output of the single-channel image at position x can be represented according to Equation (2). f o u t ( x ) = ∑ d = 1 K ∑ h = 1 K ∑ w = 1 K f i n ( p ( x , d , h , w ) ) · w ( d , h , w ) $$\begin{equation}{f_{out}}(x) = \sum_{d = 1}^K {\sum_{h = 1}^K {\sum_{w = 1}^K {{f_{in}}(p(x,d,h,w)) \cdot w(d,h,w)} } } \end{equation}$$ (2) In the convolution of video, sampling function is p(x, d, h, w) = x + p'(d, h, w), namely, x and its adjacent pixels are obtained after sampling, and w is the weight vector, which is used to calculate the inner product of the sampled eigenvectors. 3.2.1 Sampling function In the traditional convolution calculations, sampling function p(x, d, h, w) is defined as the neighbouring pixels of centre pixel x. However, in the calculation of 3D graph convolution, the sampling function represents the neighbour nodes of central node, that is, B(vti) = {vti | d(vti, v(t+Δt)i, v(t+Δt)j) = Δt+d(v(t+Δt)i, v(t+Δt)j), d(v(t+Δt)i,v(t+Δt)j) < D, Δt ≤τ}, where, d(vti, vtj) represents the path length from node vti to node vtj. Thus, the sampling function p: B(vti)→V can be rewritten as Equation (3). p ( v t i , v ( t + Δ t ) i , v ( t + Δ t ) j ) = v ( t + Δ t ) j $$\begin{equation}p({v_{ti}},{v_{(t + \Delta t)i}},{v_{(t + \Delta t)j}}) = {v_{(t + \Delta t)j}}\end{equation}$$ (3) 3.2.2 Weight function In the traditional 3D convolution, video image data has a fixed order of arrangement, so there is a rigid cuboid grid around the centre position, and the weight function can be realized through the index tensor of spatial order (c, K, K, K). However, there is no such arrangement rule in the skeleton graph. Therefore, the weighting function is realized through the node index (t+Δt, j) as shown in Equation (4). w ( v t i , v ( t + Δ t ) i , v ( t + Δ t ) j ) = w ′ ( t + Δ t , j ) $$\begin{equation}w({v_{ti}},{v_{(t + \Delta t)i}},{v_{(t + \Delta t)j}}) = w^{\prime}(t + \Delta t,j)\end{equation}$$ (4) 3.2.3 3D graph convolution Equation (5) can be obtained by rewriting Equation (2). Then, Equation (6) can be obtained through substituting Equations (3) and (4) into Equation (5). f o u t ( v t i ) = ∑ v ( t + Δ t ) j ∈ B ( v t i ) f i n ( p ( v t i , v ( t + Δ t ) i , v ( t + Δ t ) j ) ) × w ( v t i , v ( t + Δ t ) t i , v ( t + Δ t ) t j ) $$\begin{eqnarray} {f_{out}}({v_{ti}}) &=& \sum_{{v_{(t + \Delta t)j}} \in B({v_{ti}})} {f_{in}}(p({v_{ti}},{v_{(t + \Delta t)i}},{v_{(t + \Delta t)j}}))\nonumber\\ &&\times\, w({v_{ti}},{v_{(t + \Delta t)ti}},{v_{(t + \Delta t)tj}}) \end{eqnarray}$$ (5) f o u t ( v t i ) = ∑ v ( t + Δ t ) j ∈ B ( v t i ) f i n ( v ( t + Δ t ) j ) · w ( t + Δ t , j ) $$\begin{equation}{f_{out}}({v_{ti}}) = \sum_{{v_{(t + \Delta t)j}} \in B({v_{ti}})} {{f_{in}}({v_{(t + \Delta t)j}}) \cdot } w(t + \Delta t,j)\end{equation}$$ (6) Attention module Attention module includes spatial attention module and temporal attention module. As for the time attention module, as shown in Figure 3, the maximum pooling and average pooling are applied on the joint dimension, and the outputs are concatenated into effective features. Then, a 3D convolutional layer of 1×1×1 is used, and finally the attention diagram Mt is obtained through the sigmoid activation function, as shown in Equation (7). Figure 4 shows the residual 3D graph convolution with temporal attention module, and the output result is shown in Equation (8). M t ( x ) = σ ( f 1 × 1 × 1 ( [ A v g P o o l ( x ) ; M a x P o o l ( x ) ] ) ) $$\begin{equation}{M_t}(x) = \sigma ({f^{1 \times 1 \times 1}}([AvgPool(x);MaxPool(x)]))\end{equation}$$ (7) f o u t = 3 D G C N ( f i n ) ⊗ T A ( f i n ) + f i n $$\begin{equation}{f_{out}} = 3DGCN({f_{in}}) \otimes TA({f_{in}}) + {f_{in}}\end{equation}$$ (8) FIGURE 3Open in figure viewerPowerPoint Temporal attention module FIGURE 4Open in figure viewerPowerPoint The residual 3D graph convolution with temporal attention module There is a trainable spatial attention matrix Ms in each layer of GCN. The output after 3D convolution, spatial attention module and temporal attention module is shown in Equation (9). Here, Ms is an N × N attention graph, representing the importance of each vertex, and ⊙ represents the dot product. f o u t = 3 D G C N ( f i n ) ⊙ M s ⊗ T A ( f i n ) + f i n $$\begin{equation}{{f}_{out}}=3DGCN({{f}_{in}}) \odot {{M}_{s}}\otimes TA({{f}_{in}})+{{f}_{in}}\end{equation}$$ (9) 3D-GCN In ST-GCN [18], only normalized joint coordinates are used as the input of the network, so only the simplest skeleton information is used. In this paper, the proposed skeleton descriptor consists of three parts, namely, normalized coordinates, frame difference and joint angle. The normalized coordinates are carried out with the neck as the centre point, as shown in Equations (10) and (11). For frame difference, current frame skeleton and previous frame skeleton are used to calculate the joint displacement, as shown in Equations (12) and (13). Let the two joint vectors be vti(x1, y1) and vtj(x2, y2), respectively, and the joint angle calculation is shown in Equations (14) and (15). p ( v t i ) = p ( v t i ) − p ( n e c k ) $$\begin{equation}p({v_{ti}}) = p({v_{ti}}) - p(neck)\end{equation}$$ (10) P ( t ) = ( p ( v t 0 ) , p ( v t 1 ) , … , p ( v t i ) ) , i = 1 , 2 , … , N $$\begin{equation}P(t)=(p({{v}_{t0}}),p({{v}_{t1}}),\ldots ,p({{v}_{ti}})),i=1,2,\ldots ,N\end{equation}$$ (11) δ p ( v t i ) = p ( v t i ) − p ( v ( t − 1 ) i ) $$\begin{equation}\delta p({v_{ti}}) = p({v_{ti}}) - p({v_{(t - 1)i}})\end{equation}$$ (12) Δ P ( t ) = ( Δ p ( v t 0 ) , Δ p ( v t 1 ) , … , Δ p ( v t i ) ) , i = 1 , 2 , … , N $$\begin{equation}\Delta P(\text{t})=(\Delta p({{v}_{t0}}),\Delta p({{v}_{t1}}),\ldots ,\Delta p({{v}_{ti}})),i=1,2,\ldots ,N\end{equation}$$ (13) a n g l e ( v t i ) = x 1 · x 2 + y 1 · y 2 x 1 2 + y 1 2 · x 2 2 + y 2 2 $$\begin{equation}angle({v_{ti}}) = \frac{{{x_1} \cdot {x_2} + {y_1} \cdot {y_2}}}{{\sqrt {{x_1}^2 + {y_1}^2} \cdot \sqrt {{x_2}^2 + {y_2}^2} }}\end{equation}$$ (14) A n g l e ( t ) = ( a n g l e ( v t 0 ) , a n g l e ( v t 1 ) , … , a n g l e ( v t i ) ) , i = 1 , 2 , … , N $$\begin{eqnarray} Angle(t)=(angle({{v}_{t0}}),angle({{v}_{t1}}),\ldots ,angle({{v}_{ti}})),i=1,2,\ldots ,N\nonumber\\ \end{eqnarray}$$ (15) Thus, the complete skeleton descriptor is composed of the above-mentioned three different types of feature, and its skeleton descriptor at time t is shown in Equation (16). I n p u t ( t ) = [ P ( t ) , δ P ( t ) , A n g l e ( t ) ] $$\begin{equation}Input(t) = [P(t),\delta P(t),Angle(t)]\end{equation}$$ (16) On this basis, an effective 3D graph convolution network 3D-GCN is proposed, as shown in Figure 5. Before the data enters each layer of the 3D graph convolutional layer, there is a batch normalization layer to normalize the data. Filter size of the 3D graph convolution is 9*1*1. Given F as input, F ′ $F^{\prime}$ is obtained after a series of 3D-GCN-block conversion, and finally, after average pooling, it is sent to the full connection layer for classification and its output set O is shown in Equation (17), where C is the total number of categories. Category cross entropy function is employed to train 3D-GCN. Given F, the prediction probability of ith category can be calculated as shown in Equation (18). o = o 1 , o 2 , … o C $$\begin{equation}o = \left\{ {{o_1},{o_2}, \ldots {o_C}} \right\}\end{equation}$$ (17) P ( C i F ) = e o i ∑ j = 1 C e o j , i = 1 , 2 , … , C $$\begin{equation}P({C_i}\left| F \right.) = \frac{{{e^{{o_i}}}}}{{\sum_{j = 1}^C {{e^{{o_j}}}} }},i = 1,2, \ldots ,C\end{equation}$$ (18) FIGURE 5Open in figure viewerPowerPoint 3D-GCN network structure. 3D-GCN, three-dimensional graph convolutional network. 4 EXPERIMENTS The proposed framework is shown in Figure 1. Numerous existing deep learning models can be used in pedestrian detection. In the experiment of this paper, YoloV3 is selected to train pedestrian detector. Number of joints varies with the different method. For example, if OpenPose is used to obtain the 2D skeleton information, the skeleton has 18 joint nodes in total. In the tracking link, the existing tracker algorithm can also be selected, and KCF is used as the tracker in this paper. As for the parameters in 3D graph convolution, D = 1 and τ = 9 are used, which means the 1-neighbourhood in space and 9-neighbourhood in time. Dataset 4.1.1 ZCP dataset At present, there are many public datasets in the field of action recognition, such as NTU RGB+D [35]. These public datasets are normally available with 3D skeleton data. However, it is difficulty or expensive to obtain 3D skeleton data but much easier to obtain 2D skeleton data from video cameras in the real applications. Such as in the traffic scene, it is quite meaningful if the actions of the pedestrians can be recognized while passing zebra crossings. Therefore, a new dataset named Zebra Crossing Pedestrian dataset (ZCP) is created. Samples of 18 people going through a zebra crossing are collected using a selfie stick with a cell phone. Pedestrian detector and OpenPose method are used to obtain the 2D skeleton data. There are 10 kinds of action categories, including crossing by bike (242 skeleton action sequences), waiting by bike (253 skeleton action sequences), waiting by electro-bike (233 skeleton action sequences), crossing by electro-bike (226 skeleton action sequences), crossing by running (234 skeleton action sequences), crossing by walking (250 skeleton action sequences), walking while calling (240 skeleton action sequences), waiting while calling (252 skeleton action sequences), standing and waiting (260 skeleton action sequences), waiting while playing phone (263 skeleton action sequences). In total, there are 2453 skeleton action sequences. The dataset is available in the following URL: https://drive.google.com/file/d/1o37jP5eyfKsiWWQOPEzIYbUw1xXtkRQ5. 4.1.2 NTU RGB + D dataset The dataset is obtained by three Kinect V2 in different views at the same time. It consists of 56,880 movement samples, which are divided into 60 different categories, including daily activities, interactions and medical status of 40 subjects. Each sample contains RGB video, depth map sequences, 3D bone data and infrared video. The maximum length of a sequence is 300, but most sequences are around 100. Experiments on the ZCP dataset The ZCP dataset is divided into three parts for experiments: the train dataset (1475 samples), the valid dataset (733 samples) and the test dataset (245 samples). Table 1 shows the results of the proposed method and other methods on the ZCP dataset. In addition, the ablation experiments are carried out to verify the effectiveness of data preprocessing and the experimental results can also be found in Table 1. In the table, 3D-GCN (no-preprocess) represents the method without data preprocessing, 3D-GCN (frame-difference) represents the method with frame difference calculation and 3D-GCN (angle) represents the method with joint angle calculation. TABLE 1.