Multi‐stage part‐aware graph convolutional network for skeleton‐based action recognition

IET Image ProcessingVolume 16, Issue 8 p. 2063-2074 ORIGINAL RESEARCHOpen Access Multi-stage part-aware graph convolutional network for skeleton-based action recognition Xiaofei Qin, Xiaofei Qin School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorHao Li, Hao Li School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorYuru Liu, Yuru Liu College of Science, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorJiabin Yu, Corresponding Author Jiabin Yu 94607969@qq.com orcid.org/0000-0001-7875-8098 Key Laboratory of Industrial Internet and Big Data, China National Light Industry, Beijing, China School of Artificial Intelligence, Beijing Technology and Business University, Beijing, China Correspondence Jiabin Yu, Key Laboratory of Industrial Internet and Big Data, China National Light Industry, Beijing 100048, China. Email: 94607969@qq.comSearch for more papers by this authorChangxiang He, Changxiang He College of Science, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorXuedian Zhang, Xuedian Zhang School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China Shanghai Institute of Intelligent Science and Technology, Tongji University, Shanghai, China Shanghai Key Laboratory of Contemporary Optics System, Shanghai, China Key Laboratory of Biomedical Optical Technology and Devices of Ministry of Education, Shanghai, ChinaSearch for more papers by this author Xiaofei Qin, Xiaofei Qin School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorHao Li, Hao Li School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorYuru Liu, Yuru Liu College of Science, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorJiabin Yu, Corresponding Author Jiabin Yu 94607969@qq.com orcid.org/0000-0001-7875-8098 Key Laboratory of Industrial Internet and Big Data, China National Light Industry, Beijing, China School of Artificial Intelligence, Beijing Technology and Business University, Beijing, China Correspondence Jiabin Yu, Key Laboratory of Industrial Internet and Big Data, China National Light Industry, Beijing 100048, China. Email: 94607969@qq.comSearch for more papers by this authorChangxiang He, Changxiang He College of Science, University of Shanghai for Science and Technology, Shanghai, ChinaSearch for more papers by this authorXuedian Zhang, Xuedian Zhang School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China Shanghai Institute of Intelligent Science and Technology, Tongji University, Shanghai, China Shanghai Key Laboratory of Contemporary Optics System, Shanghai, China Key Laboratory of Biomedical Optical Technology and Devices of Ministry of Education, Shanghai, ChinaSearch for more papers by this author First published: 09 March 2022 https://doi.org/10.1049/ipr2.12469AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Recently, graph convolutional networks have shown excellent results in skeleton-based action recognition. This paper presents a multi-stage part-aware graph convolutional network for the problems of model over complication, parameter redundancy and lack of long-dependence feature information. The structure of this network has a multi-stream input and two-stream output, which can greatly reduce the complexity and improve the accuracy of the model without losing sequence information. The two branches of the network have the same backbone, which includes 6 multi-order feature extraction blocks and 3 temporal attention calibration blocks, and the outputs of the two branches are fused together. In multi-order feature extraction block, a channel-spatial attention mechanism and a graph condensation module are proposed, which can extract more distinguishable feature and identify the relationship between parts. In temporal attention calibration block, the temporal dependencies between frames in the skeleton sequence are modeled. Experimental results show that the proposed network outperforms many mainstream methods on NTU and Kinetics datasets, for example, it achieves 92.4% accuracy on the cross-subject benchmark of NTU-RGBD60 dataset. 1 INTRODUCTION Action recognition task can often be divided into video-based and skeleton-based. Compared to video data, skeleton data is much more robust in dynamic environments with complicated backgrounds. In addition, most action information is contained in skeleton sequences which can model human body structure information well, such as the joint positions, bone links etc. However, video sequences often introduce a lot of noise information, such as color, texture, body size etc. Meanwhile, advances in low-cost depth cameras [5] and pose estimation technology [6] have made skeleton data much easier to access than before. Therefore, skeleton-based action recognition has been widely studied by researchers in recent years, and gathered much momentum in the field of human–computer interaction, health care, robotics [1-4]. In the past few years, many methods based on convolutional neural network (CNN) [7-9] and recurrent neural network (RNN) [10-12] have been proposed, which overcome the inability of handcrafted feature methods [13-15], and effectively model the spatiotemporal information in the action sequence. However, the skeleton of a human body naturally serves as a graph in non-Euclidean space, and the dependencies between the associated joints cannot be fully expressed if the skeleton is simply viewed as a vector sequence or a 2D grid. Recently, some methods based on graph convolutional network (GCN) have been proposed for this task [16-22]. Compared with CNN and RNN, GCN can represent graph data more effectively. Among these GCN based methods, 2-stream adaptive graph convolutional networks (2s-AGCN) [19], and semantics-guided neural networks (SGN) [22] are typical and popular ones. However, there are three drawbacks in typical GCN based methods like SGN and 2s-AGCN: (1) In actual life, people often need multiple synergy among distant joints to complete an action. However, typical GCNs ignore the dependency among non-physical connection joints, because one layer of GCN only aggregates the first-order neighbor information. In order to model high-order neighbor information, GCN layers have to be stacked many times, this will cause unnecessary redundancy and weaken the network's ability to aggregate global semantic information. For example, when people clap hands, the movement of two hands are most important. But the features of two hands have to be transferred through two arms and chest, and finally connected with each other. (2) The skeleton graphs in different layers of typical GCNs are the same, which would prevent GCN network to model the semantic information contained in different layers efficiently, and affect the flexibility of the network. (3) Typical GCN networks did not fully exploit the potential of the attention mechanism, and the importance of different feature maps were not clearly modeled, leading to possible loss of distinguishable information. To address above issues, a novel multi-stage part-aware graph convolutional network (MS-PGCN) is proposed; its structure is shown in Figure 1. The backbone of MS-PGCN includes some multi-order features extraction blocks (MFEBs) and some temporal attention calibration blocks (TACBs). From the spatial perspective, the MFEB contains a channel-spatial attention mechanism (CSAM) and a graph condensation module (GCM). The CSAM can boost the spatiotemporal feature capture ability of MFEB, and the GCM is designed to condense the related joints. From the temporal perspective, the role of TACB is to integrate and calibrate the inter-frame information. Furthermore, three kinds of information from the skeleton sequence are used as the input of the network, which are bone vectors, joint positions and motion velocities. The input of each branch is the joint data and the bone data that have been concatenated with the motion data, respectively. The results of s o f t m a x $softmax$ of the two branches are fused together to generate the final recognize result. FIGURE 1Open in figure viewerPowerPoint Structure of MS-PGCN. In the feature extraction part, one green block represents an MFEB, and one yellow block represents a TACB. The prediction scores of two branches are combined to obtain final prediction score Comprehensive experiments on three large public datasets are concucted to demonstrate the effectiveness of the proposed MS-PGCN: NTU-RGB+D60 [23], NTU-RGB+D120 [24] and Kinetics [25]. According to the experiment results, MS-PGCN shows better performance than many state-of-the-art approaches. The following are the primary contributions of this work: Two novel blocks are proposed, that is, MFEB and TACB. Based on a kind of channel-spatial attention mechanism and a manually designed joints condensation scheme, the MFEB is designed. Based on the attention mechanism and convolutional operations in the temporal dimension, the TACB is constructed. Based on the MFEB and TACB, a three-stage skeleton-based action recognition network (MS-PGCN) is designed. Each stage includes two MFEBs and a TACB. After every MFEB, the skeleton graph is condensed to some extent, the features extracted become more abstract and more semantic information is aggregated. The TACB is used to model the temporal dependencies between frames. Although the proposed MS-PGCN has more layers than many GCN networks, the parameter and computational complexities are not cumbersome, mainly because GCMs make the network sparse. MS-PGCN has a two-branch structure, the motion data is concatenated with the joint and bone data to enrich the amount of information inputted to the two branches. The MS-PGCN outperforms several state-of-the-art methods on three large-scale datasets for skeleton-based action recognition. 2 RELATED WORKS 2.1 Skeleton-based action recognition Feature extraction methods in skeleton-based action recognition task fall into two categories: traditional hand-crafted ones [14, 15, 26, 27, 31] and recent deep learning-based ones [17, 18, 28-30]. Traditional methods need elaborate design, for example, Chaudhry et al. [31] used linear dynamic systems (LDS) to depict the hierarchy spatial-temporal structure of the skeleton sequence in different scales. Xia et al. [14] presented a 3D histogram joints representation method, and projected the 3D histogram to posture visual words, then used hidden Markov model to recognize the action. Compared to traditional methods, deep learning-based ones have achieved unprecedented results in many computer vision tasks. Therefore, this paper focuses on deep learning methods in skeleton-based action recognition task. 2.2 Convolutional neural network methods A considerable part of skeleton-based action recognition methods using CNN or RNN have been proposed in recent years. CNN-based methods [7-9, 29] show great advantages both in accuracy and simplicity. Cao et al. [33] created a permutation network to figure out the best way to rearrange the joints. Rongjie et al. [34] developed the geometric algebra-based shape-motion representation, which discussed the significance of both bones and joints while making full use of the information provided by the skeleton sequence. CNN-based methods tend to represent the sequence as images, which will inevitably ignore the structure dependence problem in the action recognition task, and these approaches tend to focus on the local information of the joints while overlooking the global information between the far joints. 2.3 Graph convolutional network methods As a generalization version of CNN, GCNs [35-37] can be applied to any kind of graph related problems. The skeleton data is naturally stored as a graph rather than a vector sequence or a two-dimensional grid, so GCN is more suitable for solving skeleton-based action recognition tasks than CNN. The representation of skeleton data, or how to organize raw data into a certain graph, is an important challenge in GCN-based techniques. In [36], GCN was used for the first time in the task of skeleton-based action recognition, the skeleton sequence in this paper is explicitly regarded as a graph, and achieved competitive results. Based on this, many effective methods are gradually derived. Shi et al. [19] presented a two-stream adaptive GCN model that employs the non-local block to learn a content adaptive graph and use it to transfer messages between GCN layers. The representation of skeleton data, or how to structure the original data into a certain graph, is still an essential difficulty in GCN-based approaches. 3 MULTI-STAGE PART-AWARE GRAPH CONVOLUTIONAL NETWORK In this section, the structure of multi-stage part-aware graph convolutional network will be explained in detail. 3.1 Multi-order features extraction block MFEB contains a channel-spatial attention mechanism (CSAM) and a graph condensation module (GCM). The output of the second GCN layer and the CSAM are fused together, which is used to facilitate the model ability in focusing on discriminative joints and channels. GCM is used to condense the correlated joints, which not only can increase model sparsity and reduce model complexity, but also can enlarge model receptive field and gather global semantic information between non-physically connected joints. The structure of MFEB is shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Illustration of the multi-order features extraction block (MFEB), which includes three GCN layers, a CSAM and a GCM layer. Notably, GCM only changes the graph topology but does not change the number of channels 3.1.1 Graph convolutional network The essence of a GCN layer is an operation to aggregate first-order neighbor information. Multiple GCN layers can aggregate high-order neighbor information. The main function of GCN is to update nodes' feature with its neighbor information. The calculation formula of GCN for one node is as follows: F o u t = ∑ v j ∈ N ∼ v i X j W , \begin{align} \mathbf { F}_{out}=\sum _{v_{j} \in \widetilde{N}{\left(v_{i}\right)}} \mathbf {X}_{j} \mathbf {W}, \end{align} (1)where W $\mathbf {W}$ represents the weight matrix, X j $\mathbf {X}_{j}$ represents the feature matrix, v i $v_{i}$ is the root node, and N ∼ ( v i ) $\widetilde{N}(v_{i})$ is the set of first-order neighbor nodes that are connected to the root node. As the number of GCN layers increases, the aggregation radius of nodes also increases. Once a certain threshold is reached, the nodes that can be covered by this node are almost the same as the full graph. To solve this problem, a residual connection is added in the CSAM of MFEB. 3.1.2 Channel-spatial attention mechanism Inspired by the convolutional block attention module [38], CSAM is proposed which combines spatial information and channel information. The structure of CSAM is shown in Figure 3, and its formula is shown as follows: F c = Att c ( F in ) ⊙ F in , \begin{align} \mathbf {F}_{\mathbf {c}}=\mathbf {Att}_{\mathbf {c}}(\mathbf {F}_{\mathbf {in}}) \odot \mathbf {F}_{\mathbf {in}}, \end{align} (2) F out = F in + Att s ( F c ) ⊙ F c , \begin{align} \mathbf {F}_{\mathbf {out}}=\mathbf {F}_{\mathbf {in}}+\mathbf {Att}_{\mathbf {s}}(\mathbf {F}_{\mathbf {c}}) \odot \mathbf {F}_{\mathbf {c}}, \end{align} (3)where ⊙ $\odot$ represents element-wise multiplication with broadcasting if necessary, F in ∈ R C × T × N $\mathbf {F}_{\mathbf {in}} \in \mathbb {R}^{C \times T \times N}$ is the input feature, F out ∈ R C × T × N $\mathbf {F}_{\mathbf {out}} \in \mathbb {R}^{C \times T \times N}$ is the output feature, Att c ( F in ) ∈ R C × T × 1 $\mathbf {Att}_{\mathbf {c}}(\mathbf {F}_{\mathbf {in}}) \in \mathbb {R}^{C\times T\times 1}$ is the output of channel attention, Att s ( F c ) ∈ R 1 × T × N $\mathbf {Att}_{\mathbf {s}}(\mathbf {F}_{\mathbf {c}}) \in \mathbb {R}^{1\times T \times N}$ is the output of spatial attention. M ( F ) = σ M c ( F ) + M s ( F ) , \begin{align} \mathbf{M}(\mathbf{F})=\sigma\left(\mathbf{M}_{\mathbf{c}}(\mathbf{F})+\mathbf{M}_{\mathbf{s}}(\mathbf{F})\right), \end{align} (4) FIGURE 3Open in figure viewerPowerPoint Detailed structure of channel-spatial attention mechanism Channel attention module. M c ( F ) = B N f 1 1 × 1 f 0 1 × 1 ( AvgPool ( F ) ) . \begin{align} \mathbf{M}_{\mathbf{c}}(\mathbf{F})=B N\left(f_{1}^{1 \times 1}\left(f_{0}^{1 \times 1}(\operatorname{AvgPool}(\mathbf{F}))\right)\right). \end{align} (5) First the average and max pooling operations are used in spatial dimension to get the feature map of channel dimension, then F AVG c $\mathbf {F}_{\text{AVG }}^{\mathrm{c}}$ and F MAX c $\mathbf {F}_{\text{MAX }}^{\mathrm{c}}$ are obtained. These two features are fed into the shared convolutional layer to generate Att c ( F in ) ∈ R C × T × 1 $\mathbf {Att}_{\mathbf {c}}(\mathbf {F}_{\mathbf {in}}) \in \mathbb {R}^{C\times T\times 1}$ . The shared convolution layer is composed of two 1 × 1 $1\times 1$ convolutions. In order to reduce parameter amount, the channel number of the first convolution is C / k $C / k$ , where k $k$ is the channel reduction rate. After the features F AVG c $\mathbf {F}_{\text{AVG }}^{\mathrm{c}}$ and F MAX c $\mathbf {F}_{\text{MAX }}^{\mathrm{c}}$ are, respectively, operated by the shared convolution layer, element-wise summation is used. The calculation process of the channel attention module is as follows: Att c ( F ) = σ S C L ( AvgPool ( F ) ) + S C L ( MaxPool ( F ) ) , \begin{eqnarray} \mathbf {Att}_{\mathbf {c}}(\mathbf {F})=\sigma {\left({\left(SCL(\operatorname{AvgPool}(\mathbf {F}))\right)}+{\left(SCL(\operatorname{MaxPool}(\mathbf {F}))\right)}\right)},\nonumber\\ \end{eqnarray} (6)where σ $\sigma$ represents S i g m o i d $Sigmoid$ . S C L $SCL$ represents the shared convolutional layer as shown in Figure 3. Spatial attention module. Since different joints of human skeleton play different roles when completing a certain action, the use of spatial attention can help the model to focus on important joints. In the spatial attention module, the average and max pooling operations are used in channel dimension to get the feature map of spatial dimension, then F AVG s $\mathbf {F}_{\text{AVG }}^{\mathrm{s}}$ and F MAX s $\mathbf {F}_{\text{MAX }}^{\mathrm{s}}$ are obtained. These two features are concatenated together and fed into two convolutional layer to generate Att s ( F c ) ∈ R 1 × T × N $\mathbf {Att}_{\mathbf {s}}(\mathbf {F}_{\mathbf {c}}) \in \mathbb {R}^{1\times T\times N}$ . The calculation process of the spatial attention module is as follows: M s ( F c ) = σ f 1 1 × 1 f 0 1 × 1 [ AvgPool ( F c ) ; MaxPool ( F c ) ] = σ f 1 1 × 1 f 0 1 × 1 ( F avg s ; F max s ) , \begin{align} \mathbf {M}_{\mathbf {s}}(\mathbf {F}_{\mathbf {c}}) =& \sigma {\left(f_{1}^{1 \times 1}{\left(f_{0}^{1 \times 1}{\left([\operatorname{AvgPool}(\mathbf {F}_{\mathbf {c}}) ; \operatorname{MaxPool}(\mathbf {F}_{\mathbf {c}})]\right)}\right)}\right)}\nonumber \\ =& \sigma {\left(f_{1}^{1 \times 1}{\left(f_{0}^{1 \times 1}({\left[\mathbf {F}_{\text{avg }}^{\mathbf {s}} ; \mathbf {F}_{\max }^{\mathbf {s}}\right]})\right)}\right)}, \end{align} (7)where σ $\sigma$ represents S i g m o i d $Sigmoid$ . f 1 × 1 $f^{1 \times 1}$ represents 1 × 1 $1 \times 1$ convolution operation. 3.1.3 Graph condensation module GCM is designed to condense some joints as a part, which will down-sample the skeleton graph, and aggregate high-order neighbor information. Every time such an operation is conducted, a new graph is generated. In MS-PGCN, each branch has six GCM operations, the condensation process of the six GCM operations for NTU-RGBD dataset is shown in Figure 4. FIGURE 4Open in figure viewerPowerPoint The condensation process of the human skeleton for NTU-RGBD dataset. Subfigure (a) represents the original skeleton graph, and subfigure (b) represents the division strategy of the six GCMs, where the red font represents the points that are condensed together during one GCM operation, and subfigure (c) is an example of the first GCM operation Suppose the number of graph nodes before and after one GCM is, respectively, N $N$ and M $M$ , then the condensation process can be realized by an assignment matrix K $\mathbf {K}$ . The element of K $\mathbf {K}$ can be obtained by the following formula: k i , j = 1 , if φ i ∈ Φ j 0 , if φ i ∉ Φ j , \begin{align} k_{i, j}={\left\lbrace \def\eqcellsep{&}\begin{array}{lll}1, & \text{ if } & \varphi _{i} \in \Phi _{j} \\[7pt] 0, & \text{ if } & \varphi _{i} \notin \Phi _{j} \end{array} ,\right.} \end{align} (8)where i ∈ { 1 , 2 , … , N } $i \in \lbrace 1,2, \ldots , N\rbrace$ , j ∈ { 1 , 2 , … , M } $j \in \lbrace 1,2, \ldots , M\rbrace$ , φ i $\varphi _{i}$ represents one node before condensation, Φ j $\Phi _{j}$ represents one node after condensation. As shown in Figure 4, for NTU-RGBD dataset, the initial number of nodes is 25. After six condensation, the number of nodes became 19, 17, 14, 10, 6, and 1, respectively. So, the dimensions of the six assignment matrices can be expressed as K 1 ∈ R 25 × 19 $\mathbf {K}^{1} \in R^{25 \times 19}$ , K 2 ∈ R 19 × 17 $\mathbf {K}^{2} \in R^{19 \times 17}$ , K 3 ∈ R 17 × 14 $\mathbf {K}^{3} \in R^{17 \times 14}$ , K 4 ∈ R 14 × 10 $\mathbf {K}^{4} \in R^{14 \times 10}$ , K 5 ∈ R 10 × 6 $\mathbf {K}^{5} \in R^{10 \times 6}$ and K 6 ∈ R 6 × 1 $\mathbf {K}^{6} \in R^{6 \times 1}$ . Suppose the original adjacency matrix for NTU-RGBD is A 0 ∈ R 25 × 25 $\mathbf {A}^{0} \in R^{25 \times 25}$ , then the adjacency matrix after the b -th $b \text{-th }$ condensation can be obtained as follows: A b = K ∼ b T A b − 1 K ∼ b , \begin{align} \mathbf {A}^{b}={\left(\tilde{\mathbf {K}}^{b} \right)}^{T} \mathbf {A}^{b-1} \tilde{\mathbf {K}}^{b}, \end{align} (9)where K ∼ b $\tilde{\mathbf {K} }^{b}$ is the normalized assignment matrix of K b $\mathbf {K} ^{b}$ . On the basis of the above discussion about the assignment matrix and adjacency matrix, the GCM is proposed, as shown in Figure 5, where Z i n ∈ R C × T × N $\mathbf {Z}_{in} \in R^{C \times T \times N}$ is the input feature, Z g c n 1 ∈ R C × T × N $\mathbf {Z}_{gcn1} \in R^{C \times T \times N}$ is the output feature of the first GCN, Z g c n 1 = GCN ( Z ) . \begin{align} \mathbf {Z}_{gcn1}=\operatorname{GCN}(\mathbf {Z}). \end{align} (10)After that, Z g c n 1 $\mathbf {Z}_{gcn1}$ is condensed by a weighted assignment matrix and embedded by the second GCN, then Z g c n 2 ∈ R C × T × M $\mathbf {Z}_{gcn2} \in R^{C \times T \times M}$ is obtained, Z g c n 2 = GCN Z g c n 1 K ∼ b ⊙ W , \begin{align} \mathbf {Z} _{g c n2}=\operatorname{GCN}{\left(\mathbf {Z} _{g c n1}{\left(\tilde{\mathbf {K} }^{b} \odot \mathbf {W} \right)} \right)}, \end{align} (11)where ⊙ $\odot$ is element-wise multiplication, W $\mathbf {W}$ is the trainable weight for condense N $N$ nodes into M $M$ nodes. The GCN is used to extract features of the newly generated graph. FIGURE 5Open in figure viewerPowerPoint The illustration of GCM. Z i n $Z_{in}$ and Z o u t $Z_{out}$ represent input and output features, respectively. The symbol ⊗ $\otimes$ indicates matrix multiplication. The symbol ⊙ $\odot$ indicates element-wise multiplication. The symbol ⊕ $\oplus$ indicates element-wise addition Inspired by [40], the part-wise attention is used on another branch to add weight to the important body parts over the whole action sequence, Z a t t = Z g c n 1 K ∼ b ⊙ Softmax σ G A P Z g c n 1 K ∼ b W 1 W 2 , \begin{align} \mathbf {Z} _{a t t}=\mathbf {Z}_{gcn1} \tilde{\mathbf {K} }^{b} \odot \operatorname{\mathit {Softmax} }{\left(\sigma {\left(G A P{\left(\mathbf {Z}_{gcn1} \tilde{\mathbf {K} }^{b}\right)} \mathbf {W} _{1}\right)} \mathbf {W} _{2}\right)}, \end{align} (12)where ⊙ $\odot$ is element-wise multiplication, σ $\sigma$ represents the R e L U $ReLU$ activation function, and W 1 $\mathbf {W} _{1}$ , W 2 $\mathbf {W} _{2}$ are trainable parameter matrices. Eventually, the features of Z g c n 2 $\mathbf {Z} _{g c n2}$ and Z a t t $\mathbf {Z} _{a t t}$ is summarized together, Z out = Z g c n 2 ⊕ Z a t t . \begin{align} \mathbf {Z} _{\text{out }}=\mathbf {Z}_{g c n2} \oplus \mathbf {Z}_{a t t}. \end{align} (13) 3.2 Temporal attention calibration block As shown in Figure 6, TACB is used to capture and calibrate the temporal-level information across the whole sequence, where G ∈ R C × T × N $\mathbf {G}\in \mathbb {R}^{C \times T \times N}$ is the input feature of TACB, G ∼ ∈ R C × T × N $\tilde{\mathbf {G}}\in \mathbb {R}^{C \times T \times N}$ is the output feature of TACB, G ∼ = G + F ( G ) . \begin{align} \tilde{\mathbf {G}}=\mathbf {G}+\mathbf {F}(\mathbf {G}). \end{align} (14) F ( G ) $\mathbf {F}(\mathbf {G})$ is the calibrated feature produced by the element-wise multiplication of the output of the info capture cell branch F I ( G ) $\mathbf {F}_{\mathbf {I}}(\mathbf {G})$ and the attention branch F A ( G ) $\mathbf {F}_{\mathbf {A}}(\mathbf {G})$ , F ( G ) = F I ( G ) ⊙ F A ( G ) . \begin{align} \mathbf {F}(\mathbf {G})=\mathbf {F}_{\mathbf {I}}(\mathbf {G}) \odot \mathbf {F}_{\mathbf {A}}(\mathbf {G}). \end{align} (15) FIGURE 6Open in figure viewerPowerPoint Temporal attention calibration block (TACB). TACB incorporates the input and calibrated features in a unified framework as G ∼ = G + F ( G ) $\widetilde{\mathbf {G}}=\mathbf {G}+\mathbf {F}(\mathbf {G})$ In the info capture cell branch, three CNN layers are applied, the first is a temporal convolution layer that models frame dependencies, the remainder layers are used to translate the features into a high-dimensional space by 1 × 1 $1 \times 1$ convolution, which will improve model's representation capability. f t ̂ = σ W 2 σ W 1 f t + b 1 + b 2 . \begin{align} \widehat{\mathbf{f}_{t}}=\sigma\left(W_{2}\left(\sigma\left(W_{1} \mathbf{f}_{t}+\mathbf{b}_{1}\right)\right)+\mathbf{b}_{2}\right). \end{align} (16) In the attention branch, an attention weight matrix F A $\mathbf {F}_{\mathbf {A}}$ is obtained by two 1 × 1 $1 \times 1$ convolutions, which is used to squeeze and excitation the temporal dimension of the feature, F A = σ f 1 1 × 1 f 0 1 × 1 Duplicate AvgPool G , \begin{align} \mathbf {F}_{\mathbf {A}}=\sigma {\left(f_{1}^{1 \times 1}{\left(f_{0}^{1 \times 1}{\left(\operatorname{Duplicate}{\left(\operatorname{AvgPool}{\left(\mathbf {G}\right)}\right)}\right)} \right)}\right)}, \end{align} (17)where AvgPool represents average pooling in spatial dimension, Duplicate represents broadcast, f 1 1 × 1 $f_{1}^{1 \times 1}$ and f 0 1 × 1 $f_{0}^{1 \times 1}$ represent two convolutions. = ⋮ ⋮ = ⋯ ⋯ ⋮ ⋱ ⋮ ⋮ ⋱ ⋮ ⋯ ⋯ , \begin{align} ={\left[ \def\eqcellsep{&}\begin{array}{c} \vdots \\[3pt] \vdots \end{array} \right]}= {\left[ \def\eqcellsep{&}\begin{array}{cccc} {} &\quad \cdots &\quad \cdots &\quad \\[3pt] \vdots &\quad \ddots &\quad &\quad \vdots \\[3pt] \vdots &\quad &\quad \ddots &\quad \vdots \\[3pt] &\quad \cdots &\quad &\quad \cdots \end{array} \right]}, \end{align} (18) G p = 1 N ∑ i = 1 N g i ( 1 ) , … , 1 N ∑ i = 1 N g i ( T ) T , \begin{align} \mathbf{G}_{p}=\left[\frac{1}{N} \sum_{i=1}^{N} g_{i}(1), \ldots, \frac{1}{N} \sum_{i=1}^{N} g_{i}(T)\right]^{T}, \end{align} (19) G ← F d G p , N ∈ R T × N , \begin{align} \mathbf{G} \leftarrow \mathbf{F}_{d}\left(\mathbf{G}_{p}, N\right) \in \mathbb{R}^{T \times N}, \end{ali