Multi‐modal fusion method for human action recognition based on IALC

IET Image ProcessingVolume 17, Issue 2 p. 388-400 ORIGINAL RESEARCHOpen Access Multi-modal fusion method for human action recognition based on IALC Yinhuan Zhang, Yinhuan Zhang orcid.org/0000-0003-2571-1346 School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China School of Construction Engineering, Weinan Vocational & Technical College, Weinan, China Contribution: Conceptualization, Methodology, Writing - original draftSearch for more papers by this authorQinkun Xiao, Qinkun Xiao School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China School of Electronic Information Engineering, Xi'an Technological University, Xi'an, China Contribution: Writing - review & editingSearch for more papers by this authorXing Liu, Corresponding Author Xing Liu [email protected] School of Electronic Information Engineering, Xi'an Technological University, Xi'an, China Correspondence Xing Liu, School of Electronic Information Engineering, Xi'an Technological University, Xi'an 710021, China. Email: [email protected] Contribution: SupervisionSearch for more papers by this authorYongquan Wei, Yongquan Wei CRRC Tangshan Co., Ltd., Tangshan, China Contribution: Software, ValidationSearch for more papers by this authorChaoqin Chu, Chaoqin Chu School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China Contribution: Software, Data curationSearch for more papers by this authorJingyun Xue, Jingyun Xue School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China Contribution: Visualization, ​InvestigationSearch for more papers by this author Yinhuan Zhang, Yinhuan Zhang orcid.org/0000-0003-2571-1346 School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China School of Construction Engineering, Weinan Vocational & Technical College, Weinan, China Contribution: Conceptualization, Methodology, Writing - original draftSearch for more papers by this authorQinkun Xiao, Qinkun Xiao School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China School of Electronic Information Engineering, Xi'an Technological University, Xi'an, China Contribution: Writing - review & editingSearch for more papers by this authorXing Liu, Corresponding Author Xing Liu [email protected] School of Electronic Information Engineering, Xi'an Technological University, Xi'an, China Correspondence Xing Liu, School of Electronic Information Engineering, Xi'an Technological University, Xi'an 710021, China. Email: [email protected] Contribution: SupervisionSearch for more papers by this authorYongquan Wei, Yongquan Wei CRRC Tangshan Co., Ltd., Tangshan, China Contribution: Software, ValidationSearch for more papers by this authorChaoqin Chu, Chaoqin Chu School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China Contribution: Software, Data curationSearch for more papers by this authorJingyun Xue, Jingyun Xue School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China Contribution: Visualization, ​InvestigationSearch for more papers by this author First published: 12 October 2022 https://doi.org/10.1049/ipr2.12640AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In occlusion and interaction scenarios, human action recognition (HAR) accuracy is low. To address this issue, this paper proposes a novel multi-modal fusion framework for HAR. In this framework, a module called improved attention long short-term memory (IAL) is proposed, which combines the improved SE-ResNet50 (ISE-ResNet50) with long short-term memory (LSTM). IAL can extract the video sequence features and the skeleton sequence features of human behaviour. To improve the performance of HAR at a high semantic level, the obtained multi-modal sequence features are fed into a couple hidden Markov model (CHMM), and a multi-modal IAL+CHMM method called IALC is developed based on a probability graph model. To test the performance of the proposed method, experiments are conducted on the HMDB51, UCF101, Kinetics 400k, and ActivityNet datasets, and the obtained recognition accuracy are 86.40%, 97.78%, 81.12%, and 69.36% on the four datasets, respectively. The experimental results show that when the environment is complex, the proposed multi-modal fusion method for HAR based on the IALC can achieve more accurate target recognition results. 1 INTRODUCTION Human action recognition (HAR) has a wide application in robot behaviour research [1], video surveillance [2], and human-computer interaction [3]. However, HAR is faced with many challenging problems that have been considered by scholars in multiple fields [4]. Especially, the accuracy of HAR can directly affect the popularization and application of behaviour recognition. The previous HAR frameworks had many limitations [5]. The use of obtained features by single modal video for HAR [6] is easily affected by illumination, angle of view, and other factors, resulting in low recognition accuracy. Using a multi-modal fusion model can better solve the problem of data loss in single-sensor behaviour recognition and utilize the complementarity of different modal data to improve the accuracy of behaviour recognition [7]. Meanwhile, a simple composition of features does not guarantee better results than a single modal. Methods such as Random Forest (RF) or Gaussian Mixture Models (GMMs) [8] require researchers to manually extract behaviour features from sensor data, which greatly depends on experts' previous experience. To address this issue, human action features are extracted by Convolutional Neural Networks (CNN) in RGB videos and other modal videos. CNN can avoid the negative effects of GMM. However, the latter fusion algorithm also directly affects the accuracy of behaviour recognition. In prior HAR fusion methods based on the hidden Markov model (HMM) [9], excessive parameters are involved, which slow down the training and computation of the model. In this paper, based on the transfer learning method, LSTM is combined with the couple hidden Markov model (CHMM) to achieve behaviour sequence fusion [10]. However, there is no theoretical basis for assigning appropriate weights to every feature. To address the above problems, this paper adopts CHMM for multi-modal information fusion and proposes a novel fusion method called IALC, which adopts a two-level fusion architecture. The low level combines feature recognition and complex semantic decision processes. The contributions of this paper are as follows: (1) since incomplete features reduce recognition accuracy at the low level, the recognition accuracy is improved at the second level using the CHMM method. (2) In IALC, the IAL module contains an Improved SE-block (ISE-block) that automatically scales the weights of important informative features. (3) Based on the theory of probabilistic graph models, the proposed IALC method is used for fusion, and different weights are assigned to different features in the fusion process, which improves the accuracy of classifier learning. The remainder of this article is organized as follows: Section 2 introduces related work. Section 3 describes the proposed IALC method. The experiment and discussion are presented in Section 4, and the conclusion is drawn in Section 5. 2 RELATED WORK Many recognition algorithms for HAR have been proposed, and they can be roughly divided into two categories, that is, neural network-based methods and HMM-based methods. Compared with neural network-based methods, HMM-based HAR methods generally require less training data and have lower behaviour recognition accuracy. Recently, neural network-based methods have become more and more popular, in part because the networks extract finer behavioural features and achieve higher recognition accuracy. Since AlexNet [11] was proposed, many scholars have studied how to improve the performance of CNN and the accuracy of behaviour recognition [12]. In recent years, the attention mechanism has been integrated into various neural networks, which has shown significant potential in increasing network performance and has attracted much attention [13]. The SE-block is the most representative module [14], and it learns the channel attention of each convolution block and improves the performance of various deep CNNs. Although this method has achieved some success, it greatly increases network parameter size and model complexity because it adds a full connection layer to each convolution. In [15], a model based on 3D-CNN and LSTM was proposed, which mainly considers the attention of network characteristics to achieve better network performance. However, it generates nearly 300 M parameters and requires high computing capability. Generally, the model is difficult to train and cannot be widely used. Based on the work in [16], a dynamic selection mechanism is proposed for the convolution kernel in CNN. The SE-block allows each neuron to adjust the size of the receptive field (convolutional kernel) adaptively according to the multi-scale of the input information. Meanwhile, the idea of group convolution is introduced into the attention mechanism, leading to a multiplication reduction of network parameters, an improvement in computational accuracy, and a reduction in computational costs. However, 2D convolution cannot be extended to 3D convolution. The ECA-Net was proposed in 2019, and a series of experiments were conducted to prove the importance of keeping channel numbers constant in the attention block [17]. Appropriate cross-channel learning and non-dimensionality reduction have a positive impact on channel attention learning. However, the channel interaction between the ECA and the spatial attention module has not been discussed. Ref. [18] proposed a novel expansion-squeeze-excitation fusion network for elderly behaviour recognition, and the network performance was improved by changing the SE structure. Meanwhile, it was found that the time dimension of CNN is greatly insufficient, and researchers began to focus on Recurrent Neural Networks (RNN). Long-term Recurrent Convolutional Network (LRCN) [19] was used to extract spatial features that are sent to LSTM. This process realizes end-to-end training and solves the problems of long-term behaviour recognition. By changing the link structure between CNN and LSTM, the problem of the long video sequence of HAR can be solved, thus achieving good results. A deep fusion framework was proposed [20], which strengthens the attention of salient features and enables the network to have better performance. However, the accuracy of behaviour recognition is affected by the choice of a recognition method after feature extraction. A collaborative learning framework was proposed to extract multi-order and multi-stream feature information at hand for fusion [21]. However, this method does not consider how to accurately extract the hand characteristics when the hand is small in the video. To accurately extract behaviour features in the second data fusion stage, previous HMM-based methods [22] need to model symbols one by one and increase the total number of parameters. In prior studies [23, 24], adaptive fusion algorithms were used for multi-modal signals to obtain higher identification accuracy than separate features. 3 PROPOSED METHOD The proposed IALC method for HAR is shown in Figure 1. First, a Kinect device is used to obtain human motion videos. The obtained human behaviour video is input to the colour-related IAL module to obtain RGB sequence features, denoted as IALc. Simultaneously, the skeleton video is input to the same IAL module to obtain skeleton sequence features, denoted as IALg. Finally, these two types of sequence features are fed to the CHMM model to acquire HAR results. FIGURE 1Open in figure viewerPowerPoint Multi-modal fusion based on IALC for HAR The implementation of this method is introduced as follows. Kinect devices are used to acquire RGB video data. The key frames are automatically extracted from the video. Then, the RGB video is converted into a 2D skeleton video using OpenPose. The RGB sequence and 2D skeleton sequence are obtained from IAL, which are denoted as Xr and Xg, respectively. The CHMM is used to fuse the two output results, IALc and IALg, obtained in the feature extraction stage, and the HAR recognition result is calculated. Two-stream data Kinect can accurately collect the human skeleton and RGB video in real-time, and it has been widely applied to human pose detection. To improve the performance of the IALC, this study extracts T key frames from videos and converts them to RGB and skeleton videos. Human skeleton information can be obtained from the HAR videos. IAL network To improve the accuracy of HAR, an appropriate network should be selected for feature extraction. The current mainstream networks used for feature extraction are shown in Table 1. TABLE 1. Classification results of various networks on ImageNet [25] Number Net name Year Top5 err. Net layers 1 AlexNet 2012 15.32 8 2 Overfeet 2013 13.6 8 3 VGG Network 2014 7.32 19 4 GoogleNet 2015 6.67 22 5 ResNet 2015 3.57 50 6 ResNext 2016 3.03 50 7 SE Network 2017 2.25 50 As shown in Table 1, the Squeeze-and-Excitation (SE) Network [8] is used in this study because it shows the best performance in image recognition. The structure of the SE-block is shown in Figure 2 (left), which consists of one global pooling layer, two fully connection (FC) layers, one ReLu layer, and one sigmoid activation layer. Since redundant information and important features have different contributions to the recognition results, this study introduces the SE block to improve the weight of important features. However, it can be seen from Figure 2 (left) that the SE block uses two FC layers, which leads to an exponential increase in network parameter size. Also, the FC layer is directly connected to the sigmoid function, resulting in neuron inactivation. Therefore, to improve SE block performance and reduce excessive computational overhead, this paper proposes the Improved SE-block (ISE-block) in Figure 2 (right) network to extract behavioural features, and it is inserted into ResNet50. The design of the ISE-block and its comparison with the SE-block are shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Comparison of the SE-block and ISE-block As shown in Figure 2, feature X is obtained from the previous layer, and the number of channels is C. It is our proposed module that can be seen in Figure 2 (right). The proposed ISE-block is explained as follows. Firstly, through comparative experiments, ref. [16] proved that non-dimensionality reduction of the SE block has a better function. The number of additional parameters in the SE block is: 2 r ∑ s = 1 s N s · C s 2 $$\begin{equation}\frac{2}{r}\sum_{s = 1}^s {{N}_s \cdot C_s^2} \end{equation}$$ (1)where Ns denotes the number of repeated blocks; r is the reduction ratio; Cs is the dimension of the output channels, and S refers to the number of stages (five stages are shown in Figure 4). The ISE block replaces FC with 1*1 convolution, and the additional parameters are as follows. ∑ s = 1 s N s · C s $$\begin{equation}\sum_{s = 1}^s {{N}_s \cdot {C}_s} \end{equation}$$ (2) The comparison of Equations (1) and (2) indicates that the additional parameters are significantly reduced. Therefore, to avoid excessive computational overhead caused by the increase in parameters, the FC layer is replaced with the 1*1 convolutional layer. Secondly, a Batch Normalization (BN) layer is inserted between the convolutional layer and the sigmoid layer. As shown in Figure 3 when x ≥ | 5 | $x \ge |5|$ , the sigmoid function's value is equal to 1. Thus, to avoid the problem of neuron inactivation, the input of the sigmoid function should be within ±5, and the best way is to add a BN layer before the sigmoid layer. Finally, the ISE-block is integrated into ResNet50 to form ISE-ResNet50 so that the model can automatically focus on important features and suppress irrelevant noise. FIGURE 3Open in figure viewerPowerPoint The curves of two different sigmoid functions The ISE-block is integrated into ResNet50, and the structure of ISE-ResNet50 is illustrated in Figure 4. ISE-ResNet50 consists of five stages. The size of the video input to ISE-ResNet50 is 224×224×3. After a series of convolution operations, a 2048×14×14 feature sequence is obtained. Meanwhile, the video lengths required for different actions are different, and the classic neural network can only accept video input of a fixed length (7 frames), resulting in low accuracy in action recognition for videos of any length. To use more features of video actions, this study connects LSTM behind the ISE-ResNet50 to form the IAL so that the complex HAR of long video sequences can be solved. When RGB videos and skeleton videos are input into the IAL network, two streams of feature sequences, that is, IALc and IALg are obtained. Then, multi-modal features are fed into CHMM, and the probability fusion method of HAR is constructed, which is introduced in detail in the next section. FIGURE 4Open in figure viewerPowerPoint The ISE-ResNet50 structure is made of five stages. The ISE-block is proposed and added into each stage. To change the weight of different features, the first block of every residual contains convolution and BN layers. The LSTM is connected behind ISE-ResNet50 to solve the long timing sequence. Then, the IAL is composed of the ISE-ResNet50 and LSTM. IALC fusion framework To capture the human behavioural characteristics of the elderly more accurately, multi-modal information collected from different perspectives is fused in this study. The specific implementation method is as follows. Firstly, to reduce the influence of the background noise in HAR, the IAL is used to obtain skeleton flow Z g = ( Z i g ) i = 1 T ${Z}^g = ( {Z_i^g} )_{i = 1}^T$ from one view, and Z g = ( Z i g ) i = 1 T ${Z}^g = ( {Z_i^g} )_{i = 1}^T$ indicates that the skeleton modality characteristic matrix from time 1 to time t is Zg. The feature sequence is expressed as h t = ( a , b , c , v ) ${h}_t = ( {a,b,c,v} )$ , which is a middle-hidden variable. A middle-hidden variable of the skeleton modality at t–th is denoted as h t g $h_t^g$ in Figure 5, and h 0 g $h_0^g$ is the initial value. A complete skeleton behaviour is expressed as 4 × 25 = 100 $4 \times 25 = 100$ with a vector dimension of X g = ( x 1 , x 2 , … , x 25 ) ${{\rm{X}}}_g = ( {{x}_1,{x}_2, \ldots ,{x}_{25}} )$ . Secondly, to avoid the occlusion problem, the RGB stream, I c = ( I i c ) i = 1 T ${{\rm{I}}}^c = ( {{\rm{I}}_i^c} )_{i = 1}^T$ is extracted from another perspective, and the IAL is used to directly extract RGB behaviour features. Faintly, the skeleton and RGB feature sequences are fed into CHMM. In this way, the multi-modal fusion HAR method is realized, and it is shown in Figure 5. Different feature vector types are obtained by the IAL and fed into the CHMM model to generate HAR results. X t c $X_t^c$ and X t g $X_t^g$ are respectively the RGB-related and skeleton-related input vectors. h t − 1 ${h}_{t - 1}$ is an intermediate hidden variable. h0 is an initial variable. σ ( · ) $\sigma ( \cdot )$ and s i g m o i d ( · ) $sigmoid( \cdot )$ are activation functions; w represents the network weight, and b is the deviation. The output is expressed as z t = sogmoid ( w z x t + b z ) ${z}_t = {\rm{sogmoid}}( {{w}_z{x}_t + {b}_z} )$ . To facilitate the framework, the skeleton-related and RGB-related information is defined as c and g, respectively. The output of IAL is z t c $z_t^c$ and z t g $z_t^g$ , which are RGB and skeleton sequences respectively, and are fed into CHMM as observation signals. FIGURE 5Open in figure viewerPowerPoint The IALC fusion model for HAR To simplify the model calculation process, CHMM is divided into two basic HMM models involving parameters λ = ( Ψ , A , B ) ${{\lambda}} = ( {\Psi ,{\rm{A}},{\rm{B}}} )$ [8]. The definitions of the parameters are as follows. Ψ represents the initial state, B is the observation matrix, and A is the state transition matrix. The calculation in Figure 5 can be described as z c = ( z 1 c , z 2 c , … , z T c ) ${z}^{\rm{c}} = ( {z_1^{\rm{c}},z_2^{\rm{c}}, \ldots ,z_{\rm{T}}^{\rm{c}}} )$ and z g = ( z 1 g , z 2 g , … , z T g ) ${{\rm{z}}}^g = ( {z_1^g,z_2^g, \ldots ,z_{\rm{T}}^g} )$ , where Z i c ${\rm{Z}}_i^c$ and Z i g ${\rm{Z}}_i^g$ represent the behaviour RGB and skeleton vector at time t respectively. In this study, a hybrid dynamic Bayesian network (DBN) is established, which selects symbolic state w and behaviour observation state Z to represent HAR parameters. Assuming that two stream sequences have the same contribution, the state sequence w g = ( w 1 g , w 2 g , … , w T g ) ${w}^g = ( {w_1^g,w_2^g, \ldots ,w_{\rm{T}}^g} )$ can be updated with w c = ( w 1 c , w 2 c , … , w T c ) ${w}^c = ( {w_1^c,w_2^c, \ldots ,w_{\rm{T}}^c} )$ . Then, through probabilistic reasoning, the information in the two sequences is fused to obtain the final recognition result. The IALC calculation process is shown as follows: Ψ c = p i c 1 × n = p w 0 c A c = a i j c n × n = p w t + 1 c | w t c B c = p z t c | w t c = N μ z c , Σ z c z t c $$\begin{equation} \left\{{ \def\eqcellsep{&}\begin{array}{rcl} {\Psi }^c &=& {{\left[ {p_i^c} \right]}}_{1 \times n} = p\left( {w_0^c} \right)\\[9pt] {{\rm{A}}}^c &=& {{\left[ {a_{ij}^c} \right]}}_{n \times n} = p\left( {w_{t + 1}^c|w_t^c} \right)\\[9pt] {{\rm{B}}}^c &=& p\left( {z_t^c|w_t^c} \right) = N\left( {\mu _z^c,\Sigma _z^c} \right)\left( {z_t^c} \right) \end{array} } \right.\end{equation}$$ (3)where Ψ c ${\Psi }^c$ is the initial distribution of the behaviour-related state w 0 c $w_0^c$ . If w 0 c $w_0^c$ includes n states, it can be expressed as S 1 , … , S n ${S}_1, \ldots ,{S}_n$ , where Si corresponds to P ( w 0 c ) = S i = ψ c ( i ) $P( {w_0^c} ) = {S}_i = {\psi }^c(i)$ at time 1. The state matrix is Ac, and a i j c $a_{ij}^c$ means the state transition probability from time i to time j. Therefore, a i j c = p ( w t + 1 c = s j / w t c = s i ) $a_{ij}^c = p( {w_{t + 1}^c = {s}_j/w_t^c = {s}_i} )$ , where ∑ z ${\sum }_z$ and μZ are the variance and mean, respectively. The observation matrix is Bc, and the continuous variable is Z t c ${\rm{Z}}_t^c$ . p ( z t c / w t c = s i ) $p( {z_t^c/w_t^c = {s}_i} )$ is an observation state probability. The optimal state sequence is denoted as w 1 : t s $w_{1:t}^s$ , and p ( w 1 c ) $p( {w_1^c} )$ is obtained from the Bayesian theory. p w 1 c = ∑ p w 1 c | w 0 c p x 0 c $$\begin{equation}p\left( {w_1^c} \right) = {\sum }_p\left( {w_1^c|w_0^c} \right)p\left( {x_0^c} \right)\end{equation}$$ (4) Where can be calculated by Bayesian rule p ( x 1 ) ∫ x 0 p ( x 1 / x 0 ) · p ( x 0 ) $p( {{x}_1} )\int_{{{x}_0}}{{p( {{x}_1/{x}_0} )}} \cdot p( {{x}_0} )$ . Meanwhile, the observation z1 is produced by the conditional probability rule p ( x / z ) = p ( z / x ) p ( x ) / p ( z ) $p( {x/z} ) = p( {z/x} )p( x )/p( z )$ , and initial time p ( x 0 c ) = p ( w 0 c ) $p( {x_0^c} ) = p( {w_0^c} )$ . Therefore: p w 1 c / z 1 c = p z 1 c / w 1 c p x 1 c p z 1 c = α · p z 1 c | w 1 c p x 1 c $$\begin{equation} \def\eqcellsep{&}\begin{array}{rcl} p\left( {w_1^c/z_1^c} \right) &=& \dfrac{{p\left( {z_1^c/w_1^c} \right)p\left( {x_1^c} \right)}}{{p\left( {z_1^c} \right)}}\\[12pt] &=& \alpha \cdot p\left( {z_1^c|w_1^c} \right)p\left( {x_1^c} \right) \end{array} \end{equation}$$ (5) The time state probability at time t is calculated as: p w t + 1 c = ∑ w t c p w t + 1 c / w t c p w t c $$\begin{equation}p\left( {w_{t + 1}^c} \right) = \mathop \sum \limits_{w_t^c} p\left( {w_{t + 1}^c/w_t^c} \right)p\left( {w_t^c} \right)\end{equation}$$ (6) Then, the state probability can be optimized by the observation of state sequences: p w t + 1 c / z 1 : t + 1 c = α · p z t + 1 c / w t + 1 c p w 1 g p z 1 : t c × p w t c ∑ w t c p w t + 1 c / w t c p w t c / z 1 : t c $$\begin{equation} \def\eqcellsep{&}\begin{array}{rcl} p\left( {w_{t + 1}^c/z_{1:t + 1}^c} \right) &=& \alpha \cdot p\left( {z_{t + 1}^c/w_{t + 1}^c} \right)p\left( {w_1^g} \right)p\left( {z_{1:t}^c} \right)\\[12pt] &&\displaystyle \times\; p\left( {w_t^c} \right)\mathop \sum \limits_{w_t^c} p\left( {w_{t + 1}^c/w_t^c} \right)p\left( {w_t^c/z_{1:t}^c} \right) \end{array} \end{equation}$$ (7) The multi-modal fusion probability is obtained through IALC. Similarly, skeleton-related parameters are calculated in Equation (8): Ψ g = p i g 1 × n = p w 0 g p i g A g = a i j k g n × n = p w t + 1 g / w t g , w t c B g = P w t g / w t g = N u z g , Σ z g z t g $$\begin{equation} \left\{ \def\eqcellsep{&}\begin{array}{rcl} {\Psi }^g &=& {{\left[ {p_i^g} \right]}}_{1 \times n} = p\left( {w_0^g} \right)p_i^g\\[9pt] {A}^g &=& {{\left[ {a_{ijk}^g} \right]}}_{n \times n} = p\left( {w_{t + 1}^g/w_t^g,w_t^c} \right)\\[9pt] {B}^g &=& P\left( {w_t^g/w_t^g} \right) = N\left( {u_z^g,\Sigma _z^g} \right)\left( {z_t^g} \right) \end{array} \right.\end{equation}$$ (8)where Ψ g ${\Psi }^g$ is the initial state of skeleton-related w 0 g $w_0^g$ . If w 0 g $w_0^g$ includes n states that are expressed as ( s 1 , … , s n ) $( {{s}_1, \ldots ,{s}_n} )$ , s i ${s}_i$ corresponds to skeleton state p ( w 0 g = s j ) = Ψ g ( i ) $p( {w_0^g = {s}_j} ) = {\Psi }^g( i )$ at time i. The state transaction matrix is Ag, and the observation matrix is Bg. a i j g ${a}_{ijg}$ is the transition probability from the time i to time j. Therefore, a i j k g = p ( w t + 1 g = s k / w t g = s i , w t c = s j ) $a_{ijk}^g = p( {w_{t + 1}^g = {s}_k/w_t^g = {s}_i,w_t^c = {s}_j} )$ , and z t g $z_t^g$ is a continuous variable. The observation probability p ( z t g / w t g ) = s j $p( {z_t^g/w_t^g} ) = {s}_j$ is obtained. The initial skeleton-related CHMM inference state can be expressed as: p w 1 g = ∑ w 0 g p w 1 g / w 0 g , w 0 c p w 0 g , w 0 c = ∑ w 0 g p w 1 g / w 0 g , w 0 c · p w 0 g p w 0 c $$\begin{equation} \def\eqcellsep{&}\begin{array}{rcl} p\left( {w_1^g} \right) &=&\displaystyle \mathop \sum \limits_{w_0^g} p\left( {w_1^g/w_0^g,w_0^c} \right)p\left( {w_0^g,w_0^c} \right)\\[18pt] &=& \displaystyle \mathop \sum \limits_{w_0^g} p\left( {w_1^g/w_0^g,w_0^c} \right) \cdot p\left( {w_0^g} \right)p\left( {w_0^c} \right) \end{array} \end{equation}$$ (9) As in Equation (5), w 1 g $w_1^g$ can be updated by z 1 g $z_1^g$ : p w 1 g / z 1 g = p z 1 g / w 1 g p w 1 g / p z 1 g = α · p z 1 g / w 1 g · p w 1 g = α · p z 1 g / w 1 g · p w 1 g / w 0 g , w 0 c p w 0 g p w 0 c $$\begin{equation} \def\eqcellsep{&}\begin{array}{rcl} p\left( {w_1^g/z_1^g} \right) &=& p\left( {z_1^g/w_1^g} \right)p\left( {w_1^g} \right)/p\left( {z_1^g} \right)\\[12pt] &=& \alpha \cdot p\left( {z_1^g/w_1^g} \right) \cdot p\left( {w_1^g} \right)\\[12pt] &=& \alpha \cdot p\left( {z_1^g/w_1^g} \right) \cdot p\left( {w_1^g/w_0^g,w_0^c} \right)p\left( {w_0^g} \right)p\left( {w_0^c} \right) \end{array} \end{equation}$$ (10) Generally, the state at t − t h $t{\rm{ - }}th$ is: p w t + 1 g = ∑ w t g p w t + 1 g / w t g , w t c p w t g , w t c = ∑ w t g p w t + 1 g / w t g , w t c · p w t g p w t c $$\begin{equation} \def\eqcellsep{&}\begin{array}{rcl} p\left( {w_{t + 1}^g} \right) &=& \displaystyle \mathop \sum \limits_{w_t^g} p\left( {w_{t + 1}^g/w_t^g,w_t^c} \right)p\left( {w_t^g,w_t^c} \right)\\[18pt] &=& \displaystyle \mathop \sum \limits_{w_t^g} p\left( {w_{t + 1}^g/w_t^g,w_t^c} \right) \cdot p\left( {w_t^g} \right)p\left( {w_t^c} \right) \end{array} \end{equation}$$ (11) The distribution state of c sequences and g sequences at t − t h $t{\rm{ - }}th$ is represented as p ( w t c ) $p( {w_t^c} )$ and p ( w t g ) $p( {w_t^g} )$ , respectively. Then, an optimized state can be calculated from observed sequences: p w t + 1 g / z 1 : t + 1 g = p w t + 1 g / z t + 1 g , z 1 : t g = α · p z t + 1 g / w t + 1 g · p w t + 1 g / w t g , w t c ·