# Fast Recognition Algorithm for Human Motion Posture Using Multimodal Bioinformation Fusion

Authors: Xiangbing Zhao; Jianhui Zhou
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

To address the problems of low feature extraction accuracy, large bias of human motion pose recognition and posture recognition error, poor recognition effect, and low recognition rate of traditional human motion posture fast recognition algorithm, we propose a human motion posture fast recognition algorithm using multimodal bioinformation fusion. First, wavelet packet decomposition with sample entropy is used to extract the human motion posture hand features such as kurtosis, time domain feature skewness, and frequency domain feature electromyogram (EMG) integral value and time domain features such as mean, standard deviation, and interquartile distance of leg motion amplitude. Second, after normalizing the two features, the human hand and leg motion feature set is obtained, and finally the feature set is used to construct a human motion posture fast recognition model based on multimodal bioinformation fusion, and the feature set is input into the recognition model, which completes the fusion of human motion posture information by improving the typical correlation analysis method, and the fusion result is used as the input of the minimum distance classifier to achieve human motion posture fast recognition. The results show that the proposed algorithm has high accuracy of feature extraction, small bias of human motion posture recognition, the posture recognition error is -0.21∼0.02, the recognition rate is always above 95%, and the practical application effect is good.


---

## Body

## 1. Introduction

Data fusion technology is used mainly to classify and process multiple data sources. After be processed, data can be classified into clearer and more perfect categories [1, 2]. Multimodal biological information fusion mainly uses more biological features to obtain more category information [3], that is, it can identify biological information more comprehensively. As there are a variety of human motion postures and different motion features will lead to different motion postures, so how to accurately identify human motion postures has become a topic that needs urgent study at present [4]. The recognition of human motion posture can help to solve problems in many fields, such as the analysis of the motion status of athletes, elderly people, and young people, in order to enhance the monitoring of human health levels [5]. Therefore, it is very important to study the method of human motion posture recognition.More scholars have researched on posture recognition and more excellent research results have been achieved; for example, Mou et al. [6] studied attention-based CNN-LSTM multimodal fusion driver stress detection algorithm using attention-based convolutional neural network (CNN) and long and short-term memory (LSTM) models to fuse nonintrusive data, including eye data, vehicle data, and environmental data. Then, the features are automatically extracted from each modality separately, and the features of different modalities are given different levels of attention by self-attention mechanisms. The multimodal fusion driver stress detection is achieved using different attention mechanisms. However, the recognition effect of this algorithm is not perfect, and the information of biometric features is not fully fused when fusing motion features. Zhang et al. [7] studied fall detection based on the spatiotemporal evolution of human posture, using an improved two-branch multilevel convolutional neural network to extract and construct the inverted pendulum structure of human posture in real complex scenes. Multimedia analysis is used to observe the time series changes of the human inverted pendulum structure and construct the spatio-temporal evolution of human posture motion. The visual features of the spatio-temporal evolution of human posture under potential instability are analyzed, and two key features of human falling behavior are explored, which are used for posture recognition and fall recognition. However, the algorithm is only able to recognize human fall posture and is less in recognizing other human motion situations. Han et al. [8] studied a two-stage fall recognition algorithm based on human posture features, which extracted human posture features when the human was in an unstable state by classifying the human state into three states: stable state, fluctuating state, and disordered state based on the trend sign and stable sign variables integrated by the scattered key features. Support vector machine, K-nearest neighbor, decision tree, and random forest are used to classify and combine the classification results to achieve human fall posture detection. However, the algorithm cannot recognize the human at rest and therefore has a weak recognition capability. Ding et al. [9] studied online adaptive prediction of human motion intention based on surface electromyogram (EMG) signals and designed a surface EMG feature extraction network and an online adaptive network. Highly compressed surface EMG features were obtained using a convolutional self-coding network combined with muscle synergistic features to assist motion prediction, and human motion posture recognition was achieved by combining the motion prediction results. However, the algorithm cannot accurately extract human motion features, leading to errors in recognition. Olivas-Padilla et al. [10] studied wearable devices for stochastic biomechanical modeling and recognition of human motion, constructing a gesture-based manipulation model, which uses an autoregressive model to learn the dynamics of joints by assuming associations between them. The statistical significance of each model hypothesis is calculated to identify the joints most involved in the motion, and this result is combined to achieve human motion posture recognition. However, the algorithm is more based on the device to achieve recognition, and if the device is abnormal, it leads to a decrease in the level of recognition.To solve the problems of the above algorithms, we propose a fast human motion posture recognition algorithm using multimodal bioinformation fusion. The main contributions of this paper are as follows: (1) accurate recognition of posture is achieved by fusion of multimodal biometric information. (2) The time domain features of human motion posture hand features and leg motion amplitude are extracted, using these features can lay a solid foundation for the subsequent accurate recognition of human motion posture. (3) The fusion of human motion posture information is completed by improving the typical correlation analysis method, which improves the fusion accuracy and speed, and combines the fusion results to achieve fast human posture recognition, thus improving the recognition accuracy and efficiency.


## 2. Methodology

### 2.1. Human Motion Pose Feature Extraction

#### 2.1.1. Hand Motion Feature Extraction in Human Motion Postures

In this study, the characteristic peak value and skewness of hand characteristic EMG signal and intensity in human motion are taken as the characteristics of human motion postures, and these features are extracted by wavelet packet decomposition and sample entropy method, which can be used as the basic information of human motion posture recognition [11, 12].Suppose the power signal of hand muscle issEMG. Since bioenergy is usually in the low-frequency posture of 10 ∼ 250 Hz, the “db5” wavelet basis function is used. The muscle power signal sEMG is decomposed by four layers of wavelet packet. At the same time, the signals in the first eight subspaces during the fourth layer decomposition are set as the extraction content of sample entropy, that is, the sample entropy of superficial flexor muscle of hand and brachioradialis muscle are extracted and set as Sove1 and Sove2.When the number of wavelet decomposition layers increases, the spatial resolution will decline [13, 14]. Therefore, this paper selects the kurtosis k, time-domain characteristic skewness η, and frequency-domain characteristic iEMG integral value in the muscle power signal sEMG as the features extracted in this paper. The feature parameters are defined as follows:(1)ηi=NN−1N−2∑i=0Nxti−M^uσ^u3,(2)ki=1/N∑i=0Nuti−M^u41/N∑i=0Nxi−M^u22,(3)iEMGi=∫0Nutidt,where the sequence value of muscle electronic signal is described by uti; the values of i are 1 and 2, respectively, which represent the sEMG signals of superficial flexor muscle and brachioradialis muscle in turn; N represents the discrete EMG sequence; at the same time, the mean and variance of EMG signal sequence are represented by M^u and σ^u, respectively.SupposeFeature1 is the combined characteristics of each hand motion. The vector composition of the combined features is expressed by the following equation:(4)Feature1=Sove1η1k1iEMG1,Sove2η2k2iEMG2.


#### 2.1.2. Leg Motion Feature Extraction in Human Motion Postures

Human motion postures are composed of motion characteristics of various parts. This paper extracts the leg motion features during human motion, mainly analyzes the time-domain features of the leg motion amplitude during motion, which are expressed as mean, standard deviation, interquartile distance, and so on, and extracts the detailed features in the following ways:The number of quantities trending in a certain group of data sets [15, 16]. , the mean, can be obtained by adding all the data within the group and dividing it by the number of data in the whole group, and the mean is calculated as shown in the equation (5). The specific calculation is shown as follows:(5)Mean=1n∑i=1noi,where n denotes the number of data in the group and oi denotes the sum of the data. The standard deviation refers to the arithmetic square root of the square of the deviation between the standard value and its average in all data [17].The dispersion of the data set can be displayed through the standard deviation [18]. Therefore, the standard deviation of the data in the window can represent the intensity of human leg activity at this stage. Set the standard deviation as Std, which can be calculated by(6)Std1n∑i=1noi−o¯2.Skewness can describe the symmetry of sample data distribution, which can analyze the symmetry of time series data. If the skewness is greater than zero, it indicates that the sample data distribution is positive. If the skewness is less than zero, it indicates that the data distribution is negative [19]. Set the skewness as Skewness, which can be calculated by(7)Skewness=∑i=1noi−o¯3n−1std3.SupposeFeature2 is the combined characteristics of each leg motion. The vector composition of the combined features is expressed by(8)Feature2=Mean,Std,Skewness.


### 2.2. Fast Fusion Recognition Model for Human Motion Posture

In this paper, based on the extraction of two kinds of features, we propose a fusion recognition method based on multimodal bioinformation, and the whole fusion recognition process is represented in Figure1.Figure 1
Process of rapid recognition of human motion posture.According to Figure1, in the process of human motion posture fast recognition, the hand and leg motion data are extracted first, the motion features of the hand and leg are extracted separately [20], then the features are normalized to obtain the human hand and leg motion feature sets, the fusion of human motion posture information is completed by improving the typical correlation analysis method, and the fusion results are used as the input of the minimum distance classifier to output the human motion posture fast recognition results.


#### 2.2.1. Multimodal Biometric Fusion Recognition Based on Improved Typical Correlation Analysis

Canonical correlation analysis method mainly analyzes the feature vectors of samples. In this paper, the method is improved, and the samples are analyzed based on the characteristic matrix of the samples. It is assumed that the characteristic combination matrices of hand and leg motion and areX=X∈Rmx×nx and Y=Y∈Rmy×ny. The typical correlation matrix form is expressed through(9)ZX′ZY′XYRXRY.This paper improves the canonical correlation analysis matrix, so that the category information of training samples can be further used and the recognition results can be more accurate.Suppose that the left transformation matrices ofX and Y are ZX and ZY, and the right transformation matrices are RX and RY. If the first pair of typical related variables of X and Y are a′X and b′Y, after improvement, the maximum correlation between a′X and b′Y is adjusted to the maximum correlation between matrix ZX′XRX and ZY′YRY. The improved objective function of the method can be obtained by introducing matrices ZX′XRX and ZY′YRY into the criterion function in the typical correlation analysis method:(10)JZX′,RX,ZY′,RY=covZX′XRX,ZY′YRYvarZX′XRXvarZY′YRY.ReplaceX and Y in (9) with SWX and SWY. After replacement, the improved canonical correlation analysis method (MDCCA) criterion function can be obtained according to the criterion function in the canonical correlation analysis method, as shown in(11)JMDCCAZX′,RX,ZY′,RY=covZX′SWXRX,ZY′SWYRYvarZX′XRXvarZY′YRY.Due to the theory of traditional canonical correlation analysis method, certain constraints need to be met when solving (11), as shown in:(12)varZX′XRX=varZY′YRY=1Therefore, the improved criterion function in this paper can be expressed by(13)JMDCCAZX′,RX,ZY′,RY=covZX′SWXRX,ZY′SWYRY.At this moment, the solutions to the projection vectorai and bi are transformed into the solutions to the projection matrix ZX, ZY, RX, and RY.


#### 2.2.2. Solution to Improved Canonical Correlation Analysis

When improving the traditional canonical correlation analysis method, this paper first adds the intraclass divergence matrix to the original method and then looks for the left-right transformation matrixZX, ZY, RX, and RY, which can effectively improve the correlation of canonical correlation matrices ZX′SWXRX and ZY′SWYRY. At the same time, the left-right transformation matrix also represents the projection matrix. According to the constraints of (12) and the criterion function of (13), at the same time, it is assumed that the p∗p-order covariance matrix of X is ∑11, the q∗q covariance matrix of Y is ∑12, p<q, and the covariance matrix between X and Y is ∑12=∑21′, and then(14)∑11R=∑i=1NSWXi−S¯WXRXRX′SWXi−S¯WX′N,∑22R=∑i=1NSWYi−S¯WYRYRY′SWYi−S¯WY′N,∑12R=∑i=1NSWXi−S¯WXRXRY′SWYi−S¯WY′N.Thus, it can be seen thatcovZX′SWXRX,ZY′SWYRY=ZX′∑12RZY. When the criterion function of (13) is in a state that can satisfy ZX′∑11RZX=1 and ZY′∑22RZY=1, it can be equivalent to(15)JMDCCAZX′,RX,ZY′,RY=maxJZX′∑12RZYFrom the above calculation process, the following results can be obtained:(16)∑11Z=∑i=1NSWXi−S¯WX′ZXZX′SWXi−S¯WXN,(17)∑22Z=∑i=1NSWYi−S¯WY′ZYZY′SWYi−S¯WYN,(18)∑12Z=∑i=1NSWXi−S¯WX′ZXZY′SWYi−S¯WYN.Meanwhile,∑21Z∑21Z′. When RX′∑11ZRX=1 and RY′∑22ZRY=1, (13) is equivalent to(19)JMDCCAZX′,RX,ZY′,RY=maxJRX′∑12ZRY.Introduce the setRX and RY into (15) to get the solutions to ZX and ZY. Then, use the solutions to get ZX and ZY and introduce them into (19) to calculate RX and RY. Iteration is realized after continuous cyclic calculation.WhenRX and RY are given, the optimization of the criterion function of (16) is changed to the generalized eigenvalue solution, as shown in(20)0∑12R∑21R0ZXZY=λ∑11R00∑22RZXZY.WhenRX and RY are given, the optimization of the criterion function of (16) is changed to the generalized eigenvalue solution, as shown in(21)0∑12Z∑21Z0RXRY=λ∑11Z00∑22ZZXZY.Thus, the solution process is repeated until the convergence effect is achieved, that is, the projection matrixZX, ZY, RX, and RY can be obtained. Obtain the maximum value of (22) by using Lagrange multiplier method:(22)GZ=ZX′∑12RZY−λZXZX′∑11RZX−12−λZYZY′∑22RZY−12.Lagrange multiplier factors are represented byλZX and λZY, respectively. Use GZ to get the partial derivative ZX and ZY to make the final value as zero, as shown in(23)∂GZ∂LX=∑12RZY−λZX∑11RZX=0,∂GZ∂LY=∑21RZX−λZY∑22RZY=0.ZX′ and ZY′ are multiplied by the left multiplication in (23) in turn to obtain ZX′∑12RZY=λZXZX′∑11RZX=λZX, ZY′∑21RZX=λZYZY′∑22RZY=λZY.The above calculation shows that λ=λZX=λZY.Therefore, the solution of (19) can be realized by using (23).If matrix∑22R is a convertible matrix, ZY=λ−1∑22R−1∑21RZX and ∑12R∑22R−1∑21rZX=λ2∑11RZX, i.e., through the calculation of ZX and ZY, the generalized eigenvector of (20) can be obtained. If the given contents are ZX and ZY, through the calculation of (21), RX and RY can be obtained.ZX, ZY, RX, and RY are iterated continuously through the above process until the result converges. Then, the projection matrix is calculated, and the matrix X and y are projected into the projection matrix in turn to form a typical correlation matrix.


#### 2.2.3. Fast Fusion Recognition Algorithm Process of Human Motion Postures

Fast recognition of human motion postures based on multimodal biological information fusion is realized through the following steps:(1)
Two groups of normalized features can be obtained by using the algorithm.(2)
The left and right transformation matricesZX, ZY, RX, and RY are given according to the MDCCA theory. For the training samples X and Y, the intraclass divergence matrix SWX and SWY is given in turn, and the objective function is set.(3)
Equations (20) and (21) are iteratively calculated through the criterion function until the result converges, and the value determined ZX, ZY, RX, and RY in turn.(4)
Map samplesX and Y to the projection matrices ZX, ZY, RX, and RY to obtain the typical correlation matrix ZX′XRX and ZY′YRY after projection.(5)
Taking the matrixZX′XRX and ZY′YRY and the data X as the input of the minimum distance classifier, the minimum distance classifier is used to realize the rapid recognition of human motion postures. In the minimum distance classifier, each category wj is represented by the mean vector ϑj, that is, the vector is described by the mean vector of the feature vector family of each sample, that is: ϑj=1/τj∑x∈wjx, in which τj refers to the number of training vectors, through which the summation is realized. At the same time, the category of unknown sample vector x is determined through the vector two norm, that is, the distance between sample vectors is analyzed, and the recognition process of human motion postures is realized through the measurement of distance.


### 2.3. Dataset and Experimental Index

Two data sets were selected for experimental analysis, Human3.6 M dataset and CMU Panoptic data set. Human3.6 M data set includes 3.6 million 3D human postures and corresponding images, a total of 11 experimenters, and a total of 17 action scenes. To use this data set for posture recognition can improve the simulation accuracy. The CMU Panoptic data set is produced by CMU University and consists of 480 VGA cameras, 30+ HD cameras, and 10 Kinnect sensors, which are installed in the lab or in the subject’s body to collect the subject’s motion data. The overall number of experimental samples collected during the experiment was 4000 image samples, of which 3000 images were used as training samples and the others were used as 1000 images as test samples.The proposed algorithm is compared with the multimodal fusion driver stress detection algorithm based on attention CNN LSTM in literature [6], the fall detection algorithm based on the spatiotemporal evolution of human posture in literature [7], the two-stage fall recognition algorithm based on human posture features in literature [8], the online adaptive prediction algorithm of human motion intention based on surface EMG signal in literature [9], and the stochastic biomechanical modeling and recognition algorithm of wearable devices for human motion in literature [10] and analyze the recognition performance of different algorithms.Feature extraction: the general case hand motion feature amplitude is between [-1.8,1.5] and leg motion feature value amplitude is between [-3,3], if the feature value extracted in proposed algorithm is closer to the interval, the higher the feature extraction accuracy.Human motion posture skewness: the equation for calculating this index is as follows:(24)K=k1−k2.Posture recognition error: this value is the difference between the actual human motion posture and the posture recognized by the proposed algorithm, which is calculated as follows:(25)a=b−c.ROC curve: receiver operating characteristic curve is the line of points drawn under a specific stimulus condition with the probability of false alarm P(y/N) obtained by the subject under different judgment criteria as the horizontal coordinate and the probability of hit P(y/SN) as the vertical coordinate.Recognition rate: this index refers to the ratio of the amount of data of the correctly recognized human motion posture to the total amount of experimental data, and the equation for calculating this index is as follows:(26)Z=zizj×100%.


## 2.1. Human Motion Pose Feature Extraction

### 2.1.1. Hand Motion Feature Extraction in Human Motion Postures

In this study, the characteristic peak value and skewness of hand characteristic EMG signal and intensity in human motion are taken as the characteristics of human motion postures, and these features are extracted by wavelet packet decomposition and sample entropy method, which can be used as the basic information of human motion posture recognition [11, 12].Suppose the power signal of hand muscle issEMG. Since bioenergy is usually in the low-frequency posture of 10 ∼ 250 Hz, the “db5” wavelet basis function is used. The muscle power signal sEMG is decomposed by four layers of wavelet packet. At the same time, the signals in the first eight subspaces during the fourth layer decomposition are set as the extraction content of sample entropy, that is, the sample entropy of superficial flexor muscle of hand and brachioradialis muscle are extracted and set as Sove1 and Sove2.When the number of wavelet decomposition layers increases, the spatial resolution will decline [13, 14]. Therefore, this paper selects the kurtosis k, time-domain characteristic skewness η, and frequency-domain characteristic iEMG integral value in the muscle power signal sEMG as the features extracted in this paper. The feature parameters are defined as follows:(1)ηi=NN−1N−2∑i=0Nxti−M^uσ^u3,(2)ki=1/N∑i=0Nuti−M^u41/N∑i=0Nxi−M^u22,(3)iEMGi=∫0Nutidt,where the sequence value of muscle electronic signal is described by uti; the values of i are 1 and 2, respectively, which represent the sEMG signals of superficial flexor muscle and brachioradialis muscle in turn; N represents the discrete EMG sequence; at the same time, the mean and variance of EMG signal sequence are represented by M^u and σ^u, respectively.SupposeFeature1 is the combined characteristics of each hand motion. The vector composition of the combined features is expressed by the following equation:(4)Feature1=Sove1η1k1iEMG1,Sove2η2k2iEMG2.


### 2.1.2. Leg Motion Feature Extraction in Human Motion Postures

Human motion postures are composed of motion characteristics of various parts. This paper extracts the leg motion features during human motion, mainly analyzes the time-domain features of the leg motion amplitude during motion, which are expressed as mean, standard deviation, interquartile distance, and so on, and extracts the detailed features in the following ways:The number of quantities trending in a certain group of data sets [15, 16]. , the mean, can be obtained by adding all the data within the group and dividing it by the number of data in the whole group, and the mean is calculated as shown in the equation (5). The specific calculation is shown as follows:(5)Mean=1n∑i=1noi,where n denotes the number of data in the group and oi denotes the sum of the data. The standard deviation refers to the arithmetic square root of the square of the deviation between the standard value and its average in all data [17].The dispersion of the data set can be displayed through the standard deviation [18]. Therefore, the standard deviation of the data in the window can represent the intensity of human leg activity at this stage. Set the standard deviation as Std, which can be calculated by(6)Std1n∑i=1noi−o¯2.Skewness can describe the symmetry of sample data distribution, which can analyze the symmetry of time series data. If the skewness is greater than zero, it indicates that the sample data distribution is positive. If the skewness is less than zero, it indicates that the data distribution is negative [19]. Set the skewness as Skewness, which can be calculated by(7)Skewness=∑i=1noi−o¯3n−1std3.SupposeFeature2 is the combined characteristics of each leg motion. The vector composition of the combined features is expressed by(8)Feature2=Mean,Std,Skewness.


## 2.1.1. Hand Motion Feature Extraction in Human Motion Postures

In this study, the characteristic peak value and skewness of hand characteristic EMG signal and intensity in human motion are taken as the characteristics of human motion postures, and these features are extracted by wavelet packet decomposition and sample entropy method, which can be used as the basic information of human motion posture recognition [11, 12].Suppose the power signal of hand muscle issEMG. Since bioenergy is usually in the low-frequency posture of 10 ∼ 250 Hz, the “db5” wavelet basis function is used. The muscle power signal sEMG is decomposed by four layers of wavelet packet. At the same time, the signals in the first eight subspaces during the fourth layer decomposition are set as the extraction content of sample entropy, that is, the sample entropy of superficial flexor muscle of hand and brachioradialis muscle are extracted and set as Sove1 and Sove2.When the number of wavelet decomposition layers increases, the spatial resolution will decline [13, 14]. Therefore, this paper selects the kurtosis k, time-domain characteristic skewness η, and frequency-domain characteristic iEMG integral value in the muscle power signal sEMG as the features extracted in this paper. The feature parameters are defined as follows:(1)ηi=NN−1N−2∑i=0Nxti−M^uσ^u3,(2)ki=1/N∑i=0Nuti−M^u41/N∑i=0Nxi−M^u22,(3)iEMGi=∫0Nutidt,where the sequence value of muscle electronic signal is described by uti; the values of i are 1 and 2, respectively, which represent the sEMG signals of superficial flexor muscle and brachioradialis muscle in turn; N represents the discrete EMG sequence; at the same time, the mean and variance of EMG signal sequence are represented by M^u and σ^u, respectively.SupposeFeature1 is the combined characteristics of each hand motion. The vector composition of the combined features is expressed by the following equation:(4)Feature1=Sove1η1k1iEMG1,Sove2η2k2iEMG2.


## 2.1.2. Leg Motion Feature Extraction in Human Motion Postures

Human motion postures are composed of motion characteristics of various parts. This paper extracts the leg motion features during human motion, mainly analyzes the time-domain features of the leg motion amplitude during motion, which are expressed as mean, standard deviation, interquartile distance, and so on, and extracts the detailed features in the following ways:The number of quantities trending in a certain group of data sets [15, 16]. , the mean, can be obtained by adding all the data within the group and dividing it by the number of data in the whole group, and the mean is calculated as shown in the equation (5). The specific calculation is shown as follows:(5)Mean=1n∑i=1noi,where n denotes the number of data in the group and oi denotes the sum of the data. The standard deviation refers to the arithmetic square root of the square of the deviation between the standard value and its average in all data [17].The dispersion of the data set can be displayed through the standard deviation [18]. Therefore, the standard deviation of the data in the window can represent the intensity of human leg activity at this stage. Set the standard deviation as Std, which can be calculated by(6)Std1n∑i=1noi−o¯2.Skewness can describe the symmetry of sample data distribution, which can analyze the symmetry of time series data. If the skewness is greater than zero, it indicates that the sample data distribution is positive. If the skewness is less than zero, it indicates that the data distribution is negative [19]. Set the skewness as Skewness, which can be calculated by(7)Skewness=∑i=1noi−o¯3n−1std3.SupposeFeature2 is the combined characteristics of each leg motion. The vector composition of the combined features is expressed by(8)Feature2=Mean,Std,Skewness.


## 2.2. Fast Fusion Recognition Model for Human Motion Posture

In this paper, based on the extraction of two kinds of features, we propose a fusion recognition method based on multimodal bioinformation, and the whole fusion recognition process is represented in Figure1.Figure 1
Process of rapid recognition of human motion posture.According to Figure1, in the process of human motion posture fast recognition, the hand and leg motion data are extracted first, the motion features of the hand and leg are extracted separately [20], then the features are normalized to obtain the human hand and leg motion feature sets, the fusion of human motion posture information is completed by improving the typical correlation analysis method, and the fusion results are used as the input of the minimum distance classifier to output the human motion posture fast recognition results.


### 2.2.1. Multimodal Biometric Fusion Recognition Based on Improved Typical Correlation Analysis

Canonical correlation analysis method mainly analyzes the feature vectors of samples. In this paper, the method is improved, and the samples are analyzed based on the characteristic matrix of the samples. It is assumed that the characteristic combination matrices of hand and leg motion and areX=X∈Rmx×nx and Y=Y∈Rmy×ny. The typical correlation matrix form is expressed through(9)ZX′ZY′XYRXRY.This paper improves the canonical correlation analysis matrix, so that the category information of training samples can be further used and the recognition results can be more accurate.Suppose that the left transformation matrices ofX and Y are ZX and ZY, and the right transformation matrices are RX and RY. If the first pair of typical related variables of X and Y are a′X and b′Y, after improvement, the maximum correlation between a′X and b′Y is adjusted to the maximum correlation between matrix ZX′XRX and ZY′YRY. The improved objective function of the method can be obtained by introducing matrices ZX′XRX and ZY′YRY into the criterion function in the typical correlation analysis method:(10)JZX′,RX,ZY′,RY=covZX′XRX,ZY′YRYvarZX′XRXvarZY′YRY.ReplaceX and Y in (9) with SWX and SWY. After replacement, the improved canonical correlation analysis method (MDCCA) criterion function can be obtained according to the criterion function in the canonical correlation analysis method, as shown in(11)JMDCCAZX′,RX,ZY′,RY=covZX′SWXRX,ZY′SWYRYvarZX′XRXvarZY′YRY.Due to the theory of traditional canonical correlation analysis method, certain constraints need to be met when solving (11), as shown in:(12)varZX′XRX=varZY′YRY=1Therefore, the improved criterion function in this paper can be expressed by(13)JMDCCAZX′,RX,ZY′,RY=covZX′SWXRX,ZY′SWYRY.At this moment, the solutions to the projection vectorai and bi are transformed into the solutions to the projection matrix ZX, ZY, RX, and RY.


### 2.2.2. Solution to Improved Canonical Correlation Analysis

When improving the traditional canonical correlation analysis method, this paper first adds the intraclass divergence matrix to the original method and then looks for the left-right transformation matrixZX, ZY, RX, and RY, which can effectively improve the correlation of canonical correlation matrices ZX′SWXRX and ZY′SWYRY. At the same time, the left-right transformation matrix also represents the projection matrix. According to the constraints of (12) and the criterion function of (13), at the same time, it is assumed that the p∗p-order covariance matrix of X is ∑11, the q∗q covariance matrix of Y is ∑12, p<q, and the covariance matrix between X and Y is ∑12=∑21′, and then(14)∑11R=∑i=1NSWXi−S¯WXRXRX′SWXi−S¯WX′N,∑22R=∑i=1NSWYi−S¯WYRYRY′SWYi−S¯WY′N,∑12R=∑i=1NSWXi−S¯WXRXRY′SWYi−S¯WY′N.Thus, it can be seen thatcovZX′SWXRX,ZY′SWYRY=ZX′∑12RZY. When the criterion function of (13) is in a state that can satisfy ZX′∑11RZX=1 and ZY′∑22RZY=1, it can be equivalent to(15)JMDCCAZX′,RX,ZY′,RY=maxJZX′∑12RZYFrom the above calculation process, the following results can be obtained:(16)∑11Z=∑i=1NSWXi−S¯WX′ZXZX′SWXi−S¯WXN,(17)∑22Z=∑i=1NSWYi−S¯WY′ZYZY′SWYi−S¯WYN,(18)∑12Z=∑i=1NSWXi−S¯WX′ZXZY′SWYi−S¯WYN.Meanwhile,∑21Z∑21Z′. When RX′∑11ZRX=1 and RY′∑22ZRY=1, (13) is equivalent to(19)JMDCCAZX′,RX,ZY′,RY=maxJRX′∑12ZRY.Introduce the setRX and RY into (15) to get the solutions to ZX and ZY. Then, use the solutions to get ZX and ZY and introduce them into (19) to calculate RX and RY. Iteration is realized after continuous cyclic calculation.WhenRX and RY are given, the optimization of the criterion function of (16) is changed to the generalized eigenvalue solution, as shown in(20)0∑12R∑21R0ZXZY=λ∑11R00∑22RZXZY.WhenRX and RY are given, the optimization of the criterion function of (16) is changed to the generalized eigenvalue solution, as shown in(21)0∑12Z∑21Z0RXRY=λ∑11Z00∑22ZZXZY.Thus, the solution process is repeated until the convergence effect is achieved, that is, the projection matrixZX, ZY, RX, and RY can be obtained. Obtain the maximum value of (22) by using Lagrange multiplier method:(22)GZ=ZX′∑12RZY−λZXZX′∑11RZX−12−λZYZY′∑22RZY−12.Lagrange multiplier factors are represented byλZX and λZY, respectively. Use GZ to get the partial derivative ZX and ZY to make the final value as zero, as shown in(23)∂GZ∂LX=∑12RZY−λZX∑11RZX=0,∂GZ∂LY=∑21RZX−λZY∑22RZY=0.ZX′ and ZY′ are multiplied by the left multiplication in (23) in turn to obtain ZX′∑12RZY=λZXZX′∑11RZX=λZX, ZY′∑21RZX=λZYZY′∑22RZY=λZY.The above calculation shows that λ=λZX=λZY.Therefore, the solution of (19) can be realized by using (23).If matrix∑22R is a convertible matrix, ZY=λ−1∑22R−1∑21RZX and ∑12R∑22R−1∑21rZX=λ2∑11RZX, i.e., through the calculation of ZX and ZY, the generalized eigenvector of (20) can be obtained. If the given contents are ZX and ZY, through the calculation of (21), RX and RY can be obtained.ZX, ZY, RX, and RY are iterated continuously through the above process until the result converges. Then, the projection matrix is calculated, and the matrix X and y are projected into the projection matrix in turn to form a typical correlation matrix.


### 2.2.3. Fast Fusion Recognition Algorithm Process of Human Motion Postures

Fast recognition of human motion postures based on multimodal biological information fusion is realized through the following steps:(1)
Two groups of normalized features can be obtained by using the algorithm.(2)
The left and right transformation matricesZX, ZY, RX, and RY are given according to the MDCCA theory. For the training samples X and Y, the intraclass divergence matrix SWX and SWY is given in turn, and the objective function is set.(3)
Equations (20) and (21) are iteratively calculated through the criterion function until the result converges, and the value determined ZX, ZY, RX, and RY in turn.(4)
Map samplesX and Y to the projection matrices ZX, ZY, RX, and RY to obtain the typical correlation matrix ZX′XRX and ZY′YRY after projection.(5)
Taking the matrixZX′XRX and ZY′YRY and the data X as the input of the minimum distance classifier, the minimum distance classifier is used to realize the rapid recognition of human motion postures. In the minimum distance classifier, each category wj is represented by the mean vector ϑj, that is, the vector is described by the mean vector of the feature vector family of each sample, that is: ϑj=1/τj∑x∈wjx, in which τj refers to the number of training vectors, through which the summation is realized. At the same time, the category of unknown sample vector x is determined through the vector two norm, that is, the distance between sample vectors is analyzed, and the recognition process of human motion postures is realized through the measurement of distance.


## 2.2.1. Multimodal Biometric Fusion Recognition Based on Improved Typical Correlation Analysis

Canonical correlation analysis method mainly analyzes the feature vectors of samples. In this paper, the method is improved, and the samples are analyzed based on the characteristic matrix of the samples. It is assumed that the characteristic combination matrices of hand and leg motion and areX=X∈Rmx×nx and Y=Y∈Rmy×ny. The typical correlation matrix form is expressed through(9)ZX′ZY′XYRXRY.This paper improves the canonical correlation analysis matrix, so that the category information of training samples can be further used and the recognition results can be more accurate.Suppose that the left transformation matrices ofX and Y are ZX and ZY, and the right transformation matrices are RX and RY. If the first pair of typical related variables of X and Y are a′X and b′Y, after improvement, the maximum correlation between a′X and b′Y is adjusted to the maximum correlation between matrix ZX′XRX and ZY′YRY. The improved objective function of the method can be obtained by introducing matrices ZX′XRX and ZY′YRY into the criterion function in the typical correlation analysis method:(10)JZX′,RX,ZY′,RY=covZX′XRX,ZY′YRYvarZX′XRXvarZY′YRY.ReplaceX and Y in (9) with SWX and SWY. After replacement, the improved canonical correlation analysis method (MDCCA) criterion function can be obtained according to the criterion function in the canonical correlation analysis method, as shown in(11)JMDCCAZX′,RX,ZY′,RY=covZX′SWXRX,ZY′SWYRYvarZX′XRXvarZY′YRY.Due to the theory of traditional canonical correlation analysis method, certain constraints need to be met when solving (11), as shown in:(12)varZX′XRX=varZY′YRY=1Therefore, the improved criterion function in this paper can be expressed by(13)JMDCCAZX′,RX,ZY′,RY=covZX′SWXRX,ZY′SWYRY.At this moment, the solutions to the projection vectorai and bi are transformed into the solutions to the projection matrix ZX, ZY, RX, and RY.


## 2.2.2. Solution to Improved Canonical Correlation Analysis

When improving the traditional canonical correlation analysis method, this paper first adds the intraclass divergence matrix to the original method and then looks for the left-right transformation matrixZX, ZY, RX, and RY, which can effectively improve the correlation of canonical correlation matrices ZX′SWXRX and ZY′SWYRY. At the same time, the left-right transformation matrix also represents the projection matrix. According to the constraints of (12) and the criterion function of (13), at the same time, it is assumed that the p∗p-order covariance matrix of X is ∑11, the q∗q covariance matrix of Y is ∑12, p<q, and the covariance matrix between X and Y is ∑12=∑21′, and then(14)∑11R=∑i=1NSWXi−S¯WXRXRX′SWXi−S¯WX′N,∑22R=∑i=1NSWYi−S¯WYRYRY′SWYi−S¯WY′N,∑12R=∑i=1NSWXi−S¯WXRXRY′SWYi−S¯WY′N.Thus, it can be seen thatcovZX′SWXRX,ZY′SWYRY=ZX′∑12RZY. When the criterion function of (13) is in a state that can satisfy ZX′∑11RZX=1 and ZY′∑22RZY=1, it can be equivalent to(15)JMDCCAZX′,RX,ZY′,RY=maxJZX′∑12RZYFrom the above calculation process, the following results can be obtained:(16)∑11Z=∑i=1NSWXi−S¯WX′ZXZX′SWXi−S¯WXN,(17)∑22Z=∑i=1NSWYi−S¯WY′ZYZY′SWYi−S¯WYN,(18)∑12Z=∑i=1NSWXi−S¯WX′ZXZY′SWYi−S¯WYN.Meanwhile,∑21Z∑21Z′. When RX′∑11ZRX=1 and RY′∑22ZRY=1, (13) is equivalent to(19)JMDCCAZX′,RX,ZY′,RY=maxJRX′∑12ZRY.Introduce the setRX and RY into (15) to get the solutions to ZX and ZY. Then, use the solutions to get ZX and ZY and introduce them into (19) to calculate RX and RY. Iteration is realized after continuous cyclic calculation.WhenRX and RY are given, the optimization of the criterion function of (16) is changed to the generalized eigenvalue solution, as shown in(20)0∑12R∑21R0ZXZY=λ∑11R00∑22RZXZY.WhenRX and RY are given, the optimization of the criterion function of (16) is changed to the generalized eigenvalue solution, as shown in(21)0∑12Z∑21Z0RXRY=λ∑11Z00∑22ZZXZY.Thus, the solution process is repeated until the convergence effect is achieved, that is, the projection matrixZX, ZY, RX, and RY can be obtained. Obtain the maximum value of (22) by using Lagrange multiplier method:(22)GZ=ZX′∑12RZY−λZXZX′∑11RZX−12−λZYZY′∑22RZY−12.Lagrange multiplier factors are represented byλZX and λZY, respectively. Use GZ to get the partial derivative ZX and ZY to make the final value as zero, as shown in(23)∂GZ∂LX=∑12RZY−λZX∑11RZX=0,∂GZ∂LY=∑21RZX−λZY∑22RZY=0.ZX′ and ZY′ are multiplied by the left multiplication in (23) in turn to obtain ZX′∑12RZY=λZXZX′∑11RZX=λZX, ZY′∑21RZX=λZYZY′∑22RZY=λZY.The above calculation shows that λ=λZX=λZY.Therefore, the solution of (19) can be realized by using (23).If matrix∑22R is a convertible matrix, ZY=λ−1∑22R−1∑21RZX and ∑12R∑22R−1∑21rZX=λ2∑11RZX, i.e., through the calculation of ZX and ZY, the generalized eigenvector of (20) can be obtained. If the given contents are ZX and ZY, through the calculation of (21), RX and RY can be obtained.ZX, ZY, RX, and RY are iterated continuously through the above process until the result converges. Then, the projection matrix is calculated, and the matrix X and y are projected into the projection matrix in turn to form a typical correlation matrix.


## 2.2.3. Fast Fusion Recognition Algorithm Process of Human Motion Postures

Fast recognition of human motion postures based on multimodal biological information fusion is realized through the following steps:(1)
Two groups of normalized features can be obtained by using the algorithm.(2)
The left and right transformation matricesZX, ZY, RX, and RY are given according to the MDCCA theory. For the training samples X and Y, the intraclass divergence matrix SWX and SWY is given in turn, and the objective function is set.(3)
Equations (20) and (21) are iteratively calculated through the criterion function until the result converges, and the value determined ZX, ZY, RX, and RY in turn.(4)
Map samplesX and Y to the projection matrices ZX, ZY, RX, and RY to obtain the typical correlation matrix ZX′XRX and ZY′YRY after projection.(5)
Taking the matrixZX′XRX and ZY′YRY and the data X as the input of the minimum distance classifier, the minimum distance classifier is used to realize the rapid recognition of human motion postures. In the minimum distance classifier, each category wj is represented by the mean vector ϑj, that is, the vector is described by the mean vector of the feature vector family of each sample, that is: ϑj=1/τj∑x∈wjx, in which τj refers to the number of training vectors, through which the summation is realized. At the same time, the category of unknown sample vector x is determined through the vector two norm, that is, the distance between sample vectors is analyzed, and the recognition process of human motion postures is realized through the measurement of distance.


## 2.3. Dataset and Experimental Index

Two data sets were selected for experimental analysis, Human3.6 M dataset and CMU Panoptic data set. Human3.6 M data set includes 3.6 million 3D human postures and corresponding images, a total of 11 experimenters, and a total of 17 action scenes. To use this data set for posture recognition can improve the simulation accuracy. The CMU Panoptic data set is produced by CMU University and consists of 480 VGA cameras, 30+ HD cameras, and 10 Kinnect sensors, which are installed in the lab or in the subject’s body to collect the subject’s motion data. The overall number of experimental samples collected during the experiment was 4000 image samples, of which 3000 images were used as training samples and the others were used as 1000 images as test samples.The proposed algorithm is compared with the multimodal fusion driver stress detection algorithm based on attention CNN LSTM in literature [6], the fall detection algorithm based on the spatiotemporal evolution of human posture in literature [7], the two-stage fall recognition algorithm based on human posture features in literature [8], the online adaptive prediction algorithm of human motion intention based on surface EMG signal in literature [9], and the stochastic biomechanical modeling and recognition algorithm of wearable devices for human motion in literature [10] and analyze the recognition performance of different algorithms.Feature extraction: the general case hand motion feature amplitude is between [-1.8,1.5] and leg motion feature value amplitude is between [-3,3], if the feature value extracted in proposed algorithm is closer to the interval, the higher the feature extraction accuracy.Human motion posture skewness: the equation for calculating this index is as follows:(24)K=k1−k2.Posture recognition error: this value is the difference between the actual human motion posture and the posture recognized by the proposed algorithm, which is calculated as follows:(25)a=b−c.ROC curve: receiver operating characteristic curve is the line of points drawn under a specific stimulus condition with the probability of false alarm P(y/N) obtained by the subject under different judgment criteria as the horizontal coordinate and the probability of hit P(y/SN) as the vertical coordinate.Recognition rate: this index refers to the ratio of the amount of data of the correctly recognized human motion posture to the total amount of experimental data, and the equation for calculating this index is as follows:(26)Z=zizj×100%.


## 3. Results and Discussion

The hand and leg features during human movement in the two data sets are extracted, respectively, and the extracted feature amplitude is counted, asshown in Figure2.Figure 2
Feature extraction situation analysis.According to Figure2, there is a significant difference in the amplitude of leg and hand motion features at different times, in which the leg motion features show large up and down fluctuations, while the hand features are in slow fluctuations. Therefore, there is a large gap between them, and there are disadvantages in using a single feature for human motion posture recognition, and it is necessary to obtain more accurate human motion biometric features through reasonable bioinformation fusion, the proposed algorithm uses the use of wavelet packet decomposition and sample entropy to extract the kurtosis of muscle power signal, time domain feature skewness, frequency domain feature myoelectric integral value and other hand features of human motion posture, and extract the time domain features of leg motion amplitude such as mean, standard deviation, and quartile distance. After normalizing the two features to obtain the set of human hand and leg motion features, a solid foundation can be laid for the subsequent rapid recognition of human motion posture.The change in skewness of human motion posture recognition using hand, leg, and post-fusion motion features at different times is analyzed, and the results are shown in Figure3.Figure 3
Results of the change of bias of human motion posture by different features recognition.According to the data in Figure3, the bias of human motion posture recognition using leg motion and hand motion features is relatively large at different times, in which the bias of human motion posture recognition using leg features is larger than that of hand motion in walking, sitting, standing, squatting, falling, and lying down motion postures, while the bias of recognition using hand motion features is relatively small, indicating that the magnitude of hand motion is relatively low. The proposed algorithm uses feature sets to construct a fast recognition model of human motion posture based on multimodal bioinformation fusion and inputs the feature sets into the recognition model, which completes the fusion of human motion posture information by improving the typical correlation analysis method, improves the fusion accuracy, and can lay the foundation for subsequent accurate and fast recognition of human motion posture.Comparing the posture errors of different algorithms in performing human motion posture recognition, the analysis results are shown in Figure4.Figure 4
Comparison and analysis of posture recognition error.In Figure4, when the number of frames keeps increasing, the error of different algorithms in posture recognition decreases, in which, the posture error of the algorithm of literature [6] decreases the most, but the final error of this algorithm reaches about −0.82, and the recognition effect is still poor, the posture recognition error of the algorithm of literature [7] is between −0.52 and 1.15, the posture recognition error of the algorithm of literature [8] is between −0.09 and 1.27. The posture recognition error of the algorithm in literature [9] is between −0.31 and 1.25, the posture recognition error of the algorithm in literature [10] is between 0.86 and 1.36, which means that the recognition error of these algorithms is higher than that of the proposed algorithm at the initial stage, and the recognition error of proposed algorithm maintains a decreasing trend at the same time, but the posture recognition error fluctuates slightly between −0.21 and 0.02. It can be seen that the proposed algorithm can effectively reduce the error in posture recognition.ROC curves were used to verify the recognition effects of different algorithms, and the analysis results are shown in Figure5.Figure 5
Comparison of ROC curves for different algorithms.According to Figure5, there are some differences in the ROC curve changes of different algorithms, in which the ROC curve of the algorithm of literature [10] keeps the lowest, and the ROC curves of other algorithms are higher than the algorithm of literature [10]. It means that the algorithms of literature [6], literature [7], literature [8], literature [9], and literature [10] have lower recognition real case rate, and the ROC curve of the method calculated in this paper always keeps the highest, and when the false positive rate keeps increasing, the real case rate of the proposed algorithm also keeps at a higher level. The proposed algorithm has a high recognition effect. The method uses the human hand and leg motion feature set to construct a human motion posture fast recognition model using multimodal bioinformation fusion, and inputs the feature set into the recognition model, which completes the fusion of human motion pose information by improving the typical correlation analysis method, and uses the fusion result as the input of the minimum distance classifier, so as to achieve human motion posture fast recognition. Therefore, the proposed algorithm has a better recognition effect.The results of the analysis of the variation of the recognition rate of different algorithms in the case of different postures of human motion are shown in Table1.Table 1
Analysis of recognition rate of different algorithms (%).
Motion postureProposedLiterature [6]Literature [6]Literature [8]Literature [9]Literature [10]Walking97.491.487.489.578.583.4Sitting96.592.186.589.376.484.6Falling97.590.385.784.375.685.3Squatting94.891.489.681.474.388.3Standing95.690.788.689.573.381.3Lying down96.589.589.688.571.583.4It is seen form Table1 that the recognition rate of the proposed algorithm is always the highest in posture recognition, the recognition rate of the proposed algorithm is always above 95%, the human motion posture recognition rate of the algorithm in literature [6] fluctuates in the range of 89.5% to 92.1%, the human motion pose recognition rate of the algorithm in literature [7] fluctuates in the range of 85.7% to 89.6%, the human motion posture recognition rate of the algorithm in literature [8] fluctuates in the range of 81.4% to 89.5%, the human motion posture recognition rate of the algorithm in literature [9] fluctuated in the range of 71.5% to 78.5%, and the human motion posture recognition rate of the algorithm in literature [10] fluctuated in the range of 81.3% to 88.3%. The recognition rate of the other algorithms is also always lower than that of the proposed algorithm. Therefore, the proposed algorithm has the best results for human motion posture recognition.


## 4. Conclusions

Due to single-modal bioinformation is one sided, the application of such information to human motion gesture recognition process leads to increased recognition bias, so we design a fast human motion posture recognition algorithm using multimodal bioinformation fusion, extracting biofeatures of multiple human motion modalities and fusing these features for posture recognition. The posture recognition error is −0.21∼0.02, and the recognition rate is always above 95%. The feasibility of the proposed algorithm is verified using experiments, and it is confirmed that the proposed algorithm achieves better application results. However, the recognition process of proposed algorithm is too complicated, which may lead to the degradation of recognition efficiency. Therefore, in future work, the optimization can be continued for the current algorithm, so that the algorithm can recognize more human motion posture changes faster.



---

*Source: 9538295-2022-04-15.xml*