# Multimodal Emotion Recognition Model Based on a Deep Neural Network with Multiobjective Optimization

Authors: Mingyong Li; Xue Qiu; Shuang Peng; Lirong Tang; Qiqi Li; Wenhui Yang; Yan Ma
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

With the rapid development of deep learning and wireless communication technology, emotion recognition has received more and more attention from researchers. Computers can only be truly intelligent when they have human emotions, and emotion recognition is its primary consideration. This paper proposes a multimodal emotion recognition model based on a multiobjective optimization algorithm. The model combines voice information and facial information and can optimize the accuracy and uniformity of recognition at the same time. The speech modal is based on an improved deep convolutional neural network (DCNN); the video image modal is based on an improved deep separation convolution network (DSCNN). After single mode recognition, a multiobjective optimization algorithm is used to fuse the two modalities at the decision level. The experimental results show that the proposed model has a large improvement in each evaluation index, and the accuracy of emotion recognition is 2.88% higher than that of the ISMS_ALA model. The results show that the multiobjective optimization algorithm can effectively improve the performance of the multimodal emotion recognition model.


---

## Body

## 1. Introduction

The concept of “emotional computing” was first proposed by professor Picard of the Massachusetts Institute of Technology in the bookAffective Computing published in 1997. She defined “affective computing” as the calculation of factors related to human emotion, triggered by human emotion or able to affect emotion [1]. The research of affective computing is aimed at achieving harmonious and efficient human-computer interaction, so that computers have higher and more comprehensive intelligence [2, 3].The external expression of human emotion mainly includes voice, facial expression, posture, and so on. Human speech contains not only linguistic information but also nonlinguistic information such as people’s emotional state. For example, the same sentence often feels different to the listener because of the different emotional states of the speaker. Human speech can express emotion because it contains parameters that can reflect the characteristics of emotion. Facial expression is also an important external form of emotion, which contains certain emotional information. The research of facial expression recognition can effectively promote the development of emotion recognition research and the research of automatic understanding of computer images [4–6].Since the performance of speech emotion recognition is easily disturbed by the noise of the surrounding environment, facial expressions are also easily affected by problems such as dark lighting, different angles, and blocked areas. Therefore, single-modal emotion recognition has some limitations. In order to improve the overall recognition performance and learn from each other in different emotional features, researchers propose a multimodal emotion recognition method based on the fusion of speech and facial expression, which has important research significance in the practical application of emotion recognition. According to the processing of different modal signals in different stages, it can be divided into signal-level fusion, feature-level fusion, decision-level fusion, and hybrid fusion. In this paper, the decision-level fusion method is used to independently inspect and classify the features of each modal and merge the results into a decision vector. The schematic diagram of multimodal emotion recognition is shown in Figure1.Figure 1
Speech emotion recognition based on a deep convolution neural network.In many optimization fields, such as production scheduling, artificial intelligence, combinatorial optimization, large-scale data processing, and data mining, we often encounter many complex optimization problems closer to real life [7–11]. In the real world, the optimization problem is usually multiattribute, which is usually the simultaneous optimization of multiple objectives. In order to achieve the optimization of the overall goal, it is usually necessary to consider the conflicting subgoals comprehensively. Therefore, a multiobjective optimization (MOO) algorithm is proposed. This article uses two evaluation indicators, accuracy and emotion recognition uniformity, to evaluate the performance of emotion recognition models. In order to improve the two evaluation indexes at the same time, the multiobjective optimization algorithm is used to optimize the emotion recognition model.
(1)
The first time, the multiobjective optimization algorithm is combined with multimodal emotion recognition, and the performance of multimodal emotion recognition is effectively improved by optimizing the accuracy and uniformity of the model at the same time in the decision level(2)
In this paper, a deep convolutional neural network (DCNN) and a deep separable convolutional neural network (DSCNN) are proposed for speech recognition and face recognition, respectively, and good experimental results are obtained(3)
The proposed multimodal emotion recognition model based on multiobjective optimization has a better recognition effect, and the accuracy of emotion recognition is 2.88% higher than that of the ISMS_ALA modelThe rest of this paper is organized as follows. Section2 reviews the related research work of the multiobjective optimization algorithm and multimodal emotion recognition. In Section 3, we introduce the framework and model of two basic techniques used in multimodal emotion recognition. In Section 4, we test the proposed model. Finally, the conclusion of this paper is given in Section 5.


## 2. Related Work

At present, the research on multimodal emotion recognition is a hot topic in the interdisciplinary research of cognitive science, physiology, psychology, linguistics, computer science, and so on. Multimodal emotion recognition has attracted more and more attention from scientific research institutions and researchers domestically and internationally.In 1997, Duc et al. [12] proposed “multimodality” for the first time, using facial expression and speech fusion to recognize human identity and behavior. Professor Chen et al. of the Beckman College of the University of Illinois jointly [13] proposed the research of multimodal emotion recognition, which mainly involves the emotion recognition of speech and facial expression information. The experimental results show that the recognition rate of a single mode is lower than that of a multimodal one. For the feature contribution of emotion recognition, there is a big difference between speech and expression. Normally, the feature of expression makes a greater contribution to emotion recognition. Busso and Narayanan [14] of the Viterbi School of Engineering at the University of Southern California are working together on emotion recognition. Wang and Guan [15] proposed a vision-based emotion recognition method, which extracts visual features from Gabor wavelet key frames and then uses a feature-level data fusion scheme to combine audio features with visual features; Ding et al. [16] combined convolutional neural network (CNN) and Directional Gradient Histogram (HOG) methods to extract more expression features and achieved 90% recognition accuracy in happy emotion categories; Lan and Zhang [17] proposed a joint strategy (FRN+BN) to recognize facial expressions and improve the recognition accuracy of 5.6% on the CK+ dataset.When the accuracy of emotion recognition based on single-modal speech or facial expressions is not optimistic, it is jointly proposed to integrate speech and facial expression information for emotion recognition. With the deepening of fusion algorithm research, multimodal emotion recognition has achieved rapid development. Multimodal fusion can improve the recognition rate and has better robustness [18]. At present, the common multimodal emotion detection methods mainly include physiological signal+emotional behavior combination and the combination between different emotional modalities. Multimodal fusion methods include feature-level fusion (early stage), decision-level fusion (late stage), and hybrid fusion. The typical early fusion model is EF-LSTM [19], which stitches the feature representations of the three modalities of text, speech, and image to obtain a multimodal representation, which is then input into LSTM for encoding. Late fusion [20] occurs after decoding; it is a fusion at the decision level, which can extract interactive information within modalities but cannot extract interactive information between modalities. Hybrid fusion combines the first two fusion methods.Because the two modalities of facial expression and voice can be directly extracted in the video, they have the advantages of convenient data collection, obvious features, and high precision. They are the most widely used emotion recognition methods in practical applications. Lu and Zhang [21] proposed a neural network-based audio and video emotion recognition model. The model uses data from three aspects: frontal facial expressions, side facial expressions, and audio. It belongs to a model layer fusion method and has achieved good classification results; Sahoo and Routray [22] proposed a multimodal emotion recognition method using facial image and voice data, which uses a rule-based decision-level fusion method. The M-BERT model proposed by Rahman et al. [23] applies the pretraining model to multimodal emotion recognition tasks. M-BERT adds a modal fusion layer between the input layer and the coding layer to achieve the fusion of three modalities.In this study, the Mel Frequency Bank (MFB) method is used to extract the emotional features of speech signals, and the hidden Markov model (HMM) method is used to train these features. At the same time, these speech emotional features are optimized appropriately. For expression images, the method of dividing regions is used in the research, and different weights are assigned to each region to extract features. Then, the speech and facial expression features are fused, and the speech features of each expression in each region of the face are used to classify. The experimental results show that after using the feature fusion of speech and expression, the effect is obviously better than that of only speech or expression.Many scientific and engineering problems in industry, agriculture, national defense, transportation, information, economy, and management can be transformed into optimization problems. A multiobjective optimization problem (MOP) is a kind of challenging and complex optimization problem. Because the optimization goals conflict with each other, it is extremely difficult to obtain a single global optimal solution, so it is a set of compromise Pareto optimal solutions [24, 25]. In recent decades, many similar optimization algorithms have appeared, such as PEAS [26], SPEA2 [27], NSGAII [28], MOEA [29], MOEA/D [30], IBEA [31], and HypE [32]. These algorithms have achieved better optimization results. However, affected by the background of various algorithms, there is no algorithm that can obtain the optimal solution set when solving all multiobjective optimization problems.In this paper, we propose a new multimodal emotion recognition technology, which uses a multioptimization algorithm to perform fusion operation at the decision level. The final decision is obtained by the linear weighted sum of all single-modal classification results. In this way, different modalities can be identified cooperatively, so as to give full play to their respective advantages.


## 3. Proposed Method

### 3.1. Speech Emotion Recognition Based on DCNN

Audio also contains emotional information about people. Generally speaking, multimodal emotion recognition is more reliable than single-modal recognition. Raditional speech emotion recognition algorithms use LLDs or HSFs for feature extraction and then use statistical classification models such as HMM for emotion classification, but the performance of these algorithms is not particularly satisfactory. With the continuous development of deep learning, people use deep neural networks for speech emotion recognition, and many speech emotion recognition algorithms based on deep neural networks have been proposed.The process of speech emotion recognition is divided into three parts: signal processing, feature extraction, and classification. Signal processing applies acoustic filters to the original audio signal and divides it into meaningful units. In this study, we first use the OpenSmile toolbox to extract frame-level acoustic features from speech signals. The extracted features are eighty-eight feature sets of eGeMAPS. After feature extraction, a 256-dimensional feature vector can be obtained for each frame. In order to further prepare a fixed-length feature map suitable for model input, it is necessary to perform a length normalization operation on the obtained feature sequence of variable length. Because only a few of the speech in IEMOCAP dataset are longer than one thousand frames, the algorithm abandons the features of those speech that are longer than one thousand frames. For the voice whose length is less than one thousand frames, zero filling operation is carried out to make its length reach one thousand frames. After feature extraction, the feature vector sequence with a length of one thousand and a dimension of 256 can be obtained. In the speech emotion classification stage, the speech emotion recognition algorithm based on the deep convolutional neural network proposed in this paper is used to predict. The speech emotion recognition model based on DCNN is shown in Figure2.Figure 2
Speech emotion recognition based on a deep convolution neural network.Figure2 shows the voice emotion recognition model based on CNN. The input of the model is a 256∗1000 feature map. The model uses four convolution layers to extract features, and the number of convolution kernels in the convolution layer is 4, 8, 16, and 32 in turn. The convolution kernel of the first convolution layer has a step size of one, a width of one, and a length of five, using the same convolution method. Therefore, the size of the feature map obtained through the first layer of convolution is 238∗1000∗4. Then, global k-Max Pooling (GKMP) is used, and the k value of the first pooling layer is 512. Therefore, a feature map with a size of 238∗512∗4 is obtained. The step size and width of the convolution kernel of the second convolution layer are 1, and the length is 3, and the same convolution in the same mode is used. The k value of this layer is 256. The feature map is obtained by the second convolution. The size is 238∗256∗8. The next two layers use the same step size and convolution kernel size as the second layer of convolution. The k value is 128 and 1, respectively. Then, after 4 layers of full connection, the number of hidden layer neurons is 512, 256, 128, and 4, respectively. Finally, a feature vector of length of 4 can be obtained, and then, through the softmax layer, the prediction of the model can be obtained.


### 3.2. Facial Expression Recognition Based on DSCNN

Facial expression is the main form of emotional expression, and the emotional information conveyed by facial expression is common in different countries, nationalities, and cultures. This paper constructs a deep learning algorithm based on deep separation convolution for facial expression.Szegedy et al. [33] proposed the Inception structure. The main idea is to first use a 1∗1 convolution kernel to map each channel of the feature map to a new space. In this process, the correlation between channels can be learned, and then, convolution can be carried out through the conventional 3∗3 or 5∗5 convolution kernel. At the same time, the spatial correlation and the correlation between channels can be learned. Chollet [34] proposed “extremely” this idea, using a two-dimensional depthwise separable convolution (Separableconv2D) method and the channel correlation and spatial correlation to achieve a complete separation effect. This operation increases the width of the network and plays a great role in improving the accuracy of classification.Deep separation convolution [35, 36] consists of two processes: layer-by-layer convolution and pixel-by-pixel convolution. This paper constructs an algorithm model based on deep separation convolution, as shown in Figure 3.Figure 3
Facial expression recognition based on deep separation convolution.


### 3.3. Multiobjective Optimization

For the mathematical description of the multiobjective problem, we take the minimum value problem as an example:(1)Minimizefx=f1x,f2x,⋯,flxgix≥0,i=1,2,⋯,mSubjectto:hix=0,i=1,2,⋯,pLi≤xi≤Ui,i=1,2,⋯,n,where n, l, m, and p are the number of variables, objective functions, inequality constraints, and equality constraints, respectively. gi and hi represent the i-th inequality and equality constraint, respectively, and Oi,Ui is the boundary of the i-th variable.Obviously, solutions to the multiobjective problem cannot be compared by using the above relational operators. Therefore, for the multiobjective problem, the relational operator must be extended. Four key definitions in MOO are as follows.(1)
Pareto dominanceAssume two vectors such asx=x1,x2,⋯,xk and y=y1,y2,⋯,yk. Vector x is said to dominate vector y (denoted as x≺y) if and only if
(2)∀i∈1,2,⋯,k:fxi≤fyi.(2)
Pareto optimalityA solutionx∈X is called Pareto-optimal if and only if
(3)∃y∈X∣fy<fx.(3)
Pareto-optimal setThe set of all Pareto-optimal solutions is called the Pareto set as follows:(4)Ps=x,y∈X∣∃fy≻fx.(4)
Pareto-optimal frontA set containing the value of objective functions for the Pareto solution set is(5)Pf=fx∣x∈Ps.For solving a MOP, we have to find the Pareto-optimal set, which is the set of solutions representing the best trade-offs between different objectives.


### 3.4. Decision-Level Fusion Based on a Multiobjective Optimization Algorithm

The same dataset often produces different prediction results in speech and face recognition. In order to improve the recognition accuracy and balance of different expressions after fusion, the multiobjective optimization algorithm is used to fuse the two modalities, so that the recognition results of their different modalities can make up for each other.In this article, we use two coefficients to linearly combine two basic emotion recognition technologies (based on deep separation convolutional facial emotion recognition and deep convolutional neural network-based voice emotion recognition). And use a multiobjective optimization algorithm to simultaneously optimize accuracy (precision) and the evaluation model of emotion recognition uniformity, as shown in(6)R=w1×R1+w2×R2,where R1 and R2 are the final prediction results of speech emotion recognition based on a deep convolution neural network and deep separation convolution facial emotion recognition, respectively. In addition, w1 and w2 are coefficients that need to be optimized. According to the actual meaning of the model, the constraints of the two coefficients are as follows:
(7)w1+w2=1.The function of the multiobjective optimization algorithm is to optimize the two coefficients, so that the two recognition techniques can be effectively combined. The accuracy of the optimization coefficient will directly affect the final recognition result of the model and then affect the accuracy of facial expression recognition.After obtaining the facial and voice results, the decision-level fusion method is adopted to fuse the two to obtain the final recognition result. For the first time in this article, a multiobjective optimization algorithm is used to optimize the emotion recognition model to achieve the best results. Its framework is shown in Figure4.Figure 4
The framework of the multimodal emotion recognition model with many objectives.R1 and R2 show the emotion recognition results using the two recognition techniques, respectively, in Figure 4. At the same time, w1 and w2 represent the coefficients of combining the two recognition techniques.


## 3.1. Speech Emotion Recognition Based on DCNN

Audio also contains emotional information about people. Generally speaking, multimodal emotion recognition is more reliable than single-modal recognition. Raditional speech emotion recognition algorithms use LLDs or HSFs for feature extraction and then use statistical classification models such as HMM for emotion classification, but the performance of these algorithms is not particularly satisfactory. With the continuous development of deep learning, people use deep neural networks for speech emotion recognition, and many speech emotion recognition algorithms based on deep neural networks have been proposed.The process of speech emotion recognition is divided into three parts: signal processing, feature extraction, and classification. Signal processing applies acoustic filters to the original audio signal and divides it into meaningful units. In this study, we first use the OpenSmile toolbox to extract frame-level acoustic features from speech signals. The extracted features are eighty-eight feature sets of eGeMAPS. After feature extraction, a 256-dimensional feature vector can be obtained for each frame. In order to further prepare a fixed-length feature map suitable for model input, it is necessary to perform a length normalization operation on the obtained feature sequence of variable length. Because only a few of the speech in IEMOCAP dataset are longer than one thousand frames, the algorithm abandons the features of those speech that are longer than one thousand frames. For the voice whose length is less than one thousand frames, zero filling operation is carried out to make its length reach one thousand frames. After feature extraction, the feature vector sequence with a length of one thousand and a dimension of 256 can be obtained. In the speech emotion classification stage, the speech emotion recognition algorithm based on the deep convolutional neural network proposed in this paper is used to predict. The speech emotion recognition model based on DCNN is shown in Figure2.Figure 2
Speech emotion recognition based on a deep convolution neural network.Figure2 shows the voice emotion recognition model based on CNN. The input of the model is a 256∗1000 feature map. The model uses four convolution layers to extract features, and the number of convolution kernels in the convolution layer is 4, 8, 16, and 32 in turn. The convolution kernel of the first convolution layer has a step size of one, a width of one, and a length of five, using the same convolution method. Therefore, the size of the feature map obtained through the first layer of convolution is 238∗1000∗4. Then, global k-Max Pooling (GKMP) is used, and the k value of the first pooling layer is 512. Therefore, a feature map with a size of 238∗512∗4 is obtained. The step size and width of the convolution kernel of the second convolution layer are 1, and the length is 3, and the same convolution in the same mode is used. The k value of this layer is 256. The feature map is obtained by the second convolution. The size is 238∗256∗8. The next two layers use the same step size and convolution kernel size as the second layer of convolution. The k value is 128 and 1, respectively. Then, after 4 layers of full connection, the number of hidden layer neurons is 512, 256, 128, and 4, respectively. Finally, a feature vector of length of 4 can be obtained, and then, through the softmax layer, the prediction of the model can be obtained.


## 3.2. Facial Expression Recognition Based on DSCNN

Facial expression is the main form of emotional expression, and the emotional information conveyed by facial expression is common in different countries, nationalities, and cultures. This paper constructs a deep learning algorithm based on deep separation convolution for facial expression.Szegedy et al. [33] proposed the Inception structure. The main idea is to first use a 1∗1 convolution kernel to map each channel of the feature map to a new space. In this process, the correlation between channels can be learned, and then, convolution can be carried out through the conventional 3∗3 or 5∗5 convolution kernel. At the same time, the spatial correlation and the correlation between channels can be learned. Chollet [34] proposed “extremely” this idea, using a two-dimensional depthwise separable convolution (Separableconv2D) method and the channel correlation and spatial correlation to achieve a complete separation effect. This operation increases the width of the network and plays a great role in improving the accuracy of classification.Deep separation convolution [35, 36] consists of two processes: layer-by-layer convolution and pixel-by-pixel convolution. This paper constructs an algorithm model based on deep separation convolution, as shown in Figure 3.Figure 3
Facial expression recognition based on deep separation convolution.


## 3.3. Multiobjective Optimization

For the mathematical description of the multiobjective problem, we take the minimum value problem as an example:(1)Minimizefx=f1x,f2x,⋯,flxgix≥0,i=1,2,⋯,mSubjectto:hix=0,i=1,2,⋯,pLi≤xi≤Ui,i=1,2,⋯,n,where n, l, m, and p are the number of variables, objective functions, inequality constraints, and equality constraints, respectively. gi and hi represent the i-th inequality and equality constraint, respectively, and Oi,Ui is the boundary of the i-th variable.Obviously, solutions to the multiobjective problem cannot be compared by using the above relational operators. Therefore, for the multiobjective problem, the relational operator must be extended. Four key definitions in MOO are as follows.(1)
Pareto dominanceAssume two vectors such asx=x1,x2,⋯,xk and y=y1,y2,⋯,yk. Vector x is said to dominate vector y (denoted as x≺y) if and only if
(2)∀i∈1,2,⋯,k:fxi≤fyi.(2)
Pareto optimalityA solutionx∈X is called Pareto-optimal if and only if
(3)∃y∈X∣fy<fx.(3)
Pareto-optimal setThe set of all Pareto-optimal solutions is called the Pareto set as follows:(4)Ps=x,y∈X∣∃fy≻fx.(4)
Pareto-optimal frontA set containing the value of objective functions for the Pareto solution set is(5)Pf=fx∣x∈Ps.For solving a MOP, we have to find the Pareto-optimal set, which is the set of solutions representing the best trade-offs between different objectives.


## 3.4. Decision-Level Fusion Based on a Multiobjective Optimization Algorithm

The same dataset often produces different prediction results in speech and face recognition. In order to improve the recognition accuracy and balance of different expressions after fusion, the multiobjective optimization algorithm is used to fuse the two modalities, so that the recognition results of their different modalities can make up for each other.In this article, we use two coefficients to linearly combine two basic emotion recognition technologies (based on deep separation convolutional facial emotion recognition and deep convolutional neural network-based voice emotion recognition). And use a multiobjective optimization algorithm to simultaneously optimize accuracy (precision) and the evaluation model of emotion recognition uniformity, as shown in(6)R=w1×R1+w2×R2,where R1 and R2 are the final prediction results of speech emotion recognition based on a deep convolution neural network and deep separation convolution facial emotion recognition, respectively. In addition, w1 and w2 are coefficients that need to be optimized. According to the actual meaning of the model, the constraints of the two coefficients are as follows:
(7)w1+w2=1.The function of the multiobjective optimization algorithm is to optimize the two coefficients, so that the two recognition techniques can be effectively combined. The accuracy of the optimization coefficient will directly affect the final recognition result of the model and then affect the accuracy of facial expression recognition.After obtaining the facial and voice results, the decision-level fusion method is adopted to fuse the two to obtain the final recognition result. For the first time in this article, a multiobjective optimization algorithm is used to optimize the emotion recognition model to achieve the best results. Its framework is shown in Figure4.Figure 4
The framework of the multimodal emotion recognition model with many objectives.R1 and R2 show the emotion recognition results using the two recognition techniques, respectively, in Figure 4. At the same time, w1 and w2 represent the coefficients of combining the two recognition techniques.


## 4. Experiments

### 4.1. Experimental Setup

In order to prove the effectiveness of the multiobjective optimization algorithm for improving the effect of multimodal emotion recognition, this article compares the effect of emotion recognition using the multiobjective optimization algorithm with the model effect without using the multiobjective optimization algorithm. This paper uses the IEMOCAP multimodal emotion database. In addition to the comparison test with the single-modal model proposed in this paper, the comparison experiment is also compared with the multimodal emotion recognition ISMS_ALA [37] model that has not been optimized by the multiobjective optimization algorithm.


### 4.2. Evaluation Method

The multiobjective optimization algorithm is used to optimize the accuracy of model recognition and the uniformity of emotion recognition at the same time. The confusion matrix of seven categories of emotion recognition is shown in Table1.Table 1
Confusion matrix of the evaluation indicators.
Predicted result1234Actual result1TP11FP21FP31FP412FP12TP22FP32FP423FP13FP23TP33FP434FP14FP24FP34TP44In the first evaluation, the standard accuracy is defined as the ratio of the number of correctly predicted classified emotions to the total number of all data. The accuracy formula is as follows:(8)Accuracy=TT+F,where T represents the number of samples that predict the correct sentiment and F represents the number of samples that predict the wrong sentiment.Because of this research, emotion recognition belongs to seven categories. Different emotions have corresponding prediction accuracy results. In the traditional research of multimodal emotion recognition, the accuracy of different emotion recognition often differs greatly. Therefore, in order to balance and improve the recognition accuracy of different emotions, the evaluation index of emotion recognition uniformity is proposed.In order to define the second evaluation index, the uniformity of emotion recognition, we first introduce the concept of recall rate. The recall rate refers to the number of correct predictions of each different emotion in the prediction results and the total number of corresponding emotions in all data.(9)Recalli=Tii∑j=1,j≠kNFji+Tii.Recalli represents the recall rate predicted by the current i class emotion algorithm. In order to balance and improve the recognition accuracy of different emotions, the evaluation index of emotion recognition uniformity is proposed.
(10)U=∑i=1NRecalli−RecallaverN.Recallaver represents the average recall rate of seven emotion categories.


### 4.3. Experimental Result

As shown in Tables2 and 3, the single-modal emotion recognition accuracy of the facial emotion recognition model based on the deep separation convolution and speech emotion recognition model based on the deep convolution neural network are, respectively, shown.Table 2
Speech emotion recognition results based on a deep convolution neural network.
Predicted resultAngerHappyNeutralSadActual resultAnger70.6%5.6%11.3%12.5%Happy4.5%81.3%6.7%7.5%Neutral8.7%9.3%64.1%17.9%Sad12.7%5.8%12.3%69.2%Table 3
Facial expression recognition results based on deep separation convolution.
Predicted resultAngerHappyNeutralSadActual resultAnger72.2%7.8%5.1%14.9%Happy13.4%76.7%5.8%4.1%Neutral9.2%8.2%61.3%21.3%Sad7.7%8.9%17.2%66.2%According to the analysis of the above table, the accuracy of facial emotion recognition is higher than that of speech emotion recognition. The calculation shows that the average recognition accuracy of facial emotion recognition using deep separation convolution is 71.2%. The average accuracy of speech emotion recognition based on deep convolutional neural networks is 69.1%. At the same time, through observation, we also found that there is a big gap in the uniformity of different emotion recognition of facial and voice emotion recognition. Among them, the recognition accuracy of facial and voice neutral emotions are only 64.1% and 61.3%, respectively. However, the recognition accuracy of happy emotion reached 81.3% and 76.7%, respectively. Therefore, we use the multiobjective optimization algorithm to optimize the accuracy and uniformity of emotion recognition at the same time.In the model training stage, NSGA-III [38], MOEA/DD [39], HypE [32], and PEAS [26] were used to optimize coefficients w1 and w2, respectively. Each time the algorithm runs once, pop_size individuals will be generated. Different individual experiences are fused into different recognition results. Common parameter settings in MOEA are shown in Table 4.Table 4
The setting of common parameters.
ParameterMeaningValueIterNumber of iterations500popsizeThe population size100DDimensions of decision variables2MThe number of optimization objectives2PmThe mutation probability0.2PcThe crossover probability1As shown in Figures5(a) and 5(b), the results of the four algorithms optimized multimodal emotion recognition model in the two evaluation indicators which are drawn into a box plot. In these figures, each box represents all individuals except discrete individuals in the group (discrete individuals are represented by small circles). Each line represents the performance of a basic monomodal emotion recognition technology on the evaluation index. The green dotted line represents the performance of the ISMS_ALA model on the IEMOCAP test set.Figure 5
Four multiobjective optimization algorithms for multimodal emotion recognition.
(a)
The accuracy of multimodal emotion recognition(b)
The quasi uniformity of multimodal emotion recognitionFrom the analysis in Figures5(a) and 5(b), we can conclude that compared with the other three algorithms, the model optimized by the PEAS algorithm has the shortest box line graph length on three different optimization objectives, and there is no discrete individual. This shows that PEAS has better convergence than other algorithms when solving this model. At the same time, it can be seen that the model optimized based on the NSGA-III, HypE, and MOEA/DD algorithm has a small number of scattered individuals and the length of the box plot is too long, indicating that the convergence of the population is insufficient. At the same time, we can see that the optimal model optimized by the multiobjective optimization algorithm is better than the technology only using single modal for emotion recognition and ISMS_ALA model without the multiobjective optimization algorithm. Next, compare the performance of the four algorithms in solving the model, and the results are shown in Table 5.Table 5
Algorithm performance comparison.
AlgorithmAccuracyHMERSingle modal emotion recognition technologyDCNN0.7120.05DSCNN0.6910.0535ISMS_ALA0.7250.0245Best valueNSGA-III0.74050.0085HypE0.73210.0107PEAS0.75380.0034MOEA/DD0.74160.0098Worst valueNSGA-III0.67950.0489HypE0.66150.0662PEAS0.69720.0261MOEA/DD0.67260.0427Mean valueNSGA-III0.71690.0261HypE0.71430.0328PEAS0.72680.0178MOEA/DD0.71780.0247From the analysis of Table5, it can be concluded that the results of the PEAS optimization model are the best in accuracy and uniformity. The highest accuracy is 75.38%. The accuracy of the model is improved by 2.88% compared with that of the ISMS_ALA model, and the uniformity evaluation index is reduced by 0.0211. In terms of the worst value and average value of the evaluation index, PEAS is also better than the other three optimization algorithms. At the same time, compared to the single-modal emotion recognition model that uses multiobjective optimization algorithms, the accuracy and uniformity indicators are greatly improved. Experiments show that the multiobjective optimization algorithm effectively improves the performance of the multimodal emotion recognition model.


## 4.1. Experimental Setup

In order to prove the effectiveness of the multiobjective optimization algorithm for improving the effect of multimodal emotion recognition, this article compares the effect of emotion recognition using the multiobjective optimization algorithm with the model effect without using the multiobjective optimization algorithm. This paper uses the IEMOCAP multimodal emotion database. In addition to the comparison test with the single-modal model proposed in this paper, the comparison experiment is also compared with the multimodal emotion recognition ISMS_ALA [37] model that has not been optimized by the multiobjective optimization algorithm.


## 4.2. Evaluation Method

The multiobjective optimization algorithm is used to optimize the accuracy of model recognition and the uniformity of emotion recognition at the same time. The confusion matrix of seven categories of emotion recognition is shown in Table1.Table 1
Confusion matrix of the evaluation indicators.
Predicted result1234Actual result1TP11FP21FP31FP412FP12TP22FP32FP423FP13FP23TP33FP434FP14FP24FP34TP44In the first evaluation, the standard accuracy is defined as the ratio of the number of correctly predicted classified emotions to the total number of all data. The accuracy formula is as follows:(8)Accuracy=TT+F,where T represents the number of samples that predict the correct sentiment and F represents the number of samples that predict the wrong sentiment.Because of this research, emotion recognition belongs to seven categories. Different emotions have corresponding prediction accuracy results. In the traditional research of multimodal emotion recognition, the accuracy of different emotion recognition often differs greatly. Therefore, in order to balance and improve the recognition accuracy of different emotions, the evaluation index of emotion recognition uniformity is proposed.In order to define the second evaluation index, the uniformity of emotion recognition, we first introduce the concept of recall rate. The recall rate refers to the number of correct predictions of each different emotion in the prediction results and the total number of corresponding emotions in all data.(9)Recalli=Tii∑j=1,j≠kNFji+Tii.Recalli represents the recall rate predicted by the current i class emotion algorithm. In order to balance and improve the recognition accuracy of different emotions, the evaluation index of emotion recognition uniformity is proposed.
(10)U=∑i=1NRecalli−RecallaverN.Recallaver represents the average recall rate of seven emotion categories.


## 4.3. Experimental Result

As shown in Tables2 and 3, the single-modal emotion recognition accuracy of the facial emotion recognition model based on the deep separation convolution and speech emotion recognition model based on the deep convolution neural network are, respectively, shown.Table 2
Speech emotion recognition results based on a deep convolution neural network.
Predicted resultAngerHappyNeutralSadActual resultAnger70.6%5.6%11.3%12.5%Happy4.5%81.3%6.7%7.5%Neutral8.7%9.3%64.1%17.9%Sad12.7%5.8%12.3%69.2%Table 3
Facial expression recognition results based on deep separation convolution.
Predicted resultAngerHappyNeutralSadActual resultAnger72.2%7.8%5.1%14.9%Happy13.4%76.7%5.8%4.1%Neutral9.2%8.2%61.3%21.3%Sad7.7%8.9%17.2%66.2%According to the analysis of the above table, the accuracy of facial emotion recognition is higher than that of speech emotion recognition. The calculation shows that the average recognition accuracy of facial emotion recognition using deep separation convolution is 71.2%. The average accuracy of speech emotion recognition based on deep convolutional neural networks is 69.1%. At the same time, through observation, we also found that there is a big gap in the uniformity of different emotion recognition of facial and voice emotion recognition. Among them, the recognition accuracy of facial and voice neutral emotions are only 64.1% and 61.3%, respectively. However, the recognition accuracy of happy emotion reached 81.3% and 76.7%, respectively. Therefore, we use the multiobjective optimization algorithm to optimize the accuracy and uniformity of emotion recognition at the same time.In the model training stage, NSGA-III [38], MOEA/DD [39], HypE [32], and PEAS [26] were used to optimize coefficients w1 and w2, respectively. Each time the algorithm runs once, pop_size individuals will be generated. Different individual experiences are fused into different recognition results. Common parameter settings in MOEA are shown in Table 4.Table 4
The setting of common parameters.
ParameterMeaningValueIterNumber of iterations500popsizeThe population size100DDimensions of decision variables2MThe number of optimization objectives2PmThe mutation probability0.2PcThe crossover probability1As shown in Figures5(a) and 5(b), the results of the four algorithms optimized multimodal emotion recognition model in the two evaluation indicators which are drawn into a box plot. In these figures, each box represents all individuals except discrete individuals in the group (discrete individuals are represented by small circles). Each line represents the performance of a basic monomodal emotion recognition technology on the evaluation index. The green dotted line represents the performance of the ISMS_ALA model on the IEMOCAP test set.Figure 5
Four multiobjective optimization algorithms for multimodal emotion recognition.
(a)
The accuracy of multimodal emotion recognition(b)
The quasi uniformity of multimodal emotion recognitionFrom the analysis in Figures5(a) and 5(b), we can conclude that compared with the other three algorithms, the model optimized by the PEAS algorithm has the shortest box line graph length on three different optimization objectives, and there is no discrete individual. This shows that PEAS has better convergence than other algorithms when solving this model. At the same time, it can be seen that the model optimized based on the NSGA-III, HypE, and MOEA/DD algorithm has a small number of scattered individuals and the length of the box plot is too long, indicating that the convergence of the population is insufficient. At the same time, we can see that the optimal model optimized by the multiobjective optimization algorithm is better than the technology only using single modal for emotion recognition and ISMS_ALA model without the multiobjective optimization algorithm. Next, compare the performance of the four algorithms in solving the model, and the results are shown in Table 5.Table 5
Algorithm performance comparison.
AlgorithmAccuracyHMERSingle modal emotion recognition technologyDCNN0.7120.05DSCNN0.6910.0535ISMS_ALA0.7250.0245Best valueNSGA-III0.74050.0085HypE0.73210.0107PEAS0.75380.0034MOEA/DD0.74160.0098Worst valueNSGA-III0.67950.0489HypE0.66150.0662PEAS0.69720.0261MOEA/DD0.67260.0427Mean valueNSGA-III0.71690.0261HypE0.71430.0328PEAS0.72680.0178MOEA/DD0.71780.0247From the analysis of Table5, it can be concluded that the results of the PEAS optimization model are the best in accuracy and uniformity. The highest accuracy is 75.38%. The accuracy of the model is improved by 2.88% compared with that of the ISMS_ALA model, and the uniformity evaluation index is reduced by 0.0211. In terms of the worst value and average value of the evaluation index, PEAS is also better than the other three optimization algorithms. At the same time, compared to the single-modal emotion recognition model that uses multiobjective optimization algorithms, the accuracy and uniformity indicators are greatly improved. Experiments show that the multiobjective optimization algorithm effectively improves the performance of the multimodal emotion recognition model.


## 5. Conclusions

This paper presents a multimodal emotion recognition model based on the multiobjective optimization algorithm. The model can optimize the accuracy and uniformity of recognition results at the same time. Through the model optimization experiments of four multiobjective optimization algorithms, it is found that the model optimized by the algorithm has a great improvement compared with the single-modal emotion recognition model. At the same time, compared with the traditional multimodal emotion recognition ISMS_ALA model, the accuracy is improved by 2.88%, and the uniformity is also significantly improved. To sum up, the proposed multiobjective optimization algorithm effectively improves the performance of the multimodal emotion recognition model.



---

*Source: 6971100-2021-08-26.xml*