# An Intelligent Vehicle Alarm User Terminal System Based on Emotional Identification Technology

Authors: Liping Wu; Maomao Liu; Jingpeng Li; Yanjun Zhang
Journal: Scientific Programming
 (2022)

---

## Abstract

Negative emotions could increase the risks of traffic accidents. However, the driver’s emotional identification is rarely considered in the current design of intelligent vehicle alarm user terminals (IVAUTs). To solve the problem, this paper tries to design an IVAUT system based on emotional identification technology. Firstly, the transformer network was combined with a convolutional neural network (CNN) into a voice emotional identification system for intelligent vehicle alarm, and an emotional labeling approach was provided. Next, a bimodal fusion model was developed based on decompose-CNNs, which includes an appearance module, an optical flow module, and a bimodal fusion module. The proposed emotional identification approach was proved effective through experiments.


---

## Body

## 1. Introduction

Negative emotions could increase the risks of traffic accidents [1–8]. To assist with driving, it is necessary to effectively detect the driver’s negative emotions and thus enhance his/her thinking ability, perception ability, and judgement ability [9–11]. Currently, emotions are mostly identified based on facial expressions, voices, postures, behaviors, texts, and physiological signals [12–18]. Among them, the identification technologies of voice signals, facial expressions, and physical behaviors are research hotspots. With the popularity of intelligent vehicle interaction equipment, emotional identification technology has been gradually introduced to assist with driving by monitoring fatigue driving and driver emotions [19, 20].To prevent driving risks, Ooi et al. [21] proposed a new driver monitoring system. Specifically, a deep convolutional neural network (DCNN) was designed to recognize driver emotions, and an on-demand audio mechanism was developed to automatically collect audio resources with an online crawler, aiming to eliminate the driving risks induced by the driver’s negative emotions. Based on a survey on electrodermal activities, Bi and Shen [22] recognized stress and anger as the main driver emotions that lead to accidents and develop a simulated driving operation with preset neutral, stress, and anger scenarios, according to emotional stimuli. Xie et al. [23] presented an emotion-based fatigue driving recognition algorithm for the drivers, who have been driving for a long time or have a poor mental state, and relied on the algorithm to prevent continuous fatigue driving, thereby avoiding incidents. Drawing on the Ortony-Clore-Collins (OCC) model of emotion, the Markov model of the automatic state transition of emotions, and the hidden Markov model of the state transition of emotions under stimuli, Neerincx et al. [24] modeled the driver emotions under two different road conditions and fixed road conditions and examined the variation in driver emotions during car following, lane changing, and overtaking. Riaz et al. [25] categorized the causes of driving emotions into two classes, namely, personal factors and specific driving conditions, pointing out that the nature and intensity of perceived emotions depend on the various evaluation factors under traffic conditions. Izquierdo-Reyes et al. [26] put forward an effective driver assistance model, which drives cognition by emotions. As an accident prevention scheme, the model considers the distraction of different types of drivers simultaneously.After sorting out the domestic and foreign research, the studies on intelligent vehicle alarm user terminals (IVAUTs) focus on the design of terminals based on human-computer interaction and the development of information exchange interfaces based on design psychology. However, little attention has been paid to the emotional recognition of drivers in the IVAUT design. The proposed IVAUT system realizes the relevant functions based on techniques of voice emotion identification and facial emotion identification. The two emotion identification techniques were innovatively integrated to recognize and warn the negative emotions of drivers.The available techniques of voice emotion identification cannot effectively recognize the individual difference in voice, while the current methods of facial emotion identification overlook the correlation between appearance modal and optical flow modal. Hence, this paper designs an innovative approach for the two identification technologies. Section2 combines the transformer network with a convolutional neural network (CNN) into a voice emotional identification system for intelligent vehicle alarm and explains the emotional labeling approach. Section 3 develops a bimodal fusion model based on decompose-CNNs, which includes an appearance module, an optical flow module, and a bimodal fusion module. Section 4 verifies the effectiveness of our emotional identification approach through experiments.


## 2. Voice Emotion Identification System

With the technical advancement of various sensors and monitoring equipment, it is increasingly easier to acquire human voice signals and facial expression images with high accuracy. To ensure safe driving, it is particularly important to identify and warn drivers’ negative emotions with artificial intelligence and machine learning algorithms, as well as the techniques of voice emotion identification and facial emotion identification. The IVAUTs can precisely monitor the real-time emotions of drivers, accurately grasp their psychological changes, and take timely countermeasures to prevent traffic accidents induced by negative emotions.The voice features (e.g., pitch, tone, and loudness) of humans vary with emotional states. The existing techniques of voice emotion identification usually analyze the voice features corresponding to the known types of emotions, adjust the parameters and weights of the emotion identification model, making the model more effective in identifying emotions, and test the adjusted model.This paper combines the transformer network with CNN to acquire voice emotional features more effectively. The primary voice emotional features were taken as the parameters of the two networks and transmitted them to the deep network. The selected CNN consists of four convolutional modules that optimize loss classifiers, a batch normalization module for regularization, a max-pooling layer that reduces the dimensionality of the feature map, a drop-out layer that prevents overfitting in training, and a rectified linear unit (ReLU) function to activate the standard layers. In the transformer network, the multihead attention encoder is connected to the fully connected feedforward network, which is followed by a ReLU function layer. The outputs of the transformer network and the CNN are combined and mapped by the softmax function to eight emotions.


### 2.1. Network Design

The transformer encoder structure was adopted to increase the number of features. If the data are batch-processed by instance normalization, the features extracted by the transformer network will differ slightly from those extracted by the CNN in terms of the internal relationship.Moreover, the voice sample set for the IVAUTs covers the samples of 30 users. Since each user has unique voice features, the extracted features contain lots of personal features: the intraclass distance is even greater than the interclass distance. If the voice emotional features are classified by the softmax function, the intraclass distance would be enlarged, undermining the recognition effect of the model.To solve the above problem, the voice emotion identification model with loss function was introduced to reduce the intraclass distance. The loss function learns the center of deep voice features in each class and penalizes the central features of the expected classes. In this way, the interclass distance is increased, and the intraclass distance is reduced. As shown in Figure1, the established network consists of an input layer, multiple convolutional layers, a multihead attention mechanism, a fully connected layer, and an output layer.Figure 1
Structure of our voice emotion identification network.The voice emotion model has a special requirement on the overall distribution of sample data. To keep the data distribution consistent, the batch normalization module in the network normalizes each batch of voice samples. The mean and variance of each batch are greatly affected by the data size of that batch. If the data are too few, it is impossible to characterize the sample distribution with the calculated mean and variance. Voice emotion recognition usually deals with the entire input sentence. LetF be the number of hidden units on each layer; k be the k-th hidden layer; v be the value of a node before activation. To obtain more, richer emotional features of voice sentences, layer normalization was adopted in our transformer encoder. The input of all nodes is regularized by(1)Nk=1F∑i=1Fυik,(2)εk=1F∑i=1Fυik−Nk2.By formulas (1) and (2), the expectation N and standard error ε can be obtained for each layer. The hidden units in the same layer share the same normalized expectation N and standard error ε. However, the networks trained on different cases would have different normalized expectations N and standard error ε. Let h be the gain and r be the offset. Then, we have(3)υi−k=hkεk⋅υik−Nk+r.This paper classifies the voice emotional features with softmax cross-entropy loss function. Identical to the m-classification problem, the inputs of our network can be mapped to real numbers in [0, 1] through regularization, with the sum of all inputs being 1. For any voice sampleE, the probability of being assigned to voice emotion class Di is GLi. Then, we have(4)GL1+GL2+⋯+GLm=1.The voice emotion class of voice sampleE can be expressed as(5)maxGL1,GL2,…,GLm.According to the requirements of the m-classification problem, the softmax function outputs the probabilityGLi of each voice emotion class Di. The maximum probability max (GL1, GL2, …GLm) is the most probable class of the inputs. The inputs of the softmax function consist of transformer network output and CNN output. Let δ and γ be the hyperparameters to be adjusted during the training; ωij be the j-th weight of node i related to the CNN output; ωnm be the n-th weight of node m related to transformer network output. Then, the i-th output can be expressed as(6)Ci=∑jωijC+rC+γ∑mωnmTR+rTR.Next, the softmax function is added to the network output. Then, thei-th output Ok can be expressed as(7)Oi=eCi∑j=1meCi.The cross-entropy function, which is easy to derive and compute, was selected as the softmax loss function [24]. Let bi be the actual classification result. The cross-entropy function can speed up the network learning:(8)L=−∑ibilnυi.


### 2.2. Loss Function

To facilitate the design of the classifier for voice emotion identification, the distance between different classes of voice emotions should be maximized, while that between the same class of voice emotions under different scenarios should be minimized. As a common classification method for multiclass tasks, the softmax cross-entropy loss function can learn the separable features in order to differentiate between various voice emotions. For this purpose, the loss function was introduced to the proposed voice emotion identification model, which draws voice emotions to the center of their corresponding voice emotion classes. Letdj be the center of the i-th class of voice emotions. Then, the loss function can be expressed as(9)LCenter∗=−1n∑i=1nci−dbi2.By optimizing the loss function, the distance between the same class of voice emotional features will become smaller. After initializingdj as 0 and defining the class center of each minibatch as d˙j, the class center d′j of the j-th type of features in the minibatch can be calculated by(10)dj′=∑i=1nξbi=jCi∑i=1nξbi=j>0.Ifbi = j, then ξ (bi = j)Ci = 1. Let δ be the hyperparameter that controls the update rate of dj, if the new minibatch has the voice emotional features corresponding to the j-th class of emotions; djτ and d˙jτ be the values of dj and d˙j in the τ-th iteration, respectively. In other cases, ξ (bi = j)Ci = 0. Then, the class center dj can be defined as(11)djτ+1=1−δdjτ+δd˙jτ,∑i=1nξbi=j>0,djτ,∑i=1nξbi=j=0.Letθj be the inverse ratio of class j voice emotions in the entire voice sample training set. Because of the imbalance between voice emotion classes, two weights were assigned to softmax cross-entropy loss function and loss function:(12)LSoftMax=1∑i=1nθbi∑i=1nθbilnυi,LCenter=1∑i=1nθbici−dbi2.Hence, this paper uses a joint loss function to train the proposed voice emotion identification model. Letμ be the hyperparameter that balances central loss with softmax cross-entropy loss. Then, the joint loss function composed of softmax cross-entropy loss function and loss function can be expressed as(13)L=LSoftMax+μLCenter.


### 2.3. Emotional Labeling

The proposed voice emotion identification model was trained on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), which contains eight types of emotions with a frame precision of nearly 80%. The voice samples were labeled directly by the naming rule of voice emotion data in that database. Each voice sample was named in the format of “audio number-data type-emotion label-data intensity-semantic content-repetition-user number.wav.” Specifically, the data type is speech 01 or song 02; the emotion label could be one of the following: indifferent 01, calm 02, happy 03, sad 04, angry 05, fearful 06, disgust 07, and surprise 08; the data intensity is either normal 01 or strong 02; the repetition is 01 (first repetition) or 02 (second repetition).In the RAVDESS, the emotions are classified into eight types 01–08, which are common in most datasets. To ensure the emotional recognition effect of our model on subjects of different voice features and maintain the diversity of voice emotions, this paper randomly divides the voice sample set into eight subsets. Five of them were used for training and the remaining three for testing.


## 2.1. Network Design

The transformer encoder structure was adopted to increase the number of features. If the data are batch-processed by instance normalization, the features extracted by the transformer network will differ slightly from those extracted by the CNN in terms of the internal relationship.Moreover, the voice sample set for the IVAUTs covers the samples of 30 users. Since each user has unique voice features, the extracted features contain lots of personal features: the intraclass distance is even greater than the interclass distance. If the voice emotional features are classified by the softmax function, the intraclass distance would be enlarged, undermining the recognition effect of the model.To solve the above problem, the voice emotion identification model with loss function was introduced to reduce the intraclass distance. The loss function learns the center of deep voice features in each class and penalizes the central features of the expected classes. In this way, the interclass distance is increased, and the intraclass distance is reduced. As shown in Figure1, the established network consists of an input layer, multiple convolutional layers, a multihead attention mechanism, a fully connected layer, and an output layer.Figure 1
Structure of our voice emotion identification network.The voice emotion model has a special requirement on the overall distribution of sample data. To keep the data distribution consistent, the batch normalization module in the network normalizes each batch of voice samples. The mean and variance of each batch are greatly affected by the data size of that batch. If the data are too few, it is impossible to characterize the sample distribution with the calculated mean and variance. Voice emotion recognition usually deals with the entire input sentence. LetF be the number of hidden units on each layer; k be the k-th hidden layer; v be the value of a node before activation. To obtain more, richer emotional features of voice sentences, layer normalization was adopted in our transformer encoder. The input of all nodes is regularized by(1)Nk=1F∑i=1Fυik,(2)εk=1F∑i=1Fυik−Nk2.By formulas (1) and (2), the expectation N and standard error ε can be obtained for each layer. The hidden units in the same layer share the same normalized expectation N and standard error ε. However, the networks trained on different cases would have different normalized expectations N and standard error ε. Let h be the gain and r be the offset. Then, we have(3)υi−k=hkεk⋅υik−Nk+r.This paper classifies the voice emotional features with softmax cross-entropy loss function. Identical to the m-classification problem, the inputs of our network can be mapped to real numbers in [0, 1] through regularization, with the sum of all inputs being 1. For any voice sampleE, the probability of being assigned to voice emotion class Di is GLi. Then, we have(4)GL1+GL2+⋯+GLm=1.The voice emotion class of voice sampleE can be expressed as(5)maxGL1,GL2,…,GLm.According to the requirements of the m-classification problem, the softmax function outputs the probabilityGLi of each voice emotion class Di. The maximum probability max (GL1, GL2, …GLm) is the most probable class of the inputs. The inputs of the softmax function consist of transformer network output and CNN output. Let δ and γ be the hyperparameters to be adjusted during the training; ωij be the j-th weight of node i related to the CNN output; ωnm be the n-th weight of node m related to transformer network output. Then, the i-th output can be expressed as(6)Ci=∑jωijC+rC+γ∑mωnmTR+rTR.Next, the softmax function is added to the network output. Then, thei-th output Ok can be expressed as(7)Oi=eCi∑j=1meCi.The cross-entropy function, which is easy to derive and compute, was selected as the softmax loss function [24]. Let bi be the actual classification result. The cross-entropy function can speed up the network learning:(8)L=−∑ibilnυi.


## 2.2. Loss Function

To facilitate the design of the classifier for voice emotion identification, the distance between different classes of voice emotions should be maximized, while that between the same class of voice emotions under different scenarios should be minimized. As a common classification method for multiclass tasks, the softmax cross-entropy loss function can learn the separable features in order to differentiate between various voice emotions. For this purpose, the loss function was introduced to the proposed voice emotion identification model, which draws voice emotions to the center of their corresponding voice emotion classes. Letdj be the center of the i-th class of voice emotions. Then, the loss function can be expressed as(9)LCenter∗=−1n∑i=1nci−dbi2.By optimizing the loss function, the distance between the same class of voice emotional features will become smaller. After initializingdj as 0 and defining the class center of each minibatch as d˙j, the class center d′j of the j-th type of features in the minibatch can be calculated by(10)dj′=∑i=1nξbi=jCi∑i=1nξbi=j>0.Ifbi = j, then ξ (bi = j)Ci = 1. Let δ be the hyperparameter that controls the update rate of dj, if the new minibatch has the voice emotional features corresponding to the j-th class of emotions; djτ and d˙jτ be the values of dj and d˙j in the τ-th iteration, respectively. In other cases, ξ (bi = j)Ci = 0. Then, the class center dj can be defined as(11)djτ+1=1−δdjτ+δd˙jτ,∑i=1nξbi=j>0,djτ,∑i=1nξbi=j=0.Letθj be the inverse ratio of class j voice emotions in the entire voice sample training set. Because of the imbalance between voice emotion classes, two weights were assigned to softmax cross-entropy loss function and loss function:(12)LSoftMax=1∑i=1nθbi∑i=1nθbilnυi,LCenter=1∑i=1nθbici−dbi2.Hence, this paper uses a joint loss function to train the proposed voice emotion identification model. Letμ be the hyperparameter that balances central loss with softmax cross-entropy loss. Then, the joint loss function composed of softmax cross-entropy loss function and loss function can be expressed as(13)L=LSoftMax+μLCenter.


## 2.3. Emotional Labeling

The proposed voice emotion identification model was trained on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), which contains eight types of emotions with a frame precision of nearly 80%. The voice samples were labeled directly by the naming rule of voice emotion data in that database. Each voice sample was named in the format of “audio number-data type-emotion label-data intensity-semantic content-repetition-user number.wav.” Specifically, the data type is speech 01 or song 02; the emotion label could be one of the following: indifferent 01, calm 02, happy 03, sad 04, angry 05, fearful 06, disgust 07, and surprise 08; the data intensity is either normal 01 or strong 02; the repetition is 01 (first repetition) or 02 (second repetition).In the RAVDESS, the emotions are classified into eight types 01–08, which are common in most datasets. To ensure the emotional recognition effect of our model on subjects of different voice features and maintain the diversity of voice emotions, this paper randomly divides the voice sample set into eight subsets. Five of them were used for training and the remaining three for testing.


## 3. Facial Emotion Identification System

Each facial expression is a dynamic process in a period of time, which is induced by the movement of facial muscles. Compared with the single frame-based static facial expression identification, the dynamic identification oriented at vehicle videos can acquire facial expression features with high accuracy. This paper sets up a bimodal fusion model based on decompose-CNNs (Figure2), which mainly consists of an appearance module, an optical flow module, and a bimodal fusion module. Based on three-dimensional (3D) decompose-CNN, the appearance module processes the red-green-blue (RGB) image sequence of video frames. Based on two-dimensional (2D) decompose-CNN, the optical flow module processes a single optical flow image, which normally encompasses a start frame and vertex frames. The bimodal fusion module applies a consistency constraint on the emotion labels predicted by facial expressions of different modals, such as integrating the information of the appearance module with that of the optical flow module in the feature space.Figure 2
Architecture of bimodal fusion model.The time information and spatial information of a frame can be extracted by one-dimensional (1D) convolution and 2D convolution, respectively. 1D convolution, which operates in the height and width directions, can be naturally decomposed through 2D convolution. Both 1D and 2D convolutions can be decomposed by 3D convolution. Let Γ(.) be a complete 2D convolutional layer; Qφ and Qϕ be kernels of the size 1 × R and R × l, respectively; Γ3D(.) be a 3D convolutional layer; Qρ and Qσ be the kernels of the size1 × R × R and R × l × l, respectively; P∈ℝS×Q×Uϕ and P∈ℝΨ×S×Q×Uσ be the final feature maps output by the decompose-CNNs, respectively; Ψ, S, and Q be the time length, height, and width, respectively; Uϕ and Uσ be the number of channels. Then, this paper defines the two types of decompose-CNNs. The feature maps G and G output by the preceding network layer can be, respectively, calculated by the following decompose-convolutions:(14)Ρ=ΓQφ,ΓQϕ,G,P=Γ3DQσ,Γ3DQρ,G.The decomposition of 2D and 3D convolutions greatly reduces the number of parameters and computing complexity of the network, resulting in more efficient recognition of facial expressions. Besides, the network layers are increased to enhance the ability of the network to handle nonlinear data. Figure3 shows the decomposition process of 2D convolution. The decomposition process of 3D convolution is similar to Figure 3.Figure 3
Decomposition process of 2D convolution.The optical flow indicates the direction and intensity of pixel motion in video frames. Suppose pixel (a), (b) moves to (a+Δa, b+Δb) from moment ψ to moment ψ+Δψ. Let v and s be the horizontal and vertical components of the optical flow. Then, the vector of the optical flow can be calculated by(15)q=v,sT=ΔaΔψ,ΔbΔψT.The motion state between frames can be described by the optical flow vector between the dense optical flow images of adjacent car video frames. If the car video containsk frames, it is possible to obtain k-1 optical flow images. The more the frames, the more complex the extraction of optical flow.The facial expressions in car video frames can be perceived synergistically from different angles, for example, appearance information and optical flow information. Figure4 shows the structure of 2D decompose-CNN. If the two modals of appearance features and optical flow features of facial expressions are simply connected in series, the correlation between the two kinds of features will be ignored. To fully utilize the correlation, this paper fuses the two modals consistently and constructs a classification constraint based on fused features, aiming to reduce the classification difference of facial expressions between the two modals.Figure 4
Structure of 2D decompose-CNN.
(a)(b)Letχi be the eigenvector of the i-th sample; GL1 and GL2 be the probability distributions of the same length. Then, the matching degree of GL2 to GL1 can be measured by relative entropy Φ:(16)ΦGL1‖GL2=−GL1χilogGL1χiGL2χi.Because of the asymmetry of relative entropy Φ, that is, Φ(GL1||GL2) ≠ Φ(GL2||GL1), the difference between the two probability distributions can be characterized by symmetric relative entropy Φ∗:(17)Φ∗GL1‖GL2=12ΦGL1‖GL2+12ΦGL2‖GL1.Letχt be the eigenvector obtained through the consistent fusion between appearance eigenvector χn and optical flow eigenvector χm; M be the number of samples; βi be the probability distribution of the actual labels of the i-th sample; GL(χti) be the probability distribution of predicted labels obtained based on the fused eigenvector χti of the i-th sample. Then, the classification constraint can be calculated by(18)Ω=∑i∈MΦ∗βi||GLχit.Let W be the number of classes;ζχ be the parameter vector of class w prediction. Then, GL(χti) can be calculated by softmax function:(19)GLχi=1∑w=1WeζaTχjeζ1Tχi,eζ2Tχi,…,eζwTχiT.In theory, the appearance features and optical flow features predicted for the same input should have similar modals; that is, the two types of features should be consistent. To prevent the extraction error of a single modal from affecting the model output, this paper applies a consistency constraint on the probability distributionsGL(χin) and GL(χim) of predicted feature labels corresponding to the two modals:(20)PLPD=η∑i∈MΦ∗GLχin‖GLχim,where η is a hyperparameter. Formula (20) ensures the effective synthesis of complementary information of the two modals. Let Δ = {ζn, ζm, ζt} be the online parameter of the modal fusion module. The final loss function can be given by(21)LΔ=Ω+PLPD.Figure5 explains the flow of modal fusion. Gradient descent was further adopted to optimize the network loss function, such that the probability distribution of predicted feature labels based on fused facial expressions approximates the probability distribution of actual feature labels. The partial derivative of the loss function can be solved by(22)∂LΔ∂ζn=η∑i=1M∂Φ∗GLχin‖GLχim∂ζn=η2∑i=1M∂ΦGLχin‖GLχim∂ζjkn+∂ΦGLχim‖GLχin∂ζjkn=−η2∑i=1M∑w=1WGLw|χimGLw|χin+lnGLw|χim∂GLw|χin∂ζjkn,where(23)∂GLw|χin∂ζjkn=∂eζwnTχin/∑w=1WeζwnTχin∂ζjkn=ξw=j−GLw|χinGLw|χinχikn.Figure 5
Flow of modal fusion.The probability distribution of labels predicted by appearance features approaches that of labels predicted by optical flow features, enhancing the robustness of the joint optimization and fused features. Hence, the emotion identification effect of our model could be improved. Letζnj be a subvector of ζn; ζnjk be the k-th element in ζnj. The derivatives of the first and second terms of formula (21) can be solved similarly relative to ζm and ζt.


## 4. Experiments and Results Analysis

Hyperparametersδ, μ, and γ are the weight of class center update rate, weight of loss function, and weight of encoder output, respectively. This paper designs a contrastive experiment to explore the influence of these hyperparameters on the voice emotion recognition effect. The voice emotion recognition accuracy under different hyperparameter settings is reported in Figure 6, where features A and B are Mel cepstral coefficient (MCC) and Mel frequency cepstral coefficient (MFCC), respectively. Comparing the three subgraphs of Figure 6, the voice emotion recognition accuracy was not sensitive to δ but significantly affected by μ and γ. If μ and γ are too large or too small, the recognition accuracy would be very low.Figure 6
Voice emotion recognition accuracy under different hyperparameter settings.
(a)(b)(c)Figure7 displays the confusion matrix of our voice emotion identification network based on MFCC. Our model performed excellent on the recognition of calm, sad, surprise, and disgust but made a few errors in recognition of the other four emotions.Figure 7
MFCC-based confusion matrix.Table1 compares the emotion recognition rates of different models. Compared with traditional CNN, the combined network achieved relatively high accuracy in emotion recognition: the recognition rate surpassed 80% on all emotions, except calm (77.45%); the highest recognition rate was realized on disgust (91.72%). After introducing Lcenter, our model became much more accurate than the combined network. Therefore, the introduction of Lcenter can enhance the effectiveness of our model in voice emotion identification.Table 1
MFCC-based emotion recognition rates.
Name of modelCNNCombined network (before introducingLcenter)Our model (after introducingLcenter)Indifferent (%)83.4685.7987.35Calm (%)82.7277.4597.45Happy (%)82.3588.7286.54Sad (%)87.5481.2697.72Angry (%)87.3684.5488.46Fearful (%)88.4188.4688.72Disgust (%)83.5891.7293.27Surprise (%)83.2190.2195.31Furthermore, MCC was imported to our network. The hyperparameters were set toδ = 0.85, μ = 0.15, and γ = 1 after sensitivity analysis. Six experiments were carried out, and the most accurate results were selected for analysis. Figure 8 presents the MCC-based confusion matrix, and Table 2 lists the recognition rate of each voice emotion. Obviously, the classification effect of voice emotions could be improved by transformer network and loss function.Figure 8
MCC-based confusion matrix.Table 2
MCC-based emotion recognition rates.
Name of modelCNNCombined network (before introducingLcenter)Our model (after introducingLcenter)Indifferent (%)87.2291.2694.26Calm (%)79.7583.2183.21Happy (%)82.3590.7296.72Sad (%)81.5492.3898.38Angry (%)84.2688.4688.46Fearful (%)85.5889.7289.72Disgust (%)87.2190.2196.21Surprise (%)89.5482.3492.34This paper compares the MCC- and MFCC-based emotion recognition scores of CNN, combined network, and our model. The results of the three models are compared in Table3. The results show that our model coupled with MCC achieved the best results among all possible combinations. This is the best combination for emotion recognition of the selected voice emotion sample set.Table 3
Performance scores of different models.
Name of modelPrecision (%)Recall (%)F1-score (%)MFCCCNN92.3590.7291.54Combined network (before introducingLcenter)92.3592.3192.23Our model94.2394.2094.20MCCCNN91.6291.2391.76Combined network (before introducingLcenter)93.5793.6293.85Our model95.4894.4894.48Figure9 compares the emotion recognition accuracies of facial expressions under different fusion mechanisms. Different fusion mechanisms had similar accuracy and precision trends on the two datasets, which involve different users. Our model achieved better recognition accuracy than decision fusion, which cannot characterize the distribution features of different features, and series fusion, which merely stitches up the information from different modals. The main reason is that our model further refines the loss function and thus shortens the distance between the predicted label distributions of different modals.Figure 9
Emotion recognition accuracies of facial expressions under different fusion mechanisms.


## 5. Conclusions

This paper mainly develops an IVAUT system based on emotion identification technology. Specifically, a combined network was designed to identify voice emotions for IVAUTs, based on the transformer network and the CNN, and the setting of emotion labels was explained in detail. Next, a bimodal fusion model was developed based on decompose-CNNs. There are three major modules in the model: appearance, optical flow, and bimodal fusion. After that, the voice emotion recognition accuracy was analyzed under different hyperparameter settings, the confusion matrices of our model were established based on MFCC and MCC, respectively, and the emotion recognition accuracies of different models on MFCC and MCC were calculated. The results show that our model, coupled with MCC, achieved better results than any other combination. Finally, the authors tested the emotion recognition accuracies of facial expressions under different fusion mechanisms and confirmed the effectiveness of our emotion identification approach.The significance of this research lies in identifying the negative emotions of drivers in voices and facial images and making warning and intervention via IVAUTs, aiming to prevent emotional driving behaviors. The research provides a reference for the design and development of future smart transportation systems.Due to the difficulty in acquiring lots of real data, this research faces several limitations: the conditions and means of emotional identification experiments are very limited, and the test and evaluation data were insufficiently quantified for the voice and facial emotion identification techniques applied in IVAUTs. Future work will try to overcome these limitations through in-depth research.



---

*Source: 6315063-2022-02-14.xml*