# Mental State Assessment in College English Teaching Courses Based on Deep Learning

Authors: Beibei Ji
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

College English teaching aims at students with different foundations and characteristics. Also, it is necessary to grasp the mental state of students in time during the teaching process. Based on the bimodal information of facial expressions and speech of classroom students, this paper designs a mental state assessment method based on deep learning. For bimodal emotion recognition of facial expressions and speech, a feature fusion method based on sparse canonical correlation analysis (SCCA) is proposed in this paper. First, the emotional features of the facial expression and speech are extracted, respectively. Then, SCCA is used to fuse the emotional features of the two modalities. Finally, the sparse representation-based classification (SRC) is used as the classifier for emotional prediction. Based on the prediction results, the mental state of different students can be grasped, so as to adjust the teaching strategy in a targeted manner. Experiments are carried out based on public datasets. First, the proposed method achieves the average classification accuracy of 92.4%, which is higher than those from the present methods for comparison. Second, under the condition of noise corruption, the proposed method keeps the superior robustness over the comparison methods. The results show that the proposed bimodal emotion recognition method based on SCCA and SRC can achieve higher recognition rates than some present methods.


---

## Body

## 1. Introduction

With the development of the economy and society, people begin to pay more attention to their own mental health. In the university stage, due to the rapid changes in the internal and external environment of students, many of them cannot adapt in time, which is prone to psychological problems. According to research statistics, the psychological problems of college students have obvious stage characteristics, and many students cannot detect their own psychological changes in time, which leads to the deterioration of psychological problems and has serious consequences. Taking English teaching in colleges and universities as an example, due to the different foundations of English learning and different interests in the student group, different students may have different psychological emotions towards this course. Therefore, it is an important way to correctly adjust the educational method and improve the teaching efficiency to do a good job of mental state assessment in the classroom in time. It is in this context that this paper aims to support the English teaching based on the emotional analysis of students’ expressions and speeches during the English teaching classes to obtain their corresponding mental states [1–3].In the past few decades, many researchers all over the world have used several commonly used human-computer interaction modalities (such as facial expressions, speech, and gestures) to conduct emotion recognition. Some effective expression recognition, speech emotion recognition, and action gesture recognition methods were developed [4–7]. Among these methods, the vast majority are emotion recognition methods based on a single modality. However, in future applications such as emotional robots, unmanned driving, and intelligent transportation, the emotional interaction between humans and machines is often based on bimodal or multimodality. For example, emotional robots can perform human-computer interaction through both facial expression and speech [8–10]. Therefore, the study of multimodal emotion recognition methods based on facial expressions and speech plays a very important role in the development and progress of future technologies such as emotional robots, unmanned driving, and intelligent transportation [11–13].At present, some researchers have initially carried out research on bimodal emotion recognition based on facial expressions and speech. Reference [13] proposed a multimodal method of facial expression and speech based on the fusion mechanism of the decision-making layer and achieved certain fusion effects. Reference [14] proposed a multimodal facial expression and speech method based on the multistream hidden Markov model (MHMM), which achieved good performance after fusion. Reference [15] studied a multimodal approach to facial expression and speech based on direct fusion and back-propagation (BP) neural network classifiers. Reference [16] employed the kernel cross-modal factor analysis (KCFA) algorithm to perform feature dimensionality reduction and feature fusion for speech and facial expression modalities. Reference [17] used the kernel entropy component analysis (KECA) algorithm and the decision layer fusion to study facial expression and speech bimodal emotion recognition and achieved high dual-mode emotion recognition on two commonly used emotion databases. In recent years, deep learning models have provided powerful tools for face and speech emotion recognition, and representative methods include convolution neural network (CNN), long short-term memory (LSTM), and generative adversarial network (GAN). [18, 19].In order to effectively improve the emotion recognition performance during English teaching classes so as to grasp the mental state of different students, this paper proposed a dual-modal emotion analysis method based on facial expression and speech. First, the proposed method extracts the features of facial expression and speech, respectively. Then, the sparse canonical correlation analysis (SCCA) [20–22] algorithm is used to fuse the two kinds of features to obtain a unified feature. Finally, the sparse representation-based classification (SRC) is used for bimodal emotion recognition. Based on the emotion analysis results, each student in the classroom can be observed, and their mental states toward this class can be analyzed. The main contribution of this paper can be summarized as follows. The dual-modal information during the teaching is properly analyzed and fused by SCCA. The fused result can better convey the mental state of the students. SRC performs as the classification scheme and finally gets the result of the mental state. Experiments are performed based on the public dataset. According to the experimental results, the validity and superiority of the proposed method can be verified by comparison with several published methods in this field.


## 2. Expression and Speech Emotion Feature Extraction

For facial expression and speech bimodal emotion data, this paper first performs feature extraction. For facial expression modalities, this paper uses scale-invariant feature transform (SIFT) to extract features. SIFT feature is a very effective image feature extraction method, which is widely used in action recognition, motion detection, and facial expression recognition, due to its robustness and antinoise advantages.The previous works show that the SIFT feature extraction process generally includes the extraction of extreme points in the original image, the selection of feature points, the gradient solution of feature points, and the generation of feature point descriptors. In practical applications such as facial expression recognition, the following steps of solving the gradient of feature points and generating step of feature point descriptors are the most critical. The gradient of the feature point is mainly calculated by the following two equations:(1)ρx,y=J^x,y+1−J^x,y−12+J^x+1,y−J^x−1,y2,(2)θx,y=tan−1J^x,y+1−J^x,y−1J^x+1,y−J^x−1,y.In equations (1) and (2), ρx,y, θx,y, and J^x,y represent the required gradient size, gradient direction, and Gaussian smoothed image, respectively. After the gradient is obtained via equations (1) and (2), the direction of each feature point is first calculated by the histogram method, and then, SIFT feature vectors of different dimensions are extracted based on the obtained direction.For speech modalities, this paper uses openSMILE software to extract the emotional features of speech modalities. Compared with the traditional speech emotion feature extraction method, the speech emotion feature based on openSMILE software is more convenient and direct. As long as the corresponding speech audio is input, the rich speech emotion feature can be directly extracted through a simple operation.


## 3. Classroom Psychological Assessment Based on Emotion Analysis

### 3.1. Feature Fusion

At first, SCCA is used to fuse the features of the two modalities from facial expression and speech [20–22]. The SCCA algorithm can be expressed as follows:(3)argminAW,ASLSCCA=argminAW,ASWWT−1/2W−AWASTSF2+ηWAW1+ηSAS1.In equation (3), S is the extracted speech feature matrix; W is the facial expression feature matrix extracted by the SIFT method; AW is the projection matrix of W; AS is the projection matrix of S; ηW is the sparse parameter of AW; and ηS is the sparse parameter of AS.According to the relevant researches, the problem in equation (3) can be resolved by the augmented Lagrangian algorithm. Let A˜W=AW and A˜S=AS, equation (3) can be reformulated as follows:(4)argminAW,AS,A˜W,A˜SLSCCA=argminAW,AS,A˜W,A˜SWWT−1/2W−AWASTSF2+ηSAS1+trΓWTA˜W−AW+ηWAW1+trΓSTA˜S−AS+λW2A˜W−AWF2+λS2A˜S−ASF2,where λW is the norm coefficient of A˜W−AW; λS is the norm coefficient of A˜S−AS; ΓWT is the Lagrangian multiplier matrix of A˜W−AW; ΓST is the Lagrangian multiplier matrix of A˜S−AS.Equation (4) can be further rewritten as follows:(5)argminAW,AS,A˜W,A˜SLSCCA=argminAW,AS,A˜W,A˜StrWWT−1/2WWTWWT−1/2−2trWWT−1/2WSTA˜SA˜WTWWT−1/2+trWWT−1/2A˜WA˜STSSTA˜SA˜WTWWT−1/2+ηSAS1+ηWAW1+trΓSTA˜S−ΓSTAS+λS2trA˜SA˜ST−2A˜SAST+ASAST+λW2trA˜WA˜WT−2A˜WAWT+AWAWT+trΓWTA˜W−ΓWTAW.Taking the partial derivatives of (5) with respect to A˜W and A˜S, respectively, and setting them to 0, we can get(6)∂LSCCA∂A˜W=−2WWT−1WSTA˜S+ΓW+λWA˜W−λWAW+2WWT−1A˜WA˜STSSTA˜S=0,∂LSCCA∂A˜S=−2WWT−1SWTA˜W+ΓS+λSA˜S−λSAS+2WWT−1SSTA˜SA˜WTA˜W=0.Then, the solution is(7)A˜W=AW+2λWWWT−1WSTA˜S−2λWWWTA˜WA˜STSSTA˜S−ΓWλW,A˜S=AS+2λSWWT−1SWTA˜W−2λSWWTSSTA˜SA˜WTA˜W−ΓSλS.According to the method introduced in [21], after obtaining A˜W and A˜S, the solution of AW and AS can be transformed into(8)argminAWηWλWAW+12AW−A˜W+ΓWλWF2,argminASηSλSAS+12AS−A˜S+ΓSλSF2.Then, we can get(9)AW=ξηWλWA˜W+ΓWλW,AS=ξηSλSA˜S+ΓSλS.Among them, the functionξ is the threshold function defined in [21].Finally, the bimodal feature fusion based on SCCA is obtained as follows:(10)AWTWASTS.


### 3.2. Emotion Classification

SRC uses sparse representation for pattern recognition problems, which characterizes the unknown input through training samples of known categories and then determines the category of test samples according to the reconstruction errors of different categories. SupposingD=D1,D2,…,DC∈Rd×N is a global dictionary, where Di∈Rd×Nii=1,2,…,C represents Ni training samples from ith class, and the sparse representation process of the test sample y can be built as follows:(11)x^=argminxx0,s.t.y−Dx22≤ε.In the above equations,x represents the sparse coefficient vector. According to relevant works, the algorithms commonly used to solve sparse representation problems include ℓ1 norm optimization and orthogonal matching pursuit algorithm (OMP) [13–16]. Based on the solution of x^, the reconstruction errors for the test samples are calculated according to the categories, and finally, the category of the test samples is determined as follows:(12)ri=y−Dixi22i=1,2,…,C,identityy=argminiri,where xi is the coefficient vector corresponding to the ith class; ri is the corresponding reconstruction error.Compared with CNN, the classification mechanism of SRC is relatively less dependent on the number of test samples. At the same time, existing research results show that SRC has certain adaptability to complex situations such as noise interference and occlusion. Figure1 shows the basic process of the proposed method in this paper. Based on the extracted facial expressions and speech features, SCCA is used for feature fusion, and finally, the category is confirmed based on SRC to obtain the emotional and mental state of the current test sample. Specifically, during English teaching courses, the proposed method can be used to automatically obtain the current emotional state of different students, reflecting their psychological acceptance of the current course. Such analysis results can be used to assist in the adjustment and optimization of teaching methods and teaching forms.Figure 1
Basic flowchart of the proposed method.


## 3.1. Feature Fusion

At first, SCCA is used to fuse the features of the two modalities from facial expression and speech [20–22]. The SCCA algorithm can be expressed as follows:(3)argminAW,ASLSCCA=argminAW,ASWWT−1/2W−AWASTSF2+ηWAW1+ηSAS1.In equation (3), S is the extracted speech feature matrix; W is the facial expression feature matrix extracted by the SIFT method; AW is the projection matrix of W; AS is the projection matrix of S; ηW is the sparse parameter of AW; and ηS is the sparse parameter of AS.According to the relevant researches, the problem in equation (3) can be resolved by the augmented Lagrangian algorithm. Let A˜W=AW and A˜S=AS, equation (3) can be reformulated as follows:(4)argminAW,AS,A˜W,A˜SLSCCA=argminAW,AS,A˜W,A˜SWWT−1/2W−AWASTSF2+ηSAS1+trΓWTA˜W−AW+ηWAW1+trΓSTA˜S−AS+λW2A˜W−AWF2+λS2A˜S−ASF2,where λW is the norm coefficient of A˜W−AW; λS is the norm coefficient of A˜S−AS; ΓWT is the Lagrangian multiplier matrix of A˜W−AW; ΓST is the Lagrangian multiplier matrix of A˜S−AS.Equation (4) can be further rewritten as follows:(5)argminAW,AS,A˜W,A˜SLSCCA=argminAW,AS,A˜W,A˜StrWWT−1/2WWTWWT−1/2−2trWWT−1/2WSTA˜SA˜WTWWT−1/2+trWWT−1/2A˜WA˜STSSTA˜SA˜WTWWT−1/2+ηSAS1+ηWAW1+trΓSTA˜S−ΓSTAS+λS2trA˜SA˜ST−2A˜SAST+ASAST+λW2trA˜WA˜WT−2A˜WAWT+AWAWT+trΓWTA˜W−ΓWTAW.Taking the partial derivatives of (5) with respect to A˜W and A˜S, respectively, and setting them to 0, we can get(6)∂LSCCA∂A˜W=−2WWT−1WSTA˜S+ΓW+λWA˜W−λWAW+2WWT−1A˜WA˜STSSTA˜S=0,∂LSCCA∂A˜S=−2WWT−1SWTA˜W+ΓS+λSA˜S−λSAS+2WWT−1SSTA˜SA˜WTA˜W=0.Then, the solution is(7)A˜W=AW+2λWWWT−1WSTA˜S−2λWWWTA˜WA˜STSSTA˜S−ΓWλW,A˜S=AS+2λSWWT−1SWTA˜W−2λSWWTSSTA˜SA˜WTA˜W−ΓSλS.According to the method introduced in [21], after obtaining A˜W and A˜S, the solution of AW and AS can be transformed into(8)argminAWηWλWAW+12AW−A˜W+ΓWλWF2,argminASηSλSAS+12AS−A˜S+ΓSλSF2.Then, we can get(9)AW=ξηWλWA˜W+ΓWλW,AS=ξηSλSA˜S+ΓSλS.Among them, the functionξ is the threshold function defined in [21].Finally, the bimodal feature fusion based on SCCA is obtained as follows:(10)AWTWASTS.


## 3.2. Emotion Classification

SRC uses sparse representation for pattern recognition problems, which characterizes the unknown input through training samples of known categories and then determines the category of test samples according to the reconstruction errors of different categories. SupposingD=D1,D2,…,DC∈Rd×N is a global dictionary, where Di∈Rd×Nii=1,2,…,C represents Ni training samples from ith class, and the sparse representation process of the test sample y can be built as follows:(11)x^=argminxx0,s.t.y−Dx22≤ε.In the above equations,x represents the sparse coefficient vector. According to relevant works, the algorithms commonly used to solve sparse representation problems include ℓ1 norm optimization and orthogonal matching pursuit algorithm (OMP) [13–16]. Based on the solution of x^, the reconstruction errors for the test samples are calculated according to the categories, and finally, the category of the test samples is determined as follows:(12)ri=y−Dixi22i=1,2,…,C,identityy=argminiri,where xi is the coefficient vector corresponding to the ith class; ri is the corresponding reconstruction error.Compared with CNN, the classification mechanism of SRC is relatively less dependent on the number of test samples. At the same time, existing research results show that SRC has certain adaptability to complex situations such as noise interference and occlusion. Figure1 shows the basic process of the proposed method in this paper. Based on the extracted facial expressions and speech features, SCCA is used for feature fusion, and finally, the category is confirmed based on SRC to obtain the emotional and mental state of the current test sample. Specifically, during English teaching courses, the proposed method can be used to automatically obtain the current emotional state of different students, reflecting their psychological acceptance of the current course. Such analysis results can be used to assist in the adjustment and optimization of teaching methods and teaching forms.Figure 1
Basic flowchart of the proposed method.


## 4. Experiment and Analysis

### 4.1. Introduction of Dataset

The RECOLA dataset is a commonly used dataset for emotion recognition containing speech and visual data, providing audio and video recordings, image, and sound features, some time-specific events, and some other metadata of 46 different experimental participants. The voice module in the dataset contains the original recording, the start and end times of speech, the predicted probability of voice activity, and other characteristics of the voice. The image module in the dataset contains the original videos, the corresponding time of each frame in the video, the predicted probability of face detection, and the features of the image. The original footage was captured by a Logitech webcam, 1080 × 720 pixels, YUV color mode, and fixed FPS at 25 frames per second. The raw data are annotated with emotion (Arousal and Valence) and type of laughter (silent laughter, normal laughter, talking, and talking laughter). The dataset also provides some other information, such as physiological signals, age, gender, and native language.In the specific use of this paper, we selected 1200 adult facial expression and speech data samples and selected a fivefold cross-validation strategy for testing. In the process, in order to reflect the performance of the proposed method, several categories of methods from the existing literature are used for comparison, including SVM, CNN, and LSTM. At the same time, the performance test is also carried out under the condition of facial expression and voice single mode, which is also used as a comparative experiment for the method in this paper. The average classification accuracy is used as a quantitative evaluation index for the performance of various methods, which is the proportion of correctly classified samples to all test samples.


## 4.1. Introduction of Dataset

The RECOLA dataset is a commonly used dataset for emotion recognition containing speech and visual data, providing audio and video recordings, image, and sound features, some time-specific events, and some other metadata of 46 different experimental participants. The voice module in the dataset contains the original recording, the start and end times of speech, the predicted probability of voice activity, and other characteristics of the voice. The image module in the dataset contains the original videos, the corresponding time of each frame in the video, the predicted probability of face detection, and the features of the image. The original footage was captured by a Logitech webcam, 1080 × 720 pixels, YUV color mode, and fixed FPS at 25 frames per second. The raw data are annotated with emotion (Arousal and Valence) and type of laughter (silent laughter, normal laughter, talking, and talking laughter). The dataset also provides some other information, such as physiological signals, age, gender, and native language.In the specific use of this paper, we selected 1200 adult facial expression and speech data samples and selected a fivefold cross-validation strategy for testing. In the process, in order to reflect the performance of the proposed method, several categories of methods from the existing literature are used for comparison, including SVM, CNN, and LSTM. At the same time, the performance test is also carried out under the condition of facial expression and voice single mode, which is also used as a comparative experiment for the method in this paper. The average classification accuracy is used as a quantitative evaluation index for the performance of various methods, which is the proportion of correctly classified samples to all test samples.


## 5. Result and Discussion

Based on the constructed dataset, the proposed method and the comparison methods are tested. The statistical results are shown in Table1. It can be seen that the method in this paper has a performance advantage over the five comparison methods, and the average classification accuracy is at the highest level. Compared with the traditional machine learning methods such as SVM, deep learning models such as CNN and LSTM show more advantageous performance. Compared with the results under the single-modal condition of facial expression and speech, this paper significantly improves the classification accuracy by combining the two modalities, showing the effectiveness of the proposed method.Table 1
Comparison of performance of different methods.
MethodAverage classification accuracy (%)Proposed92.4SVM85.7CNN89.3LSTM89.5Face only87.1Speech only86.2In the actual process, both the facial expression image and the voice signal may be disturbed by noise. Therefore, it is necessary to improve the ability of emotion analysis under the condition of noise interference. To this end, this paper uses the form of signal-to-noise ratio (SNR) to measure the noise level in the two modalities of facial expression and speech and verifies the classification ability of various methods under the same noise conditions. Figure2 shows the classification accuracy of different methods with the changing of SNR. It can be seen that the method in this paper maintains the optimal classification performance under various noise levels, showing its robustness.Figure 2
Classification accuracy of different methods under noises.


## 6. Conclusion

Aiming at the problem of psychological state monitoring in the process of English teaching in colleges and universities, this paper designs an intelligent evaluation method based on emotion recognition. First, the emotional features of facial expression and speech modalities are extracted from the facial expression and speech bimodal emotional database, respectively. Then, the SCCA algorithm is used to fuse the emotional features of the two modalities, and finally, SRC is used to perform emotion recognition. The experimental results show that the dual-modal emotion recognition method based on SCCA and SRC proposed in this paper can achieve a higher recognition rate than some present methods in the same field. The proposed method can carry out targeted analysis for different students and then adjust teaching strategies according to the statistical results, so as to improve the overall quality of English teaching. In the future, the research will be deepened in two ways. First, more available information will be used besides the facial expression and voices. They can be combined to achieve more reliable results. Second, some new intelligent algorithms can be used to process these modalities to further improve the effectiveness and efficiency.



---

*Source: 8795842-2022-08-08.xml*