# Emotion Recognition of Students Based on Facial Expressions in Online Education Based on the Perspective of Computer Simulation

Authors: Weiqing Wang; Kunliang Xu; Hongli Niu; Xiangrong Miao
Journal: Complexity
 (2020)

---

## Abstract

Online education has developed rapidly due to its irreplaceable convenience. Under the severe circumstances caused by COVID-19 recently, many schools around the world have delayed opening and adopted online education as one of the main teaching methods. However, the efficiency of online classes has long been questioned. Compared with traditional face-to-face classes, there is a lack of direct, timely, and effective communication and feedback between teachers and students in the online courses. Previous studies have shown that there is a close and stable relationship between a person’s facial expressions and emotions generally. From the perspective of computer simulation, a framework combining a face expression recognition (FER) algorithm with online courses platforms is proposed in this work. The cameras in the devices are used to collect students’ face images, and the facial expressions are analyzed and classified into 8 kinds of emotions by the FER algorithm. An online course containing 27 students conducted on Tencent Meeting is used to test the proposed method, and the result proved that this method performs robustly in different environments. This framework can also be applied to other similar scenarios such as online meetings.


---

## Body

## 1. Introduction

Facial expression is one of the most powerful, natural, and universal signals for human beings to convey their emotional states and intentions regardless of national borders, race, and gender [1, 2], and there were multitudinous related applications such as the health management [3], aided driving [4, 5], and others [6–9]. In the earlier researches on facial expressions of emotion, Ekman and Friesen argued that human beings perceive certain basic emotions in the same way regardless of their cultural background, and they defined the typical facial expressions into 6 categories: anger, disgust, fear, happiness, sadness, and surprise [10, 11]. Based on studies of Ekmanand Friesen, Ekman and Heider [12–14], Matsumoto [15] provided enough proofs for another universal facial expression, contempt. Additionally, FER2013 [16], a large-scale and unconstrained database introduced in the ICML 2013 Challenges in Representation Learning, labeled its facial images into anger, disgust, fear, happiness, sadness, surprise, and neutral, which has been widely used in designing facial expression recognition (FER) systems. In subsequent researches, although researchers introduced many models that can provide a wider range of emotions to deal with the complexity and subtlety of facial expressions [17–20], the classification that describes basic emotions discretely is still the most widely used method in FER due to its generality and intuitive definition of facial expressions [21], and Figure 1 displays the 8 basic facial expression phenotypes from datasets CK+ [22] and FER2013 [16].Figure 1
Basic facial expression phenotypes from CK+ and FER2013: from left to right are anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise.For determining facial expressions, Ekman and Friesen [23] proposed a Facial Action Coding System (FACS) for determining facial expressions, which is based on a fact that expressions result from the change of facial parts. With the assistance of the computers, more advanced methods have been proposed during the last decades [24], and the feature points can be seen in Figure 2.Figure 2
Facial analysis points used for facial expression [25].With the development of artificial intelligence and deep learning, numerous FER algorithms have been proposed to deal with the expression information in facial representations, which has improved the accuracy of recognition gradually and achieved better performance than traditional methods [26, 27]. The tasks of FER can be mainly divided into two categories: static images (represented by photographs) [28–30] and dynamic sequence (represented by videos) [31–33] that take the dynamic relationship between the continuously changing images into account and therefore pose additional challenges than the former. In addition to the vision-based methods, other biometric techniques [34, 35] can also be adopted to assist the recognition of expression.Sufficient labeled training databases that include as many variations of the populations and environments as possible are important for researchers to design and test a FER model or system; the existing databases are mainly divided into controlled and uncontrolled. On one hand, the controlled databases, represented by CK+ [22], Jaffe [36], and MMI [37], are collected from laboratory environments with sufficient light and simple backgrounds. Nowadays, because most actual scenes are complex and changeable due to factors such as lighting, FER in laboratory or controlled environments is generally considered to be of little practical significance and used mainly for the proof of concept of features extraction and classification methods. On the other hand, the uncontrolled databases, such as FER2013 [15] and AFEW [38], are collected from complex environments with vastly different backgrounds, occlusions, and illuminations; these scenes are more similar to the actual situations and have been increasingly used in more and more researches.Limited by the hardware and insufficient processing capability, the majority of the traditional methods for FER employed hand-craft features or shallow learning, such as local binary patterns (LBP) [28] and nonnegative matrix factorization (NMF) [39]. With the development of processing capabilities and computer simulation, all kinds of machine learning algorithms, such as Artificial Neural Networks (ANNs), Support Vector Machines (SVM), and Bayesian classifiers, were applied to FER, and the high accuracy has been verified in controlled environments so that the faces can be detected effectively. However, these methods were weak in generalization ability while this is the key to evaluate the practicality of a model [40]. Deep learning algorithms can solve this problem, and it is also robust in the uncontrolled environments. Recent works have shown that convolutional neural networks (CNNs), because of their effectiveness in feature extraction and classification tasks, performed well in addressing the computer vision problems especially in FER [41, 42], and numerous models based on CNN structure are proposed constantly and have achieved better results than previous methods. Simonyan and Zisserman [43] adopted an architecture of very small (3 × 3) convolution filters to conduct a comprehensive evaluation of networks with increasing depth and the two best-performing ConvNet models were available publicly to facilitate the further research in this field. By increasing the depth and width of the network while keeping the computational budget constant, Szegedy et al. [44] introduced a deep convolutional neural network architecture named “Inception” in which the utilization of the computing resources can be improved significantly, and Jahandad et al. [45] worked on 2 convolutional neural network architectures (Inception-v1 and Inception-v3) based on “Inception” and proved that these 2 models performed better than others, and Inception-v1 with 22-layer-deep network performed better than 42-layer-deep Inception-v3 network when facing low-resolution input images and 2D images of signatures; however, Inception-v3 outperformed in ImageNet challenge. The general trend of neural networks is to increase the depth of the network and the width of layer. In theory, the deeper the neural network models, the stronger the learning capabilities, but the more difficult to train. He et al. [46] proposed a residual learning framework to reduce training difficulty of deeper networks and proved thoroughly that these residual networks are easier to optimize while increasing accuracy from the considerably increased depth. In addition, a part of researchers proposed that the accuracy of recognition can be further improved by combining CNNs with recurrent neural networks (RNNs) in which the CNNs are adopted as the inputs to RNNs [47, 48].During the past decades, online education has developed rapidly whether at universities or training institutions [49], which offers potential application opportunities for FER. Significantly different from the traditional face-to-face courses, online courses are often considered of less constraining force and effective communication, which will inevitably lead to faculty’s suspicions towards this novel educational method [50, 51] while there are several studies that argue that the students’ learning outcomes achieved by online education may be comparable to traditional face-to-face courses [52, 53], except for the skills that require optimum precision and a greater degree of haptic awareness [54]. It is undeniable that the rapid growth of online education can effectively provide the convenience and flexibility for more students, so it also has broad development space in the future; therefore, how to ensure that students keep the same level of concentration and learning efficiency as the traditional courses during online education is critical to promote the further development of online education.In brief, the main contribution of this paper is as follows. By combining the existing online education platforms with facial expression recognition model based on the architecture of convolutional neural network, this work proposed a framework that enables real-time monitoring of students’ emotions in online courses and ensures that the feedback expressed by facial expression can be provided to teachers timely, so that they can flexibly adjust the teaching programs and ultimately improve the quality and efficiency of online education.


## 2. Proposed Framework

The framework mainly consists of two parts: online courses platforms, in this paper we took Tencent Meeting as an example for mode testing, and a deep learning model based on CNN, inspired by Kuo et al. [27], before which it is noted that the original images collected from online courses need to be preprocessed, including face detection, alignment, rotation, and resize, according to the different elements in the original images. Figure 3 exhibits the process of the FER, and the detailed steps of the proposed framework are as follows: first, the cameras built in the electronic devices are utilized to capture the facial images of the attending students. Second, the facial expression recognition algorithm trained by the standard facial expression database is employed to detect the faces and classify the facial expressions in terms of anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral. Third, the histogram of probability distribution about the expression is plotted and provided for the teacher so that the teaching plan can be adjusted timely.Figure 3
The process of FER.


### 2.1. Online Education Platforms

The advances in technological delivery modalities have spawned a large number of online education platforms and greatly improved the flexibility of education, enabling teachers to adopt diverse technical methods to assist teaching without worrying about the limit on the number of students in traditional classroom-based course, and students in different regions can communicate in real time without having to consider the traffic and other issues. The same teaching materials as traditional classes can be uploaded to these platforms for students’ reference. Currently, in platforms that have online teaching functions, such as DingTalk, Zoom, and Rain classroom, teachers can adopt the method of video meeting and take advantage of the camera built in devices to capture and recognize students’ facial expressions in real time. The captured images will be preprocessed and then used as the input of CNN.


### 2.2. The Preprocessing Based on IntraFace

Effective preprocessing can reduce the interference of face-like objects in the background when detecting faces in an image and then standardize the face images according to the heuristic knowledge, which will effectively improve the efficiency of the deep learning model. We employed IntraFace [55], a publicly available software package that integrates algorithms for facial feature tracking, head pose estimation, facial attribute detection, etc., as the tool of preprocessing. As shown in Figure 4, IntraFace can also be used to detect multiple faces at the same time. The key features of each face including eyebrows, eyes, nose tip, and mouth can be recognized effectively, and the facial expression can be detected by rectangular outlines accordingly; these outlines are constructed by the feature points at the edge of every face, including the uppermost and the lowermost, which determined the vertical width, and the rightmost and the leftmost, which determined the horizontal width of the face image. In order to prevent the omission of facial information while reducing the noise of background, we enlarge the rectangular outlines by 1.05 times to cover more facial content. Furthermore, considering that the size of images input into the learning model is preset to 48 × 48, the detected images will be rotated with nose tip as the center and resized appropriately to make it consistent with the input size.Figure 4
Automatic output of IntraFace to measure audience reaction [55].


### 2.3. The Learning Model Based on CNN

The architecture of the applied deep learning model based on CNN is illustrated in Figure5, which referred to the research results proposed by Kuo et al. [27], and the prior performance of this model in FER over the other similar has also been proven. After a convolutional layer of 32 feature maps, the input layer is followed by 2 blocks, which consists of 2 convolutional layers and 1 max-pooling layer with 64 feature maps separately. And the size of kernels in the first convolutional layer is set to 3 × 3, the second is 5 × 5, the max-pooling layers both consist of a kernel of size 2 × 2 and stride 2, and as a consequence, the input image will be compressed to a quarter. And there are 2 following fully connected layers of 2048 and 1024 neurons, respectively, in which Rectified Linear Units (ReLUs) [56–59] are adopted as the activation function. In order to prevent overfitting, a Dropout is added after each of the 2 fully connected layers, which will release a part of neurons according to the presetting drop-probability; in this paper, the 2 values are both set to 0.5. The following output layer is composed of 8 units, and softmax [60] is adopted as the activation function to classify the expressions examined in terms of anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral.Figure 5
The structure of the applied CNN model.The proposed model was trained by databases Jaffe, CK+, and FER2013 that include the above 8 basic expressions at the same time. Because the small FER databases usually only contain a few hundreds of images, obviously this is not enough for model training; we adopt online augmentation strategy with both horizontal flipping and random shifting to increase the images of training sets. More details about the CNN model are given in Table1.Table 1
The parameters of the applied CNN model.
LayersKernel numberKernel sizeStepsImage sizeInput00None48 × 48 × 1Convolutional 0321 × 1148 × 48 × 32Convolutional 1-1643 × 3148 × 48 × 64Convolutional 1-2645 × 5148 × 48 × 64Pooling 102 × 2224 × 24 × 64Convolutional 2-1643 × 3124 × 24 × 64Convolutional 2-2645 × 5124 × 24 × 64Pooling 202 × 2212 × 12 × 64Fully connected 1Dropout = 0.51 × 1 × 2048Fully connected 2Dropout = 0.51 × 1 × 1024Output00None1 × 1 × 8And in this model, the output sizeN of each convolutional layer can be formulated as(1)N=I−F+2PS+1,where I,F,P, and S denote the input size, kernel size, padding size, and stride size, respectively.In each max-pooling layer, the padding size is 0, and the output size can also be expressed as(2)N=I−FS+1.Rectified Linear Units (ReLUs) are adopted as the activation function in the convolutional layers and max-pooling layers to avoid gradient explosion and ensure faster convergence speed during the back-propagation operation, which can be formulated as(3)fx=0,x≤0;x,x>0.Softmax is used as the activation function in the output layer, the input of which is the matrixzi=wiTx+bi output from the fully connected layers. The formulation is given as(4)Si=ezi∑k=1Kezk,where K represents the output dimensions of the layer, meaning there are K kinds of results, and Si represents the probability of result i,i=1,2,…,K.And Softmax loss, which is used for gradient derivation and update, can be calculated by the following:(5)L=−∑i=1KyilogSi,∂L∂zk=∂L∂Si∂Si∂zk,∂L∂Si=∑i=1K−yiSi,∂Si∂zk=ezi∑k=1Kezk−eziezk∑k=1Kezk2=Si1−Sk,i=k;ezi·ezk∑k=1Kezk2=−SiSk,i≠k,∂L∂zk=∂L∂Si∂Si∂zk=−∑i=1KyiSi·∂Si∂zk=−Sk1−Sk⋅ykSk+∑i≠kK−yiSi⋅−SiSk=−yk+ykSk+∑i≠kKyiSk=Sk−yk,where L denotes the loss function and yi is the label variable, the value of which is 1 or 0 according to whether the output is consistent with the actual value.


## 2.1. Online Education Platforms

The advances in technological delivery modalities have spawned a large number of online education platforms and greatly improved the flexibility of education, enabling teachers to adopt diverse technical methods to assist teaching without worrying about the limit on the number of students in traditional classroom-based course, and students in different regions can communicate in real time without having to consider the traffic and other issues. The same teaching materials as traditional classes can be uploaded to these platforms for students’ reference. Currently, in platforms that have online teaching functions, such as DingTalk, Zoom, and Rain classroom, teachers can adopt the method of video meeting and take advantage of the camera built in devices to capture and recognize students’ facial expressions in real time. The captured images will be preprocessed and then used as the input of CNN.


## 2.2. The Preprocessing Based on IntraFace

Effective preprocessing can reduce the interference of face-like objects in the background when detecting faces in an image and then standardize the face images according to the heuristic knowledge, which will effectively improve the efficiency of the deep learning model. We employed IntraFace [55], a publicly available software package that integrates algorithms for facial feature tracking, head pose estimation, facial attribute detection, etc., as the tool of preprocessing. As shown in Figure 4, IntraFace can also be used to detect multiple faces at the same time. The key features of each face including eyebrows, eyes, nose tip, and mouth can be recognized effectively, and the facial expression can be detected by rectangular outlines accordingly; these outlines are constructed by the feature points at the edge of every face, including the uppermost and the lowermost, which determined the vertical width, and the rightmost and the leftmost, which determined the horizontal width of the face image. In order to prevent the omission of facial information while reducing the noise of background, we enlarge the rectangular outlines by 1.05 times to cover more facial content. Furthermore, considering that the size of images input into the learning model is preset to 48 × 48, the detected images will be rotated with nose tip as the center and resized appropriately to make it consistent with the input size.Figure 4
Automatic output of IntraFace to measure audience reaction [55].


## 2.3. The Learning Model Based on CNN

The architecture of the applied deep learning model based on CNN is illustrated in Figure5, which referred to the research results proposed by Kuo et al. [27], and the prior performance of this model in FER over the other similar has also been proven. After a convolutional layer of 32 feature maps, the input layer is followed by 2 blocks, which consists of 2 convolutional layers and 1 max-pooling layer with 64 feature maps separately. And the size of kernels in the first convolutional layer is set to 3 × 3, the second is 5 × 5, the max-pooling layers both consist of a kernel of size 2 × 2 and stride 2, and as a consequence, the input image will be compressed to a quarter. And there are 2 following fully connected layers of 2048 and 1024 neurons, respectively, in which Rectified Linear Units (ReLUs) [56–59] are adopted as the activation function. In order to prevent overfitting, a Dropout is added after each of the 2 fully connected layers, which will release a part of neurons according to the presetting drop-probability; in this paper, the 2 values are both set to 0.5. The following output layer is composed of 8 units, and softmax [60] is adopted as the activation function to classify the expressions examined in terms of anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral.Figure 5
The structure of the applied CNN model.The proposed model was trained by databases Jaffe, CK+, and FER2013 that include the above 8 basic expressions at the same time. Because the small FER databases usually only contain a few hundreds of images, obviously this is not enough for model training; we adopt online augmentation strategy with both horizontal flipping and random shifting to increase the images of training sets. More details about the CNN model are given in Table1.Table 1
The parameters of the applied CNN model.
LayersKernel numberKernel sizeStepsImage sizeInput00None48 × 48 × 1Convolutional 0321 × 1148 × 48 × 32Convolutional 1-1643 × 3148 × 48 × 64Convolutional 1-2645 × 5148 × 48 × 64Pooling 102 × 2224 × 24 × 64Convolutional 2-1643 × 3124 × 24 × 64Convolutional 2-2645 × 5124 × 24 × 64Pooling 202 × 2212 × 12 × 64Fully connected 1Dropout = 0.51 × 1 × 2048Fully connected 2Dropout = 0.51 × 1 × 1024Output00None1 × 1 × 8And in this model, the output sizeN of each convolutional layer can be formulated as(1)N=I−F+2PS+1,where I,F,P, and S denote the input size, kernel size, padding size, and stride size, respectively.In each max-pooling layer, the padding size is 0, and the output size can also be expressed as(2)N=I−FS+1.Rectified Linear Units (ReLUs) are adopted as the activation function in the convolutional layers and max-pooling layers to avoid gradient explosion and ensure faster convergence speed during the back-propagation operation, which can be formulated as(3)fx=0,x≤0;x,x>0.Softmax is used as the activation function in the output layer, the input of which is the matrixzi=wiTx+bi output from the fully connected layers. The formulation is given as(4)Si=ezi∑k=1Kezk,where K represents the output dimensions of the layer, meaning there are K kinds of results, and Si represents the probability of result i,i=1,2,…,K.And Softmax loss, which is used for gradient derivation and update, can be calculated by the following:(5)L=−∑i=1KyilogSi,∂L∂zk=∂L∂Si∂Si∂zk,∂L∂Si=∑i=1K−yiSi,∂Si∂zk=ezi∑k=1Kezk−eziezk∑k=1Kezk2=Si1−Sk,i=k;ezi·ezk∑k=1Kezk2=−SiSk,i≠k,∂L∂zk=∂L∂Si∂Si∂zk=−∑i=1KyiSi·∂Si∂zk=−Sk1−Sk⋅ykSk+∑i≠kK−yiSi⋅−SiSk=−yk+ykSk+∑i≠kKyiSk=Sk−yk,where L denotes the loss function and yi is the label variable, the value of which is 1 or 0 according to whether the output is consistent with the actual value.


## 3. Experiment and Results

In order to test the performance of the proposed framework in practical applications, we captured an image that includes 27 people from an online meeting held on Tencent Meeting and input it into the CNN model. This image is taken before the end of the meeting; the moderator was making a concluding speech in a pleasant atmosphere. In addition, everyone was told that the meeting was coming to an end, according to the experiment conducted by Tonguç and Ozkara [61], students’ happiness will be significantly improved within a few minutes before the end of a lecture, so under the similar circumstance, it can be inferred that the emotions presented by most of the faces in this image are happy or neutral. Figure 6 shows the input (left) and output (right) images of the CNN model. It can be seen clearly from the result that all the faces were recognized and marked by the rectangular outlines, and the responding facial expressions were also labeled. In the total 27 faces, 10 were labeled “happy,” 15 were labeled “neutral,” and 2 were labeled “sad,” noting that the 2nd image in the last line and the 3rd image in the 4th line from last, marked by red outlines, were not detected by the outlines precisely; the reason may be that the 2 face images are so incomplete that the features presented are too insufficient to recognize. Figure 7 shows the probability distribution histogram of emotions, from which we can observe the overall emotions intuitively and judge the emotional state of class accordingly. It is worth noting that the probability of happiness is significantly higher than that of neutral in this figure, while the faces labeled “happy” are less than “neutral” as exhibited in Figure 6. The difference can be explained as follows: there may be features of multiple expressions on a face at the same time; the expression presented on this face will be labeled according to the most likely expression decided by these features, but the overall expression of an image including multiple faces is decided by the sum of various expression features contained in each face. In some faces that are marked as “happy,” the probability of happiness may be much higher than neutral, while in some faces that are marked as “neutral,” the probability of happiness may be only slightly lower than neutral. Overall, the result of this experiment can provide favorable support for the performance of the model when applied to real environment.Figure 6
The input (a) and output (b) images of the FER model.Figure 7
The probability distribution of emotions.


## 4. Conclusion and Discussion

In this study, by combining the online courses platforms and a compact deep learning model based on the architecture of CNN, we construct a framework to analyze students’ emotions according to their facial expressions from the perspective of computer simulation. The overall result can be presented in a histogram intuitively, and teachers can adjust their teaching strategies accordingly to improve the efficiency of online teaching.With reference to the studies of Ekman et al. and FER2013, the emotions were classified into anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral in the proposed framework. To verify the applicability of this framework in a real environment, we captured an image including the facial images of all participants at one time in a real online meeting; there were 12 participants in this meeting, and the captured time was determined at the end of meeting. A total of 12 faces were captured, of which 11 were effectively recognizable faces that contain enough feature points. By inputting this image into the applied CNN model, we obtained the emotional tags for each valid face and got the overall emotion at that time. It has been proved that the framework has good applicability in practical activities and plays a positive role in solving the problems, such as the lack of binding force on students, and teachers cannot get timely feedback. Ultimately, it will contribute to improving the quality of online education.Despite the above benefits, there is still much room for improvement in this framework and its applications. From the perspective of technologies, with the development of computer simulation, algorithms with better performance and shorter operation time, including preprocessing and deep learning models, will be continuously developed over time. For instance, the image preprocessing contains face detection, alignment, rotation, and resize, but when facing problems, such as backlight, shadows, and facial incompleteness, caused by complex environments, these current methods are always powerless, and these shortcomings may be solved in the future. What is more, although the CNN model in the proposed framework currently performs well, it will be replaced by models with higher learning capabilities and higher classification accuracy in the future. In order to ensure the competitiveness of the framework in a longer period, it needs to be adjusted and maintained regularly, and more advanced algorithms and technologies should be adopted to update it.In addition, with a large number of participants in the online courses, we have no way to ensure that everyone keeps the high level of concentration, and then students’ expressions may not fully represent their emotions due to these subjective factors. Taking measures like setting thresholds can filter out some invalid information and highlight the main emotions in the image. Finally, the teaching efficiency can be improved as a result.



---

*Source: 4065207-2020-09-11.xml*