Artificial Intelligence for Evaluation of Emotions behind Face Masks

Widespread use of face masks has dramatically affected the ability to interpret emotional expression and communication. Wearing a mask that covers the lower face challenges the transmission of emotions, which can be misinterpreted. We evaluated the effect of face masks on facial emotion interpretation using artificial intelligence. In our study, 102 facial images that resemble real human faces (51 unmasked and 51 masked) were produced randomly by generative adversarial networks using the website https://thispersondoesnotexist.com, created by Karras et al. and Nvidia.1,2 Generative adversarial network is a type of machine-learning framework that produces new data from a fixed statistic used for training.1–3 We analyzed frontal view facial images of people 18 years of age or older. Low-quality images and those in which the face was covered were excluded. For each face image, a corresponding photograph including a face mask was created using Adobe Photoshop 2021. The masked and unmasked images were analyzed and compared using a validated facial expression recognition software package (FaceReader; Noldus Information Technology BV, Wageningen, the Netherlands).4 The software analyzes the proportion of each facial emotion (neutral, happy, sad, angry, surprised, scared, and disgusted), intensity of each facial action in units (0 = none, 4 = maximum intensity), age estimate, and sex of each image4,5 (Fig. 1).Fig. 1.: Facial action units using the facial emotion recognition software.Our results showed that covering the face with a mask leads to a significant loss of emotional information conveyed. Most of the unmasked images displayed a predominantly happy emotion (43.8%). With mask coverage, the happy emotions were misinterpreted as being neutral (60.9%) (P = 0.0008). This could be explained by the loss of action units associated with happy emotions, such as the lip corner puller and cheek raiser action units.6,7 The median age attributed to masked faces was lower compared with the unmasked faces [25 (range, 20 to 25) versus 30 (range, 25 to 35) years; P < 0.0001]. These images were classified in two groups, neutral or happy, according to the predominant emotion. For the happy group, wearing a mask resulted in decreased cheek raise [−1 (−2, 0) versus 0; P = 0.0002], lip corner puller [−3 (−3, −2) versus 0 (−1, 1); P < 0.0001], and lips apart [−4 (−5, −1) versus 0; P < 0.0001] and increased lid tightener [1 (0, 1) versus 0; P = 0.0024] and upper lip raiser [0 (Q1-90: 0, 0.2) versus 0 (−0.6, 0); P = 0.0470] compared with the neutral group (Table 1). After donning a mask, the happy group had higher cheek raiser [0 (Q1-90: 0, 1.2) versus 0 (0, 0.6); P = 0.0097] and lid tightener [1 (Q1-90: 0, 2.0) versus 0 (0, 2.0); P = 0.0018] compared with the neutral images (Fig. 2). This could be because the cheek raiser unit is activated by the orbicularis oculi in addition to the influence from the zygomaticus major. There was no difference in lip corner puller, lip tightener, or lip raiser between the two groups. Table 1. - Differences in Facial Emotion Interpretations before and after Masking Facial Action Unit Difference before and after Masking Facial Expression, Median (First, Third Quartile) Neutral Happy P L inner brow raiser 0 0 0.3560 R inner brow raiser 0 0 0.3560 L outer brow raiser 0 0 1.0000 R outer brow raiser 0 0 1.0000 L brow lowerer 0 0 0.1415 R brow lowerer 0 0 0.2215 L upper lid raiser 0 0 0.1298 R upper lid raiser 0 0 0.0648 L cheek raiser 0 −1 (−2, 0) 0.0002a R cheek raiser 0 −1 (−2, 0) <0.0001a L lid tightener 0 1 (0, 1) 0.0024a R lid tightener 0 1 (0, 1) 0.0011a L nose wrinkler 0 0 0.3560 R nose wrinkler 0 0 1.0000 Upper lip raiser 0 (Q1-90: −0.6, 0) 0 (Q1-90: 0, 0.2) 0.0470a L lip corner puller 0 (−1, 1) −3 (−3, −2) <0.0001a R lip corner puller 0 (−2, 1) −3 (−4, −2) <0.0001a L dimpler 0 0 0.1217 R dimpler 0 0 1.0000 L lip corner depressor 1 (0, 1) 1 (0, 2) 0.6890 R lip corner depressor 0 0 0.8359 Chin raiser 2 (2, 3) 3 (2, 3) 0.1059 L lip pucker 0 0 1.0000 R lip pucker 0 0 1.0000 L lip stretcher 0 0 1.0000 R lip stretcher 0 0 0.2786 L lip tightener 0 (0, 2) 0 (0, 2) 0.7765 L lip pressor 1 (0, 2) 1 (0, 2) 0.6869 L lips part 0 −4 (−5, −1) <0.0001a Jaw drop 0 0 0.0680 Mouth stretch 0 0 1.0000 aStatistically significant. Fig. 2.: Cheek raiser action unit before and after donning a face mask.Our study reveals an increased interpretation of sad and angry emotions on faces wearing masks, suggesting that faces may be perceived negatively when missing the positive input from the perioral action units. During mask use, the periorbital region becomes more crucial for emotional expression and social interactions. People may attempt to overcome their inability to express emotions though perioral subunits by exaggerating their periorbital movements. Alternative tools such as verbal communication and body language are useful to optimize communication. DISCLOSURE The authors have no financial interest to declare in relation to the content of this article.