Emotion Recognition in HMDs: A Multi-task Approach Using Physiological Signals and Occluded Faces

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 17:06:17

Abstract: Prior research on emotion recognition in extended reality (XR) has faced challenges due to the occlusion of facial expressions by Head-Mounted Displays (HMDs). This limitation hinders accurate Facial Expression Recognition (FER), which is crucial for immersive user experiences. This study aims to overcome the occlusion challenge by integrating physiological signals with partially visible facial expressions to enhance emotion recognition in XR environments. We employed a multi-task approach, utilizing a feature-level fusion to fuse Electroencephalography (EEG) and Galvanic Skin Response (GSR) signals with occluded facial expressions. The model predicts valence and arousal simultaneously from both macro-and micro-expression. Our method demonstrated improved accuracy in emotion recognition under partial occlusion conditions. The integration of temporal physiological signals with other modalities significantly enhanced performance, particularly for half-face emotion recognition. The study presents a novel approach to emotion recognition in XR, addressing the limitations of facial occlusion by HMDs. The findings suggest that physiological signals are vital for interpreting emotions in occluded scenarios, offering potential for real-time applications and advancing social XR applications.

Main paper: 
 introduction: 
 Head-mounted displays (HMDs) are essential for experiencing extended reality (XR), which includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). In this context, people's emotions are naturally and immediately stimulated due to the high immersion and realism of XR. For instance, Virtual Reality Therapy studies human emotional reactions to explore treatments for mental health disorders such as phobic disorders [18] , obsessivecompulsive disorder [41] , and eating disorders [6] , or for social cognition training [31] . Moreover, VR content is adapted to the user's emotional state for relaxation in the virtual world [14] . However, HMDs pose a challenge to external recording techniques as they cover the user's upper face [11] (see Figure 1 (b) ). This limitation greatly affects social XR applications, especially Facial Expression Recognition (FER), a key method for studying and differentiating human emotions. In these applications, facial features play a critical role in creating an immersive user experience [5] . The accuracy of FER decreases if the upper face information is discarded [5] . Another difficulty in FER in an immersive XR context is that conventional FER methods rely on public databases, which contain complete facial information. To adapt these datasets to the XR context, some studies have made adjustments, such as placing a "VR patch" on a detected face [15, 33] and processing images of people wearing VR headsets into their corresponding Grad-CAM [37] explanation masks [10] .Apart from the issue of HMDs obscuring the face, the brief duration of facial expressions, whether they are facial macro-expressions that are voluntary muscle movements covering a large area of the face and lasting between 0.5 and 4 seconds [9] , or facial microexpressions that are brief, involuntary facial changes like nose wrinkling over a short time frame of between 65 and 500 ms [45] , and their subtle movements make them difficult for humans to identify [36] . Human facial expressions may be a mix of facial microexpressions and macro-expressions. For example, both the surprise from micro-expression and macro-expression involve raised eyebrows and opened eyes [44] . Micro-expressions are almost impossible to fake [39] . On the contrary, macro-expressions may not convey hidden emotions that determine true human feelings and state-of-mind [44] , and can be easily controlled and manipulated.Physiological responses, which are difficult to fake, provide a deeper understanding of underlying emotions [36] . These responses originate from the central (brain stem) and autonomic nervous systems, which regulate body functions like heart rate, respiration, blood pressure, swallowing, and pupillary movement [22] . Electroencephalography (EEG) can effectively reflect the electrical activity of the central nervous system and is associated with emotion [1, 25] . EEG is commonly used in emotional recognition studies in immersive virtual environments [47] . Peripheral physiological signals such as Galvanic Skin Response (GSR), photoplethysmogram (PPG), or Heart Rate Variability (HRV) can also be used to reliably measure emotional state and have been widely used in emotional engagement studies in immersive virtual environments [3, 16] . Although physiological signals cannot be deliberately controlled or concealed [43] , they are suitable for real-time emotion recognition [27] . However, these signals can be very weak and easily contaminated by noise like artifacts [40] .Thus, recognizing emotions using only physiological signals or incomplete facial expressions can be quite challenging.A few studies have used incomplete facial expressions and physiological signals instead of only incomplete facial expressions [10] , but this area still requires further research and exploration. This paper combines various modalities (incomplete facial expressions and physiological signals) to address the limitations of each individual modality.The contributions of this paper include:  
related work 2.1 discrete and dimensional emotion model: 
 Emotion models are primarily divided into two categories: discrete and dimensional [13] . The discrete model includes six universally recognized emotions: anger, happiness, fear, surprise, disgust, and sadness [7, 8] . However, Saffaryazdi et al. [36] argue that categorizing emotional states into discrete emotions is incorrect as human emotions are often a blend of several feelings. For instance, reported happiness may be a mix of positive feelings. In situations of positive and negative smiles, brain patterns and physiological signals differ, leading to misclassification when grouped into a single class.The dimensional model, in contrast, views emotions as a combination of three psychological dimensions: arousal, valence, and either dominance or intensity [36] . Russel's Circumplex Model, proposed by Posner et al. [34] , uses only valence and arousal to represent emotions, with valence indicating a range from negative to positive emotions, and arousal representing a spectrum from passive to active emotions. Most neurophysiological emotion recognition research [32] and benchmark datasets use the Circumplex Model for assessment. Our study uses the Circumplex Model to evaluate our method of detecting emotions using neurophysiological cues and facial expressions on two multimodal datasets.Multimodal datasets for emotion recognition have garnered significant interest among researchers [36] . A few such datasets, including facial video, EEG, and physiological signals, are available. The AMI-GOS [21] and DEAP [30] datasets are popular for macro-expression and micro-expression studies, respectively. Facial macro-expression dataset. The AMIGOS dataset [21] includes neuro-physiological signals (EEG, ECG, GSR), frontal HD video, and full-body RGB and depth videos from 37 participants. It uses a variety of videos to elicit affective responses in individual and group settings. The EEG data, collected using a state-of-the-art device, along with self-assessment manikins (SAM) [2] , provide a comprehensive study of affective responses.Facial micro-expression dataset. The DEAP dataset [30] contains neuro-physiological signals (EEG, peripheral physiological signals) from 22 participants watching music video excerpts. It uses a wide range of music videos to induce affective responses. The EEG data, collected using a cutting-edge device, and participant ratings on arousal, valence, like/dislike, dominance, and familiarity, provide an extensive analysis of affective responses.Relationship between modalities. Some studies have explored the relationship between behavioral responses and physiological changes in multimodal emotion recognition. Saffaryazdi [36] suggested using facial micro-expressions in conjunction with brain Fusion between modalities. Some methods fuse modalities at the feature level. Sun et al. [38] extracted discrete features from functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and simply concatenated these discrete features from both modalities for fusion. Zhang et al. [48] used a cross-attention mechanism to align the relationship between PPG and GSR modalities, highlighted important information within a single modality using a self-attention mechanism, and finally used a predictor to perform stress recognition on the representations of both modalities. Pan et al. [32] proposed an online emotion recognition method based on multimodal physiological signals (such as ECG, EEG, GSR, etc.), using hypergraph learning techniques to fuse multimodal information and capture complex and nonlinear relationships between data. Other methods fuse modalities at the decision level. Saffaryazdi et al. [36] fed preprocessed sequences into two 3D CNN models to extract features for classifying arousal and valence separately, used a Dense layer with Adam optimizer [20] to classify the physiological data for arousal and valence labels separately, and fused the classification results at the decision level.Emotion recognition on occluded faces. Current half-face emotion recognition is mainly divided into two categories. One uses the half-face as input for emotion modeling, which requires a simple network structure and has low training time overhead. However, due to the absence of important information from the upper face, such as eyes, eyebrows, and nose [5] , the recognition accuracy is lower [10, 15] . The other category uses additional devices to supplement the occluded facial information, which helps improve recognition accuracy [5] . However, additional devices, such as cameras, may disrupt immersion and increase head load. Table 1 presents the related work on Uni/Multi-modal Full/Half Face Micro-/Macroexpression Recognition.Face reconstruction on occluded faces. Some work uses methods like generative adversarial network (GAN) to "repair" the occluded part. For example, Zhao et al. [50] collected pre-recorded sequences to reconstruct a 3D head, and Wang et al. [42] used GAN to repair incomplete facial information. These works may help achieve higher emotion recognition accuracy. However, these methods may not meet the real-time or low-latency requirements of XR (Mixed Reality) [24] .Single-task learning vs multi-task learning. Emotion recognition tasks for HMD XR applications may require real-time processing of multiple tasks, such as simultaneously predicting valence, arousal, etc. [4, 35] . Compared to training models for each task separately, building a multi-task learning model to simultaneously predict valence and arousal is more efficient. For example, Priyasad et al. [35] showed that multi-task learning can achieve fewer parameters, faster convergence, and higher performance for emotion recognition with EEG signals. Similarly, the multi-modal multi-task (M3T) approach [49] fused both visual features from facial videos and acoustic features from the audio tracks to estimate the valence and arousal simultaneously. AMIGOS Dataset. Out of 40 participants, 37 (excluding participants 8, 24, and 28) took part in both short and long video experiments designed to evoke emotional responses. Their facial videos, EEG, and peripheral physiological signals were fully recorded. 
methodology 3.1 data processing module: 
 DEAP dataset. For the first 22 out of 32 participants, complete facial videos, EEG, and peripheral physiological signals were preserved. However, due to technical issues, some of the last trials for participants 3, 5, 11, and 14 are missing.Segmentation based on annotations. In the AMIGOS dataset, videos capturing participants' faces were divided into multiple 20second segments. The first 5 seconds before the stimuli presentation were extracted as the initial clip. Subsequently, starting from the 5second mark (when the stimuli began), non-overlapping 20-second segments were extracted, with the number of segments depending on the duration of the stimuli video. The final clip included the last segment of the video, which was less than 20 seconds. To ensure uniform segment length, the first and last segments shorter than 20 seconds were discarded. Three annotators provided external annotations of valence and arousal for the 20-second segments of the face videos in both experiments. In the DEAP dataset, each face video lasted approximately 60 seconds and was not segmented due to only one annotation within that duration. Apex frame finding.We used an open-source facial toolkit 1 to obtain 68 facial landmarks (red points) in each frame of the facial expression sequence (See Figure 2(a) ). However, due to factors like poor video quality and dim lighting, faces were not detected in some videos, with successful detection in approximately 70% of the videos. Following previous research, we defined 10 regions on the face based on the detected landmarks (yellow boxes), representing areas where muscle movements frequently occur. The size of each region was estimated to be half the width of the mouth. These regions are depicted in Figure 2 (a). A facial video sequence starts with the Onset frame (when muscle movements occur in response to emotional stimuli) and ends with the Offset frame (when emotional reactions disappear and the face returns to a relaxed state). The Apex frame is the one with the most significant change in intensity within the sequence. To identify the apex frame, we calculated the absolute pixel differences between the current frame and the onset and offset frames in the ten regions. To minimize environmental noise, we normalized the sum of the differences by dividing it by the difference between the considered frame and its consecutive frame. We then obtained the per-pixel average value for each frame in the micro-expression (ME) sequence. The apex frame should indicate the peak of intensity differences with the onset and offset frames. Thus, we selected the frame with the highest per-pixel value of intensity differences as the apex frame:EQUATIONwhere𝑓 (Frame 𝑛 , Frame 𝑚 ) = |Frame 𝑛 -Frame 𝑚 | + 1 |Frame 𝑛 -Frame 𝑛-𝜖 | + 1 (2)Function (f ) measures the absolute pixel differences between two frames, normalized by the difference between the current frame and its third preceding frame to reduce noise. We calculate the absolute pixel differences in 10 region blocks generated by facial toolkit between the current frame and the onset frame, and the offset frame, respectively. To reduce the impact of environmental noise, divide the difference by the absolute pixel difference between the current frame and the continuous frame with a deviation of 𝜖. Sum the normalized pixel differences to represent the change 1 https://github.com/ageitgey/face recognition/ intensity value of the current frame, denoted as 𝑆 𝑖 . 𝑖 𝑚𝑎𝑥 identifies the apex frame showing the peak intensity of the facial expression. This frame is crucial for facial expression recognition tasks.Considering the two datasets have frame rates of 25fps and 50fps, we reduced any video with a frame rate above 25fps to 25fps for consistency (100 frames collected in 4s). We then selected 100 frames (4s) surrounding the peak frame as the region of interest (ROI). If the peak frame is at least 2s away from the video segment's edge, we included 2s (50 frames) on either side of it. But if the peak frame is less than 2s from any edge, we added frames from the opposite side until we had a total of 100 frames.After detecting facial key points, we align the face by rotating the image based on the eye key points [19] (See Figure 2(b) ). For example, we calculate the center coordinates of the left and right eyes ((𝑥 left , 𝑦 left ) and (𝑥 right , 𝑦 right )), and compute the angle 𝜃 between the line connecting the centers of the left and right eyes and the horizontal direction (arctan 𝑦 right -𝑦 left 𝑥 right -𝑥 left ). We then rotate the image counterclockwise by 𝜃 with the center coordinates of the two eyes as the base point.After the image is rotated, the landmark coordinates in the image also need to be rotated accordingly so that the landmark can match the rotated image. The rotation transformation can be represented as:EQUATIONwhere (𝑥 ′ , 𝑦 ′ ) are the coordinates after rotation, (𝑥, 𝑦) are the original coordinates, and (𝑥 center , 𝑦 center ) are the center coordinates of the two eyes.After aligning the face, we crop the face to a fixed size based on the landmark [19] . For instance, the vertical direction is divided into three parts: the middle part, which is the pixel distance from the center of the two eye landmarks to the center of the mouth landmark, accounts for 35% of the vertical direction of the cropped face image; the bottom part accounts for 35%; and the top part accounts for 30%. The horizontal direction is centered on the midpoint of the leftmost and rightmost landmarks. After cropping the face, the size becomes 320 × 320.We define a landmark transformation function. Since the image is cropped, the landmark coordinates need to be transformed again. The transformation can be represented as:EQUATIONwhere (𝑥 ′ , 𝑦 ′ ) are the coordinates after cropping, (𝑥, 𝑦) are the original coordinates, and (𝑥 crop , 𝑦 crop ) are the top-left coordinates of the cropping rectangle. After transforming the landmark coordinates, we obtain aligned face images.In an extended reality setting, the upper face is partially occluded by a Head-Mounted Display (HMD). Since there are no standard occlusion face image databases containing persons wearing HMDs, we established XR-occluded images by masking the upper region on the standard facial micro-and macro-expression images, based on the "VR patch" method [15] . We simulated facial occlusion based on the 68 landmarks detected in the apex frame. We initialized XR dimensions to an aspect ratio of approximately 2:1, based on the Hololens 2 headset. To uniformly scale the XR patch on the training images, we used the distance between the two temporal bones of the facial landmarks as a reference length. We then generated the polygonal occluding patch by setting the midpoint of the line passing through the eye center points as the center coordinate of the XR headset. To account for face rotations, we aligned the resized patch with the axis running through the eye centers. We obtained the angle of incline by determining the inverse tangent function of changes in y-coordinates to changes in x-coordinates of eye center points, and then we utilized the rotation matrix to rotate corner points of the blocking patch about its central pivot point on the coordinate plane accordingly. This geometric model provides a more realistic occlusion resulting from wearing an XR headset, rather than simply covering the upper half of the face, as done in [15] . The same operation was performed for both datasets.3.1.6 EEG and peripheral physiological signals. Downsampling and Filtering. The physiological data in two datasets were downsampled to 128Hz. A bandpass frequency filter from 4.0-45.0Hz was applied for EEG. GSR was calculated and low-pass filtered with a 60Hz cut-off frequency in the AMIGOS dataset. In the DEAP dataset, no filter was processed for GSR.Region of Interest Finding. In two datasets, the physiological signal's ROI corresponds to the ROI in the related face video, i.e., we take the physiological signal within 4s adjacent to the apex frame.AMIGOS Dataset. We relabeled the valence and arousal scores greater than 0 as high, and scores less than 0 as low. The total number of low and high classes for all participants' trials for valence were 1784 and 510, and for arousal, 1980 and 314 in the short video experiment, and the total number of low and high classes for all participants' trials for valence were 3097 and 949, and for arousal, 3312 and 734 in the long video experiment. As can be seen, both datasets were not balanced among classes. To solve the long tail effect, we finally selected 1706 samples that could satisfy the total number of low and high classes for all participants' trials for valence and arousal were equal. DEAP Dataset. We relabeled the valence and arousal scores greater than 5 as high, and scores less than or equal to 5 as low. The total number of low and high classes for all participants' trials for valence were 409 and 471, and for arousal, 367 and 513 in the experiment. As can be seen, both datasets were not balanced among classes. To solve the long tail effect, we finally selected 680 samples that could satisfy the total number of low and high classes for all participants' trials for valence and arousal were equal.As depicted in Figure 3 , we used a 3D VGG-like backbone to extract spatial-temporal features for every frame from 100 input frames (size: 320 × 320) around each apex frame. The 3D VGG-like backbone consists of five layers. It takes the 100 frames from the ROI as input and outputs 𝐹 ℎ𝑎𝑙 𝑓 -𝑓 𝑎𝑐𝑒 ∈ R 61×128 .To enhance emotion recognition operations for affective computing tasks applied to physiological signals, the multi-modal Emo-tionNet model has been introduced [26] (see Figure 3 ). This model is designed to capture both the heterogeneity and interactivity between EEG and GSR signals. It features a dual-stream transformer structure and an Interactivity-based Modal Fusion (IMF) module.The dual-stream transformer structure comprises two convolutional blocks (EEG Filter and GSR Filter) followed by transformer encoders (TEs). These TEs employ a multi-head attention mechanism to capture dependencies between elements of the input feature maps. The mechanism is defined as: Multi-Head(𝑄, 𝐾, 𝑉 ) = Concat(head 1 , . . . , head ℎ ) (5) where 𝑄, 𝐾, and 𝑉 are the query, key, and value matrices, and hh is the number of heads. Each head is computed as:EQUATIONwhere 𝑊 𝑖 represent the weights matrices of 𝑄, 𝐾, and 𝑉 , respectively. The output is then concatenated and projected back to the original dimensionality.The Interactivity-based Modal Fusion Module fuses features from both modalities based on interactivity scores. It is defined as:EQUATIONwhere 𝑋 1 and 𝑋 2 are the input features from different modalities, and ⊕ denotes element-wise summation. The Interactivity Extractor (IE) generates interactivity scores from the concatenated features of different modalities. The interactivity extraction module achieves the following:IE 𝑂𝑢𝑡𝑝𝑢𝑡 = ReLU(FC 2 (ReLU(FC 1 (IE 𝑖𝑛𝑝𝑢𝑡 )))) ⊙ IE 𝑖𝑛𝑝𝑢𝑡 (8)where FC 1 and FC 2 are fully connected layers, ReLU is the rectified linear unit activation function, and ⊙ denotes the Hadamard product (element-wise multiplication) between the interactivity score and the input feature 𝐼𝐸 𝑖𝑛𝑝𝑢𝑡 .The result obtained after merging EEG and GSR is 𝐹 𝑝ℎ𝑦𝑠𝑖𝑜 ∈ R 61×128 . We concatenate the half-face features 𝐹 ℎ𝑎𝑙 𝑓 -𝑓 𝑎𝑐𝑒 and the physiological features 𝐹 𝑝ℎ𝑦𝑠𝑖𝑜 . The concatenated output is then sent to two classification layers, which predict the emotional dimensions of valence and arousal (See Figure 3 ).In multitask learning, we simultaneously predict the binary classification problem of valence and arousal. For each task, we use the cross-entropy loss function to measure the difference between the model's predictions and the true labels. Specifically, the loss functions 𝐿 𝑉 and 𝐿 𝐴 for valence and arousal can be defined as follows:𝐿 𝑉 = - 1 𝑁 𝑁 ∑︁ 𝑖=1 𝑦 𝑉 𝑖 log(𝑝 𝑉 𝑖 ) + (1 -𝑦 𝑉 𝑖 ) log(1 -𝑝 𝑉 𝑖 ) 𝐿 𝐴 = - 1 𝑁 𝑁 ∑︁ 𝑖=1 𝑦 𝐴 𝑖 log(𝑝 𝐴 𝑖 ) + (1 -𝑦 𝐴 𝑖 ) log(1 -𝑝 𝐴 𝑖 ) (9)where 𝑁 is the number of samples, 𝑦 𝑉 𝑖 and 𝑦 𝐴 𝑖 are the true labels of the 𝑖-th sample for valence and arousal, respectively, and 𝑝 𝑉 𝑖 and 𝑝 𝐴 𝑖 are the predicted probabilities of the 𝑖-th sample being in the positive class for valence and arousal, respectively.During training, the model tries to minimize the total loss 𝐿 = 𝐿 𝑉 + 𝐿 𝐴 , which is the sum of the cross-entropy losses for each task. Note that this is a simple example, and in a real-world scenario, you might want to consider weighting the losses based on the importance of each task.To evaluate the effectiveness of our method, we conducted a realworld test set by inviting 10 participants complete experiments for the facial expression recognition task under partial occlusion from XR headsets. This section outlines the test set creation process. The test set creation process, shown in Figure 4 , includes six components divided into hardware and software categories, and two questions from the SAM questionnaire [2] on valence and arousal.The BrainCo Oxyzen portable EEG device2 records EEG data at a sampling rate of 256Hz. It's designed based on the voltage difference between two forehead electrodes and weighs about 57g.The Shimmer 3 GSR+ Unit Sensing device 3 aptures GSR signals at 128Hz and weighs about 30g.The Hololens 2 MR HMD4 displays the AR environment and allows interactions. It offers a resolution of 1440 × 936 per eye, a refresh rate of 60 FPS, and a FoV of 29°/43°(vertical/horizontal). It weighs about 566g and is compatible with the BrainCo Oxyzen EEG device, which we attach to the inside of the HoloLens lens. We use the camera of Lenovo Legion R7000P to capture the occluded face when wearing the HMD Hololens 2. The camera operates at a resolution of 1280 × 720 and a frame rate of 30 fps.The Python application for BrainCo Oxyzen streams and collects EEG data. The complete code is available in the official open-source GitHub repositories of BrainCo 5 .The Python application for Shimmer streams and collects GSR data. The full code will be open-sourced.Emotions and senses induced by 2D display may differ from those by the three-dimensional (3D) real world [17, 29, 46, 47] . Therefore, to verify the generalizability of our method, we use 3D video as mood induction procedures (MIPs).We invited 10 participants (age: 22-29, 2 females) to watch four 3D VR videos from DEO VR 6 each lasting about 3 minutes. While watching the videos, we asked the participants to avoid large head 5 https://github.com/BrainCoTech/oxyzen-example 6 https://deovr.com/ movements. At the same time, we used BrainCo Oxyzen and Shimmer 3 GSR+ Unit to synchronously collect the participants' EEG and GSR signals. We eventually obtained about 1 hours of data.We use two questions in the SAM questionnaire of valence and arousal, and ask participants to complete the questions to annotate their valence and arousal after watching the 3D videos. We finally obtained about 150 test set samples. 
experiment 4.1 dataset split: 
 In our experimental evaluation, we conduct tests on the AMIGOS (macro-expression) [30] and DEAP (micro-expression) [21] datasets. Following the data preprocessing guidance [21, 30] and a similar protocol to [36] , we split the dataset into training and validation sets in an 8:2 ratio Each sample includes consecutive frames around the apex frame, as well as EEG and GSR signals. We carefully establish the baselines by comparing our novel method with state-of-the-art methods on the facial expression recognition task under partial occlusion from XR headsets. Experiments are performed with the Pytorch framework on an NVIDIA 4090 GPU.Following the standard evaluation metric for facial expression recognition tasks, we employ three commonly used metrics, namely accuracy, F1-score, and Prediction time to measure the performance of our method for both valence prediction and arousal prediction. Additionally, we report the "time cost of prediction in the validation set" as the model may be applied to real-time emotion recognition of HMD in XR applications. Our method's accuracy, F1-score surpass the method [15] of facial expression under partial occlusion, which uses a single frame as input, by 5.69%, and 6.35% on AMIGOS dataset and 5.47%, and 10.98% on DEAP dataset, respectively, (see Table 2 ). Moreover, the inference time is suitable for real-time tasks. This is attributed to our method's superior handling of occluded face information and the incorporation of physiological signals. The continuous signal frames capture more temporal information, mitigating the impact of single frame noise or detail loss. Our approach not only enhances facial expression recognition performance under partial occlusion (FERPO) for XR HMDs but also reduces prediction time, making it ideal for real-time XR applications. Furthermore, our multi-task approach offers benefits over single-task methods, including fewer parameters, faster convergence, and improved performance. Additionally, Micro-expression datasets underperformed in tests compared to macro-expressions, likely due to their subtler facial changes. Thus, for HMD users, merging physiological signals with facial data enhances emotion recognition when micro-expressions occur.Using half-face images and physiological data for simultaneous valence and arousal prediction yields higher accuracy, improved F1score, and faster results than full-face images. This implies that the fusion of half-face and physiological signals achieves comparable emotion prediction accuracy to full-face images.In this subsection, we aim to conduct a detailed analysis of the contributions of visual and physiological modalities in our proposed approach. Table 2 shows that FERPO 𝑝ℎ𝑦𝑠𝑖𝑜 -𝑜𝑛𝑙 𝑦 performs significantly worse than both FERPO ℎ𝑎𝑙 𝑓 -𝑓 𝑎𝑐𝑒 -𝑜𝑛𝑙 𝑦 and FERPO 𝑓 𝑢𝑙𝑙 on the AMIGOS and DEAP datasets. This is because FERPO 𝑝ℎ𝑦𝑠𝑖𝑜 -𝑜𝑛𝑙 𝑦 only uses physiological data and ignores facial information that provides details about the subject's emotional expressions. As a result, when physiological signals alone may not accurately determine the emotional state, it leads to inaccurate valence and arousal outcomes. This highlights the importance of facial information in improving the accuracy of emotional state prediction. Furthermore, FERPO ℎ𝑎𝑙 𝑓 -𝑓 𝑎𝑐𝑒 -𝑜𝑛𝑙 𝑦 also performs significantly worse than FERPO 𝑓 𝑢𝑙𝑙 . This is because FERPO ℎ𝑎𝑙 𝑓 -𝑓 𝑎𝑐𝑒 -𝑜𝑛𝑙 𝑦 only uses halfface details and ignores the spatial relationship of the complete facial key points. Consequently, this greatly increases the difficulty of predicting positive emotions, negative emotions, and arousal levels, leading to more imprecise valence and arousal outcomes.This underlines the importance of supplementing additional physiological modalities for resolving the incomplete information problem caused by partial occlusion. The inclusion of physiological information greatly enhances the model's ability to accurately identify complex emotional states.The test set experiment demonstrates the effectiveness of our method in handling complex real-world scenarios where participants use actual HMDs, and has a extremely low inference time. This implies that our method may be able to fulfill the task of real-time emotion recognition under the use of HMDs, such as in VR Therapy [6, 18, 31, 41] , VR Relaxation [14] , AR education [12, 23] . 
limitations and future work: 
 The datasets we used, AMIGOS and DEAP, use non-immersive two-dimensional (2D) videos or images as mood induction procedures (MIPs), such as images, audios, and videos. However, the emotion and sense induced by 2D display are different from that by the three-dimensional (3D) real world; for example, the 2D display lacks the sense of presence and depth information [47] . Moreover, considering the known differences in EEG dynamics between 2D and 3D presentations [17] , there may be a gap between the experimental studies based on 2D display and real-world applications [17] . Therefore, we plan to introduce 3D extended reality (XR) MIPs, which enable researchers to simulate real-world conditions in a controlled laboratory environment, and are gaining popularity in studying emotion.Additionally, the facial images collected after the subject wears an HMD may be affected by interference factors such as lighting, head shaking or deflection. The facial detection algorithm currently used is sensitive to disturbances such as blurred images and weak lighting. In the future, we will explore more robust and stable facial detection algorithms. 
conclusion: 
 We've introduced a novel multi-task method for emotion recognition in XR, using physiological signals and occluded facial expressions. Our approach integrates EEG and GSR data with half-face expressions via a feature-level fusion, enabling simultaneous prediction of valence and arousal. Our approach notably enhances emotion recognition accuracy under partial occlusion. This improvement is evident in AMIGOS and DEAP datasets, as well as in real-world situations, and it's validated for use in emotion recognition tasks in HMD-based XR. Future enhancements will focus on robustness against disturbances like lighting and head movements, and the application of 3D XR mood induction procedures for a more immersive experience. Our work significantly advances emotion recognition in XR, setting the stage for more responsive virtual environments. 
 
 Appendix: https://www.brainco-hz.com/https://www.shimmersensing.com/https://www.microsoft.com/en-us/hololens/buy 


Decision: This paper proposes a novel method for emotion recognition in XR environments that addresses the challenge of facial occlusion caused by Head-Mounted Displays (HMDs). The method integrates physiological signals (EEG and GSR) with partially visible facial expressions using a multi-task approach with feature-level fusion. While all reviewers acknowledged the novelty of the approach and its potential significance for XR, they also raised several limitations that need to be addressed.

**Reviewer Consensus:**

* Strengths:
    * **Novelty:** All reviewers agreed that the combination of physiological signals and facial expressions for emotion recognition in XR is a novel approach.
    * **Improved Accuracy:** The proposed method achieves better accuracy compared to using only facial expressions, especially under occlusion conditions.
    * **Rigorous Testing:** The model is validated on established datasets (DEAP and AMIGOS) and demonstrates its effectiveness.
* Weaknesses:
    * **Limited Comparisons:** Several reviewers pointed out the lack of comparisons with state-of-the-art methods and the need for ablation studies to isolate the effectiveness of the multi-task approach.
    * **Missing Details:** Reviewers requested more details about the network architecture, training procedures, and data processing steps.
    * **Unclear Writing:** Minor grammatical errors and inconsistencies were identified.

Considering the overall reviewer sentiment and the paper's potential contribution to the field, I recommend accepting this submission.

Reviews: review: paper_summary: paper_summary: It is about the emotion recognition in XR. Since the HMD occlude people's faces, the tradition facial expression recognition methods are not applicable in this context. Thus, it proposes a multi-modal emotion recognition method which incorporates EEG and GSR.
strength_weakness: strength_weakness: It uses multi-modal and multi-task emotion recognition method to solve the occlusion problem in XR based facial expression recognition. The research view is novel.
limitations: limitations: It is better to perform more comprehensive benchmark test.
paper_summary: paper_summary: The paper titled "Emotion Recognition in HMDs: A Multi-task Approach Using Physiological Signals and Occluded Faces" addresses the challenge of recognizing emotions in Extended Reality (XR) environments, where Head-Mounted Displays (HMDs) partially occlude facial expressions, thereby hindering accurate Facial Expression Recognition (FER). The authors propose a multi-task model that integrates physiological signals (Electroencephalography (EEG) and Galvanic Skin Response (GSR)) with partially visible facial expressions to predict emotional states in terms of valence and arousal. By employing feature-level fusion and temporal analysis of physiological signals, the study demonstrates improved accuracy in emotion recognition under partial occlusion conditions. The model is validated on the DEAP and AMIGOS datasets, showing significant performance enhancements.
strength_weakness: strength_weakness: 1、The study introduces a novel multi-task model that effectively combines physiological signals with occluded facial expressions, addressing a critical challenge in XR environments.
2、The model is rigorously tested on established datasets (DEAP and AMIGOS) and real-world scenarios, demonstrating its robustness and applicability.
3、The integration of EEG and GSR signals with facial expressions significantly enhances the accuracy of emotion recognition, especially under conditions of partial facial occlusion.
4、The inclusion of temporal physiological signals in the model provides a deeper understanding of underlying emotions and improves recognition performance.
5、The research has practical implications for real-time applications in social XR, virtual therapy, and other immersive environments where understanding user emotions is crucial.
limitations: limitations: 1、The model's performance is highly dependent on the quality and variety of the datasets used for training. Real-world applicability might vary with different user demographics and contexts not covered by the DEAP and AMIGOS datasets.
2、The multi-task model and the integration of multiple signals increase the computational complexity, which may pose challenges for real-time processing and scalability in practical applications.
3、Physiological signals like EEG and GSR can be noisy and subject to individual differences, which might affect the model's generalizability across different users.
4、While the study focuses on partially occluded faces, it primarily considers the lower face. The model might not perform as well if other regions are occluded or if the occlusion pattern changes dynamically.
5、The feature-level fusion method, although effective, might not fully capture the complex interactions between different modalities. 
6、In the experimental section, the performance comparison only involves one method which is from 2020. It is suggested to incorporate comparisons with multiple state-of-the-art methods to establish the advancement of the proposed approach.
7、The tense of the article is confusing.
8、There are some punctuation errors in the text. For example, some double quotation marks in the text are Chinese symbols.
9、The title of Table 2 should be positioned above the table.
10、Some parts of the article are incorrectly expressed. For example, the function f below Equation 2 should be “Function f(.)” rather than “Function (f )”.
11、There are some grammatical errors in the text. For example, below equation 6 in Section 3.2.2 “where  represent the weights matrices of Q, K, and V, respectively.”, “represent” should be “represents”.
paper_summary: paper_summary: The authors have implemented a recognition method that adopts a fusion approach of occluded facial expressions and physiological signals, achieving recognition performance surpassing that of  full-face images. This study proposes a new method to recognize emotion in XR that addresses the limitations of face occlusion. The results of this study show that for the half-face emotion recognition task, the introduction of physiological signals can significantly improve the performance of the model.
strength_weakness: strength_weakness: The paper is novel in the field of emotion recognition in XR by realizing feature fusion of visual images and physiological signals. Adequate experiments were also conducted to verify the effectiveness and advancedness of the method. Furthermore, the paper constructs data from real scenarios and tests the effectiveness and performance of the proposed method on real-world situations, and the results perform well.
limitations: limitations: 1. The novelty of this work is not clear, and the authors must clarify their innovations. I suggest the authors rewrite the introduction.
2. On line 153 of the introduction section, if the recognition method combining occluded images and physiological signals is not proposed by the authors for the first time, it should be explicitly stated in the introduction.
3. Many details of the method are not provided, such as network details, training details, etc.
4. The experimental section lacks comparison with peer works, especially with benchmark works.
5. Additional ablation experiments are needed to separately demonstrate the effectiveness of the multi-task method compared to the single-task method.
paper_summary: paper_summary: In this study, the authors address the challenge of facial expression occlusion by HMDs in XR environments by integrating physiological signals (EEG and GSR) with partially visible facial expressions for improved emotion recognition. Using a multi-task approach featuring feature-level fusion, the proposed model enhances emotion detection under occlusion, demonstrating improvements in accuracy for half-face emotion recognition.
strength_weakness: strength_weakness: The field of study is in the scope of the ACMMM and is highly crucial in XR, and the manuscript is well-written and easy to follow. The proposed approach is novel enough for the publication.
limitations: limitations: There are some minor changes which are necessary to strengthen the claims and findings:


Have the authors explored the use of HMDs equipped with eye and gaze-tracking capabilities? It would be beneficial to discuss this. Additionally, the role of eye contact and the significance of eye features in FER should also be addressed.

1:
“We tested our multi-modal, multi-task method on DEAP, AMIGOS datasets, and real-world situations:” needs citations here.

The term “multi-task method” needs explanation in the Introduction, before summarizing the contributions. 

2.1: 
“The discrete model includes six universally recognized emotions”: eight classes of emotions also should be mentioned including neutral and contempt.

Please use abbreviations instead of the full form after the first appearance of each term throughout the text, like electroencephalography (EEG) in 2.3 or XR (Mixed Reality) in 2.4.

2.3: 
CNN: mention the full form. 

Table 1: needs more explanation and comparison in the text ( in 2.4).

3: 
The transition to the METHODOLOGY section is quite abrupt; please provide an introductory paragraph to smooth into this section.

Figure 2: The caption needs more explanation, not informative. 

3.1: 
The title may better be updated, Data “Processing” Module, do you mean preprocessing? 

3.1.2: 
“Following previous research”: not clear how you defined the yellow boxes.

Figure 2(c) and (d) should be discussed in the text. 

3.2.2: 
“where 𝑄, 𝐾, and 𝑉 are the query, key, and value matrices, and hh”: correct the typo.

Why R61×128?


Figure 4: caption needs more info. Also, discuss it accordingly in the text. 
 

Table 2: discuss clearly why the accuracy and F1 score in the D dataset are much lower compared to the A dataset in 4.4.1 ("Additionally, Micro-expression datasets underperformed in tests compared to macro-expressions, likely due to their subtler facial changes").  

3.5.3: “MIPs” need more explanation.

3.5.4: include more details of participants' demography. 

4.2: baselines and comparison models should be specifically defined here.