# Game Difficulty Prediction Based on Facial Cues and Game Performance
## 3 Methods
### Dataset Collection  
We use facial expressions, gaze directions, head poses, and game performance to predict game difficulty. Among them, facial expressions can mirror the user's inner emotions to a certain extent, gaze directions and head poses are related to the user's attention, and game performance reflects proficiency. Therefore, by combining these data from various modalities, we expect to predict the perceived game difficulty experienced by users during interaction.  
Figure 1: Game interface.  
Figure 2: Experimental procedure.  
We use OpenFace [32], a prominent facial behavior analysis toolkit that has been employed in many studies [11; 33; 34], to obtain gaze directions and head poses. The gaze directions include pitch and yaw, and the head poses include pitch, yaw, and roll. Then, ResMaskingNET [35], which has achieved promising performance on several publicly available facial expression datasets, is utilized to detect facial expressions. Facial expressions include happy, sad, angry, surprise, fear, disgust, and neutral. Figure 3 shows the facial cues obtained via OpenFace 2.0 (available at [[https://github.com/[PERSON]/OpenFace](https://github.com/[PERSON]/OpenFace)]([https://github.com/[PERSON]/OpenFace](https://github.com/[PERSON]/OpenFace)) Last accessed: 10 September 2024) and ResMaskingNET (available at [[https://github.com/phamquiluan/ResidualMaskingNetwork](https://github.com/phamquiluan/ResidualMaskingNetwork)]([https://github.com/phamquiluan/ResidualMaskingNetwork](https://github.com/phamquiluan/ResidualMaskingNetwork)) Last accessed: 10 September 2024). Participants' facial expressions, gaze directions, and head poses in each frame during gaming were saved to a local Excel file for subsequent processing.  
After completing each game mode, we recorded the player's game performance, including the score and the number of failures. Then, we combined the averages of their facial cues and game performance as the final input features. Thus, we obtained 14-dimensional features (facial expression: 7, gaze direction: 2, head pose: 3, game performance: 2). Finally, we built a game difficulty prediction dataset with these features as inputs and the users' subjective difficulty rating scores as labels. Figure 4 shows an example of generating a dataset sample from a test video containing n-frame images. The first column shows the facial cues computed in each frame, the second column displays the average results of each facial feature and the game performance, and the last column shows the subjective difficulty rating and the corresponding difficulty category. The mapping rules between the subjective difficulty rating and the difficulty category will be described in detail in Section 3.5.  
Since we integrated the facial data processing module into the game, there is no desynchronization problem between the acquired facial cues, as in physiological signal-based methods.  
Figure 3: Facial cues obtained via OpenFace and ResMaskingNET. Here, happiness indicates the estimated expression category, the blue cube is the head pose direction, the red dots are facial landmarks, and the green lines denote the gaze directions.