# Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait
## 4 Experiment
### User Study  
The user study was conducted in three phases, each with 126 participants. The first phase focused on assessing image quality and lip syncing, while the second and third phases focused on assessing the emotional validity of the videos. The input video and audio were obtained from MEAD, and the input image was the first frame of the video in this database. The first stage was a cross-sectional comparison, with the four groups included in the study consisting of real data, our proposed method, [PERSON] et al. [13], and [PERSON] et al. [15]. The selected target identity pictures had neutral emotion (no emotion on the face) in three frontal poses and three voices were randomly selected for each emotion, except for the neutral emotion (seven in total), generating a total of sixty-three animations per group. The emotion source input to our method was the first frame of the video of the target identity for the remaining seven emotions, except for the neutral emotion. Participants were required to evaluate the videos on a scale of 5 out of 5 on the authenticity of the video and the degree of lip-sync. The comparison results of the four groups are shown in Figure 6, where error bars indicate standard deviation. Asterisks represent a significant difference, ** indicates \(p<0.01\). Pink, Orange, Blue and Green histograms denote [PERSON] et al. [13] and [PERSON] et al. [15], with similar scores, our method and real data, respectively. The second phase investigated whether the emotional rendering of our generated videos was authentic and effective. Participants at this stage were asked to classify the sentiments of the videos generated on the basis of real data, [PERSON] et al. [10], and our proposed method. It was decided that the video samples for this stage would all be taken from the same portrait and the same audio, but with different emotions, for a total of 64 videos for each method. Additionally, the videos would have no background sound to allow participants to classify their feelings using visual cues. The confusion matrix for the subjective emotion classification in the user survey is shown in Figure 7. Our videos were more similar to ground-truth video patterns and produced more diagonal confusion matrices than those of [PERSON] et al. The overall classification accuracies of 64.2% (ground-truth), 43.5% ([PERSON]), and 50.9% (our) validate the effectiveness of the proposed model in generating emotionally controlled talking face videos, showing the power of the emotional landmark animation generation network. Figure 6: User study results of audio-visual sync and video authenticity (refs. [13; 15]). In the last stage, an emotionally perceptive user survey was used regarding emotionally mismatched videos as in [PERSON] et al. (2018), to ask participants to identify the major and secondary emotions (if any) generated by each video. The input consisted of identity source portraits of neutral emotions as well as audio and emotion source portraits covering eight emotions. Sixty-four audiovisual emotion videos were generated by our system through permutation, of which eight were matched with each other. The evaluations of the participants on the emotions displayed in these videos can be used to some extent as a reference for determining the emotion recognition mechanism people rely on the most in recognizing the emotions depicted in these videos. As can be seen from the confusion matrix in Figure 8, the matching value of the visual emotion as the primary emotion is greater than the matching value of the auditory emotion as the primary emotion, with ratios of 41.4% and 34.9%, respectively. This indicates that, for the emotional perception of audiovisual content, the subjects relied much more on the visual modality than on the audio modality. Perceiving these eight emotions, such as happiness, sadness, and anger through the visual modality is relatively simple. The pairs of disgust/contempt and fear/surprise are more easily confused.