# Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait
## 2 Related Work  
### Audio-Driven Talking Face Generation  
Audio-driven facial animation generation is a popular topic in computer vision and in generative models. Its purpose is to transform a static portrait into a realistic talking face animation through arbitrary audio. Audio-driven facial animation does not require as many video clips of the target individual for training as graphics-based portrait facial animation to maintain the phonemic information of the target [7; 23; 24]. The methods used by previous researchers for audio-driven facial animation can be divided into two groups based on the presence or absence of an intermediate structural representation. **Direct Image Reconstruction Methods. ** Raw pixels or dots with lip movements synchronized with the speech signal are directly generated through the audio driver. With the development of audiovisual cross-modals, [PERSON] et al. [11] proposed to generate lip-synced videos by the encoder-decoder CNN model named Speech2 Vid. [PERSON] et al. [12] used conditional recursive generative networks to combine image and audio features in recursive units to achieve temporal dependence across animation frames. [PERSON] et al. [14] developed a new lip-sync discriminator that enhances the robustness of lip synchronization in the wild. [PERSON] et al. [16] successfully added head motion to the generated animation by defining a 12-dimensional pose encoding in latent space. The above methods focus on matching lip movements to audio, but they were unable to control the video emotion. More recently, a few methods [10; 19; 20; 21; 25] have added an emotional element to the generated talking face. [PERSON] et al. [25] suggested an end-to-end network with a more realistic emotional component using a 3D-based method; however, the learned emotional states are difficult to comprehend and do not model facial features. [PERSON] et al. [19] used a generative adversarial network (GAN) to directly synthesize animations from audio and pixels to express the emotions conveyed by audio; however, their results suffered from fidelity problems (facial distortion, blurring, and other artefacts). Recently, [PERSON] et al. [20] developed an end-to-end emotionally controllable conversation face generation system using speech, individual face images, and categorized sentiment labels as inputs. Although the implementation idea is innovative, emotion control is affected by the classification expressions, and it can only cover a limited range of expressions. In addition, some studies have opted to add additional visual input to control emotion. [PERSON] et al. [10] could only achieve a transition from neutral to emotional; [PERSON] et al. [21] controlled head pose and facial expression through multiple visual inputs, but this also resulted in a model with low robustness to complex backgrounds. Figure 1: The abbreviated framework of the proposed method. Two portraits and audio are input, and a video of the identity source portrait talking face with new emotional expressions is obtained as output. **Intermediate Representation Method. ** In the portrait reconstruction pipeline, structural information was employed as an intermediate representation to help the model learn the features of the target. The neural network of [PERSON] et al. [26] can generate talking face landmarks from arbitrarily long noisy speeches. [PERSON] et al. [13] divided the process into two stages: an audio-to-landmark network and a landmark-to-animation network based on facial landmarks. This cascade method avoids fitting erroneous audiovisual signal correlations that are unrelated to speech content. [PERSON] et al. [15] used decoupled learning to separate speech into content and speaker identity components trained separately, whereby spontaneous head posture movement is predicted after successfully completing the lip-sync task. In addition to facial landmarks, previous approaches have also used other intermediate representations to facilitate animated synthesis, such as the approach of [PERSON] et al.