# Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait
###### Abstract  
With the continuous development of cross-modality generation, audio-driven talking face generation has made substantial advances in terms of speech content and mouth shape, but existing research on talking face emotion generation is still relatively unsophisticated. In this work, we present _Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait_ to synthesize lip-sync and an emotionally controllable high-quality talking face. Specifically, we take a facial reenactment perspective, using facial landmarks as an intermediate representation driving the expression generation of talking faces through the landmark features of an arbitrary emotional portrait. Meanwhile, decoupled design ideas are used to divide the model into three sub-networks to improve emotion control. They are the _lip-sync landmark animation generation network_, the _emotional landmark animation generation network_, and the _landmark-to-animation translation network_. The two landmark animation generation networks are responsible for generating content-related lip area landmarks and facial expression landmarks to correct the landmark sequences of the target portrait. Following this, the corrected landmark sequences and the target portrait are fed into the translation network to generate an emotionally controllable talking face. Our method controls the expressions of talking faces by driving the emotional portrait images while ensuring the generation of animated lip-sync, and can handle new audio and portraits not seen during training. A multi-perspective user study and extensive quantitative and qualitative evaluations demonstrate the superiority of the system in terms of visual emotion representation and video authenticity. t 202220220220220220220220220220220220220220220220220220220220220220222022022202220222022202222022202222022220222222222222222222222222222  
generate emotional animations while ensuring the high fidelity of the animation. However, due to the complexity of audio emotion itself, determining emotion in animation can lead to ambiguity. Furthermore, for the generation of the emotional animations studied above, the target portrait must be trained by the model, which results in its having a limited ability to render emotions and more limited application scenarios [21]. In this study, we propose Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait, a method that selects portrait images as an emotion-driven source and inputs audio and target identity portraits to generate emotionally diverse facial animations. Meanwhile, facial landmarks are used as intermediate representatives, dividing the model into an audio-to-landmark network and a landmark-to-animation network. Facial landmarks connect audio to pixels and have the following advantages for facial emotion animation generation tasks. On the one hand, facial geometry information is effectively maintained while deleting millions of pixels, resulting in a natural low-dimensional representation for the portrait. This allows the model to capture subtle dynamic landmark displacements of the head, avoiding the mishmash between portrait outlines and background pixels that miss these details. On the other hand, landmark features can convey sufficient emotional information for downstream emotional reenactment tasks and are more structured than intermediate representations such as action units (AU) [22]. In addition, to make the animation more emotionally controllable, we use the design idea of decoupling to divide the audio-to-landmark network into two sub-networks for the prediction of lip-sync landmarks and facial emotion landmarks. Firstly, the prediction of lip-sync landmarks is handled by the _lip-sync landmark animation generation network_, which is based on MakItTalk [15], by decoupling the audio content in order to improve accuracy. Secondly, for the prediction of facial emotion landmarks, an additional input emotion source portrait is used, which can be derived from an untrained arbitrary portrait.