# Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait
## 3 Method
### Lip-Sync Landmark Animation Generation Network  
The goal of the lip-sync landmark animation generation network is to generate the target portrait lip region landmark animation synchronized with audio content, which requires mapping the audio to the facial landmark locations. This network uses an encoder-decoder network structure containing a multilayer perceptron (MLP) encoder, long short-term memory (LSTM) [39] encoder, and an MLP decoder. Content-independent features in audio reduce the prediction accuracy to a certain degree. Therefore, adopting the idea in [15], disentangled learning in the field of voice conversion using AutoVC [40] was used to design a state-of-the-art approach to disentangle audio into content embedding and style embedding. The speaker-independent content embedding \(Ec\in R^{T\times C}\) was separated from  
Figure 2: The general framework of the proposed method consists of three network models in dashed lines in three colors. Using the above network structures, the identity source portrait image, emotion source portrait image, and audio are used as inputs to generate an emotionally controllable talking face.  
audio \(V\), where \(T\) is the total number of input audio frames (\(t\) denotes any frame), and \(C\) is the content dimension. In addition to content embedding, a landmark embedding vector is also extremely important, providing facial shape and identity information for this network. To this end, a 3D landmark detector [41] is first used to create a landmark \(L_{a}\in R^{68\times 3}\), followed by an MLP encoder to extract the landmark feature from \(L_{a}\). The deconstructed content embedding with landmark feature is then fed into an LSTM encoder that captures the audio content and continuous lip landmark dependencies. Finally, the MLP decoder accepts the output of the LSTM encoder and predicts the lip landmark displacement \(\Delta P_{t}\), which is summed with the original landmarks to obtain the lip-sync landmark sequence, \(P_{t}\). Mathematically, this network can be expressed as follows:  
\[\Delta P_{t}=MLP_{c}\Big{(}LSTM_{c}\Big{(}Ec_{t\to t+\lambda},MLP_{L} \Big{(}L_{a};W_{mlp,t}\Big{)};W_{lstm}\Big{)};W_{mlp,c}\Big{)} \tag{1}\]  
\[P_{t}=L_{a}+\Delta P_{t} \tag{2}\]  
where \(\{W_{mlp,t},W_{lstm},W_{mlp,c}\}\) are the learnable parameters of the MLP encoder, LSTM encoder, and MLP decoder, respectively; the MLP encoder \(MLP_{L}\) consists of two layers with internal hidden state vectors of size 256 and 128; the LSTM encoder \(LSTM_{c}\) is made up of three layers of units, each with a hidden state vector of 256; and the MLP decoder \(MLP_{c}\) comprises three layers with internal hidden state vectors of size 512, 256, and 204 (\(68\times 3\)); \(\{Ec,L_{a},\Delta P_{t},P_{t}\}\) is consistent with the above descriptions; for content embedding \(Ec\), the window size for each frame \(t\) is \(\lambda=18\).  
To improve the realism of the lip landmark displacement, we used the loss function \(\mathcal{L}_{C}\) to minimize the distance between the reference landmark \(\hat{P}_{t}\) and the predicted landmark \(P_{t}\) (where \(i\) is the index of each individual landmark):  
\[\mathcal{L}_{L}=\sum_{t=1}^{T}\sum_{i=1}^{N}\left\|P_{i,t}-\hat{P}_{i,t}\right\| _{2}^{2} \tag{3}\]