# Improved First-Order Motion Model of Image Animation with Enhanced Dense Motion and Repair Ability
## 2 Methodology
### Algorithm Framework  
Our improved FOMM is mainly composed of the keypoint detector module, the dense motion module, the attention module, the multi-scale occlusion restoration module and the generator module. The network structure is shown in Figure 4. The keypoint detector module and the dense motion module are basically the same as the corresponding modules in FOMM.  
Figure 3: Incorrectly located key points in FOMM, resulting in poor frame generation results. (Different colorful dots represent different key points located by FOMM, and the red boxes indicate the incorrect key points.)  
We denote \(S\) and \(D\) as the source and the driving frames extracted from the same video, respectively. The keypoint detection module detects the unsupervised key points of the image. The dense motion module first establishes sparse motion and affine transformation between individual rigid regions through the key points, and then generates dense optical flow and an occlusion map of the whole image through the \(S\) warped by the sparse motion.  
Keypoints prediction based on U-net output \(K\) heat maps, \(M^{k},\ldots,M^{k}\) for the input image at unsupervised keypoints, followed by softmax, s.t. \(M^{k}\in[0,1]^{H\times W}\), where \(H\) and \(W\) are the height and width of the image, respectively, and \(\sum_{z\in Z}M^{k}(z)=1\), where \(z\) is a pixel location (\(x\), \(y\) coordinates) in the image, the set of all pixel locations being \(Z\), and \(M^{k}(z)\) is the \(k\)-th heatmap weight at pixel \(z\). Equation (1) estimates the translation component of affine transformation in the abstract coordinates mapped by the input picture.  
\[\mu^{k}=\sum_{z\in Z}M^{k}(z)z. \tag{1}\]  
For both frames \(S\) and \(D\), the keypoints prediction also output four additional channels \(P^{k}_{ij}\in\mathbb{R}^{H\times W}\) for each keypoint, where \(i\in\{0,1\},j\in\{0,1\}\) indexes the affine matrix \(A^{k}_{X\gets R}[i,j]\), where \(X\in\{S,D\}\), and \(R\) is the assumed reference frame. This is shown in Equation (2).  
\[A^{k}_{X\gets R}[i,j]=\sum_{z\in Z}M^{k}(z)P^{k}_{ij}(z). \tag{2}\]  
The dense motion module uses an encoder-decoder structure to predict the rigid mask of each keypoint through the heatmap representation of the keypoints and the warped \(S\) frame of the sparse optical flow field. Equations (3) and (4) represent the heatmap representation of the keypoints of the input image to \(R\) and the heatmap representation of \(S\) to \(D\), respectively. Where \(\sigma\) is a hyper-parameter, we usually take 0.01 based on experience.  
\[G^{k}_{X}=e^{-0.5\times\sum_{z\in Z}{(z-\mu^{k}_{X})}^{2}/\sigma} \tag{3}\]  
\[H=[(G_{D}-G_{S}),\ 0] \tag{4}\]  
In order to obtain the sparse optical flow field, FOMM needs to obtain the affine matrix \(A^{k}_{S\gets D}\) (Equation (5)) from \(S\) to \(D\), and calculate the sparse optical flow field through Equation (6).  
\[A^{k}_{S\gets D}=A^{k}_{S\gets R}\Big{(}A^{k}_{D\gets R}\Big{)}^ {-1} \tag{5}\]  
\[\mathcal{F}^{k}=\mu^{k}_{S}+A^{k}_{S\gets D}\Big{(}z-\mu^{k}_{D}\Big{)} \tag{6}\]  
With inputting \(S\) (warped by \(\mathcal{F}^{k}\)) and \(H\), the dense motion module outputs an intermediate feature \(\xi\). In Equation (7), \(\mathcal{F}\) represents warping operation and _Unet_ represents model architecture.  
\[\xi=Unet(H,f_{w}(S,\mathcal{F})) \tag{7}\]  
Figure 4: The framework of our method.  
Then, the intermediate features \(\xi\) will enter the attention module to obtain the dense optical flow field and the multi-scale occlusion restoration module to obtain multiple occlusion maps. We will introduce our attention module and multi-scale occlusion restoration module in detail in Sections 2.2 and 2.3.