SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiale Tao $^{1*}$  Shuhang Gu $^{1}$  Wen Li $^{2\dagger}$  Lixin Duan $^{1,2}$
School of Computer Science and Engineering, UESTC<sup>1</sup>
Shenzhen Institute for Advanced Study, UESTC2
{jialetao.std, shuhanggu, liwenbnu, lxduan}@gmail.com
Unsupervised face animation aims to generate a human face video based on the appearance of a source image, mimicking the motion from a driving video. Existing methods typically adopted a prior-based motion model (e.g., the local affine motion model or the local thin-plate-spline motion model). While it is able to capture the coarse facial motion, artifacts can often be observed around the tiny motion in local areas (e.g., lips and eyes), due to the limited ability of these methods to model the finer facial motions. In this work, we design a new unsupervised face animation approach to learn simultaneously the coarse and finer motions. In particular, while exploiting the local affine motion model to learn the global coarse facial motion, we design a novel motion refinement module to compensate for the local affine motion model for modeling finer face motions in local areas. The motion refinement is learned from the dense correlation between the source and driving images. Specifically, we first construct a structure correlation volume based on the keypoint features of the source and driving images. Then, we train a model to generate the tiny facial motions iteratively from low to high resolution. The learned motion refinements are combined with the coarse motion to generate the new image. Extensive experiments on widely used benchmarks demonstrate that our method achieves the best results among state-of-the-art baselines.
Face animation aims to generate a human face video based on the appearance of a source image to mimic the motion from a driving video. It has gained increasing attention from the community in recent years, due to its great potential in various applications including face swapping, digital humans, video conferencing, and more. Additionally, the emergence of large-scale face video datasets [18, 44] has further contributed to this growing interest.
Existing works on face animation can generally be divided into two categories: model-based and model-free methods (a.k.a supervised and unsupervised methods). On one hand, model-based methods [7, 33] typically use predefined 2D or 3D facial priors, such as 2D landmarks [20] and 3DMM parameters [2], to provide pose and expression information in the generated videos. While these methods have the advantage of describing accurate face poses, they are limited by overly strict structure priors that are not robust to hair and neck motions, which can heavily affect their performance in real-world applications. On the other hand, model-free animation is a more challenging and interesting problem, which aims to automatically learn the motion patterns without using predefined 2D or 3D facial models, thus being more flexible and robust in real-world scenarios. With the emergence of large-scale face video datasets, model-free methods have become more popular in
*Codes will be available at https://github.com/JialeTao/MRFA/
†The Corresponding Author
Figure 1: Illustration of the non-prior based motion refinement. Initially, a prior motion such as an affine transformation defines a motion flow. However, after the refinement, this flow can be matched to a new deformation that more realistically models underlying face deformations.
recent years [24, 25]. Among model-free methods, a local prior motion model is usually assumed to transform the sparse corresponding keypoints to the dense motion flow between the source and driving image. More recently, researchers have explored combining the two types of methods [40], such as using prior motion models to learn motion flow for warping source features while simultaneously employing 3DMM parameters to refine and enhance warped features.
In this work, we focus on the model-free face animation task, considering its flexibility for practical applications. Existing model-free face animation methods typically operate under the assumption of a local prior motion model, that is to assume the local motion between a source and driving image generally follows a parametric model such as affine transformation. For example, Siarohin et al. [24] proposed a local affine motion model, Siarohin et al. [25] and Tao et al. [29, 30] further constrained the learning process of the affine matrix, while others proposed a local thin-plate-spline motion model [43]. While these methods have the advantage of facilitating the process of learning motion and the ability to learn significant keypoints, their limitations are also obvious. Firstly, assuming that the human face is locally rigid may not be appropriate, resulting in sub-optimal motion models that fail to capture tiny motions in local facial areas. Secondly, accurately predicting geometric parameters such as the affine matrix from a single image is not easy. To address these issues, we propose a non-prior-based motion refinement approach to compensate for the inadequacy of existing prior-based motion models as illustrated in Fig. 1. Specifically, inspired by recent advances in optical flow [12, 28, 31], we utilize the keypoint features in building a correlation volume between the source and driving images, which represents the structure correspondence between the two images across all spatial locations. This correlation volume serves as non-prior motion evidence, according to which we employ a model to iteratively generate finer motions that refine the coarse motion predicted by prior motion models. In summary, compared to existing prior-based motion models, our approach exhibits a more powerful motion representation in terms of learning finer motions and thus is capable of achieving more realistic face animations.
We conduct extensive experiments on challenging face video benchmarks, and the superior performance of our method over state-of-the-art baselines highlights that learning motion refinement is advantageous for enhancing the existing prior-based motion models on face animation.
Face animation: Recent works on face animation can be roughly categorized into model-based and model-free methods, and more recently some works [40] tried to combine the two types of methods.
Model-based methods [4, 6, 7, 15, 21, 32, 36, 37, 38, 39] leverage predefined facial prior, such as 2D landmarks and 3D face models, to guide the pose or expression of the generated output. For instance, FSGAN [20] used facial landmarks as input to a generator directly to reenact a source face. Similarly, SAFA [33] used a 3D morphable model to aid in expression transfer. These methods are advantageous in modeling accurate face poses but suffer from the too-strict structure prior, which is not robust to neck and hair motions that are prevalent in practical scenarios.
In contrast, model-free methods [9, 10, 23, 24, 25, 30, 35, 42] tend to have better generalization performance and have become more popular due to the advent of large-scale face video datasets. These methods assume a local prior motion between subjects and learn local keypoints and corresponding motion transformation parameters in an unsupervised manner from videos. FOMM [24] is a pioneer work that proposed a local affine transformation model for image animation. Subsequently, MRAA [25], DAM [30], and MTIA [29] constrained the affine motion model learning process with stronger prior, such as PCA decomposition and cooperative motions. In fact, these methods assume a locally rigid object and mainly apply to human bodies, having limited performance on
human faces due to the easily violated locally rigid assumption. TPSM [42] proposed to adopt local thin-plate-spline motions for general animations, while DaGAN [10] used face depth information to better learn the keypoints and affine matrices. However, these methods still fall into the paradigm of prior-based motion models, which have difficulty in modeling tiny motions in local face areas. There are also some works [17, 34] that extremely abandoned the prior-based motion representation, they either used 3D head rotation supervision or utilized multiple source images for compensating the loss of motion prior. In this paper, we try to push the limit of single-image-based unsupervised face animation and propose a non-prior-based motion refinement approach, which overcomes the inadequacy of existing prior-based motion models by learning the finer motions.
Optical flow: Optical flow estimation has been a longstanding research problem in computer vision [11, 12, 27, 28]. Its objective is to estimate the motion between consecutive video frames. It's worth noting that the key part of unsupervised facial animation is also to estimate the motion flow between two images of the face. The main difference between the two tasks lies in the fact that optical flow deals with frames depicting the same scene, whereas facial animation focuses on transferring the motion between video frames featuring different subjects. This fundamental difference leads to variations in the methods employed for the two tasks. In more detail, optical flow computes the visual feature similarities between two frames. On the other hand, facial animation aims to disentangle the structural and visual appearance features and uses the structural feature to establish motion representations. Our method is inspired by the optical flow technique named RAFT [31]. Its fundamental idea is to develop a 4D correlation volume based on the visual features of two frames and use it to iteratively refine the motion flow. Similar to this concept, we build a structural correlation volume in face animation based on keypoint features; moreover, we introduce a multi-scale refinement mechanism to maximally leverage the source features, which helps to capture finer face motions.
Our method consists of four key components: (1) a coarse motion estimation module based on a prior motion model, (2) a structure correlation volume calculator, (3) a motion flow updater, and (4) an image generator. While the first and last components are similar to existing face animation methods [24, 25, 29, 30], the second and third components form our proposed non-prior-based motion refinement module. The overview of our method is presented in Fig. 2. Similar to existing methods [24, 25, 29, 30] for achieving face animation, we first estimate the motion flow between the driving and source images. Then, we use the motion flow to warp the source features which is decoded by the image decoder, and generate the final animation video frame by frame. The difference is that in addition to estimating a coarse motion flow based on a prior motion model, we further estimate the motion refinement by our proposed motion refinement module. The coarse motion and the motion refinement are combined to warp source features. In the following subsections, we will first describe the existing prior-based motion models and then introduce our proposed non-prior-based motion refinement module.
3.1 Prior based Motion Estimation
Existing methods [24, 25, 29, 30, 40, 42] generally employ the affine prior and thin plate spline prior to model local facial motion. For the sake of simplicity, we adopt the local affine motion model as our formulation. This model assumes that a human face is composed of several individual parts and each part's motion is defined by an affine transformation. The parameters of these affine transformations are predicted by a keypoint detector, as shown in Fig. 2(a). The local affine transformations are then transformed into a set of part motion flows. A dense motion network predicts a composition mask, which is used to weigh the set of part motion flows to obtain a single motion flow, and an occlusion map which is used to mask the warped source feature in the occluded area.
Formally, we denote by  $S$  and  $D$  the source and driving images. A set of keypoints  $\{p_s^k,p_d^k\} \in \mathbb{R}^{2\times 1}$  and corresponding affine parameters  $\{A_s^k,A_d^k\} \in \mathbb{R}^{2\times 2}$  are predicted by the keypoint detector, where  $k = 1,\dots,K$ . These affine parameters predicted from a single image are assumed to be the transformation to an abstract reference image [24]. Additionally, let  $z\in \{0,1,\dots,h\times w - 1\}$  denote any pixel coordinate of the driving image, where  $h$  and  $w$  define the resolution of the motion flow which in this stage is set to 4 times downsampled image resolution. The part motion flow between
Figure 2: Overview of the pipeline. Our method consists of three modules: (a) Prior-based coarse motion estimation module first estimates a coarse motion flow based on a prior motion model. (b) Our proposed non-prior based motion refinement module constructs a 4D structure correlation volume that provides the non-prior motion evidence, based on which the coarse motion flow is iteratively refined. (c) The image generation module for both encoding multi-scale source features and decoding multi-scale warped source features.
the driving and source images is derived using the following formula:
$$
\mathcal {T} _ {S \leftarrow D} ^ {k} (z) = p _ {s} ^ {k} + A _ {s} ^ {k} \left(A _ {d} ^ {k}\right) ^ {- 1} \left(z - p _ {d} ^ {k}\right), \tag {1}
$$
Intuitively, Equation (1) represents the flow format of an affine transformation that displays the coordinate correspondence between two images. In addition to the  $K$  part motion flows, a motion flow of the background, which is typically considered to be static, is also represented by:
$$
\mathcal {T} _ {S \leftarrow D} ^ {0} (z) = z, \tag {2}
$$
Using the predicted composition mask  $M \in \mathbb{R}^{H \times W \times (K + 1)}$ , we can obtain the motion flow between the driving and source image as follows:
$$
\mathcal {T} _ {S \leftarrow D} (z) = \sum_ {k = 0} ^ {K} M ^ {k} (z) \cdot \mathcal {T} _ {S \leftarrow D} ^ {k} (z), \tag {3}
$$
We treat the Equation (3) as a coarse estimation of the motion flow, and in the following sections, we denote it using  $\mathcal{F}_0 = \mathcal{T}_{S\leftarrow D}(z)$  without special indication. Additionally, we also denote  $O_0$  as the occlusion map estimated by the prior motion model. Note that we derived  $\mathcal{F}_0$  and  $O_0$  by the affine prior, it is also suitable to use the thin plate spline prior for estimating the coarse motion flow.
3.2 Non-prior based Motion Refinement
Prior-based motion models have the advantage of facilitating learning motion in the early stages and can learn meaningful keypoints. However, the prior assumption such as a locally rigid face may not be appropriate, limiting them on learning finer facial motions. In this subsection, we present our non-prior-based motion refinement approach that compensates for the limitations of prior-based motion models.
Our motion refinement process is iterative, using the initial flow estimation obtained from a prior motion model, as described in the previous subsection. First, we construct a structure correlation volume to represent pixel-level structure correspondence between driving and source images. This volume remains fixed throughout later iterations. For each iteration, we use the current flow estimation to access the correlation volume and extract neighboring patch features. The cut correlation values, along with the warped source feature and current flow estimation, are sent to the flow updater to
Figure 3: Details of the motion flow updater. It outputs the updated motion flow and occlusion map by processing three types of input features: warped source image feature, correlation feature extracted from the structure correlation volume using a lookup operation, and the current motion flow. The correlation feature and input motion flow are passed through two separate convolution modules, and their results are concatenated to yield an intermediate motion feature. This intermediate feature, together with the input motion flow and the warped source feature, is combined and input to a convolution module, which produces the residual flow and occlusion map.
obtain a refined flow. Note that the iteration proceeds from low to high resolutions, thereby enhancing the ability to capture finer facial movements. Fig. 3 provides a more detailed depiction of a single iteration of our motion flow update procedure. We will now introduce each component and operation.
Structure correlation volume: Inspired by recent advancements in optical flow [31], we propose a similar approach of constructing a structure correlation volume, which is considered to provide non-prior motion evidence for refining a coarse motion flow. We accomplish this by first encoding the source and driving structure features with two separate encoders. The driving structure encoder takes only the driving keypoints as input, while the source structure encoder concatenates the 4 times downsampled source image and keypoints as input. The source image is considered to provide dense visual context information for the sparse keypoints, which helps to densify the source structure. Keypoints are encoded with a fixed variance using Gaussian heatmaps centered on their positions. After encoding, the driving and source structure features are multiplied along the channel dimension to obtain a  $4D$  structure correlation volume  $C \in \mathbb{R}^{h \times w \times h \times w}$ . The first two dimensions align with the driving coordinate system, while the last two represent the source coordinate system. Each indexed value in the first two dimensions represents the correspondence confidence between the driving pixel position and all source pixels. Note that the calculation of the structure correlation volume is carried out at a resolution that is 4 times lower than that of the original image. We also used a pyramid of correlations, which is similar to the optical flow approach [31]. The pyramid is created by pooling the structure correlation volume repeatedly with a series of scale factors. Further details can be found in the supplementary material.
Correlation lookup: For each iteration of the updates, we utilize the current input motion flow to lookup a  $\{(2r + 1) \times (2r + 1)\}$ -patch correspondence values centered at the flow position, where  $r$  is the radius of the patch. This patch feature is further flattened to the channel dimension, resulting in a  $h_i \times w_i \times (2r + 1)^2$  correlation feature  $C_i$ , where  $h_i$  and  $w_i$  are the flow resolutions of the  $i$ th iteration. The lookup process is depicted on the left side of Fig. 3 and the correlation features are denoted by the yellow cuboid.
Motion flow update: The motion flow updater is comprised of four convolution modules that share parameters across all iterations. During each iteration, the flow updater takes in the current motion flow, extracted correlation values, and the warped source feature as inputs to produce outputs of updated motion flow and occlusion map.
To be more specific, we represent the current input motion flow as  $\mathcal{F}_{i-1}^{\uparrow}$  which is upsampled twice from the previous iteration's output. The correlation feature is represented by  $C_i$ , and the source input feature encoded by the image encoder is represented by  $f_i$ . Firstly, the residual flow  $d\mathcal{F}_i$  and the residual occlusion map  $d\mathcal{O}_i$  are generated by the flow updater  $\psi$ :
$$
d \mathcal {F} _ {i}, d \mathcal {O} _ {i} = \psi \left(\mathcal {F} _ {i - 1} ^ {\uparrow}, C _ {i}, \mathcal {F} _ {i - 1} ^ {\uparrow} \left(f _ {i}\right)\right), \tag {4}
$$
where  $\mathcal{F}_{i - 1}^{\dagger}()$  denotes the warping operation. The update process is then formulated as follows:
$$
\mathcal {F} _ {i} = \mathcal {F} _ {i - 1} ^ {\uparrow} + d \mathcal {F} _ {i}, \mathcal {O} _ {i} = \mathcal {O} _ {i - 1} ^ {\uparrow} + d \mathcal {O} _ {i}. \tag {5}
$$
We begin the iteration process at the resolution of the image downsampled by 32 times. The initial input motion flow for the first iteration is provided by:
$$
\mathcal {F} _ {0} ^ {\uparrow} = \text {D o w n s a m p l e} (\mathcal {F} _ {0}, 8), \tag {6}
$$
as  $\mathcal{F}_0$  is computed at resolution  $H / 4\times W / 4$ . The iteration is stopped once the maximal resolution iteration is accomplished, i.e., the image size. It should be noted that the initial flow input,  $\mathcal{F}_0^\uparrow$ , loses a significant amount of information due to the downsampling operation. To ensure preserving as much information from  $\mathcal{F}_0$  as possible, we reformulate Equation (5) as:
$$
\mathcal {F} _ {i} = \operatorname {R e s i z e} \left(\mathcal {F} _ {0}\right) + d \mathcal {F} _ {i} + d \mathcal {F} _ {i - 1} ^ {\uparrow} + d \mathcal {F} _ {i - 2} ^ {\uparrow \uparrow} + \dots , \tag {7}
$$
where the initial flow  $\mathcal{F}_0$  is directly resized to the current resolution. The similar rule is also applied to the occlusion map.
To summarize, the non-prior-based motion refinement module yields multi-scale refined motion flows and occlusion maps  $\{\mathcal{F}_i,\mathcal{O}_i\}_{i = 1}^N$ , which are utilized in the process of image generation for warping source features.
Non-prior-based initialization: As previously discussed, we have been utilizing a prior motion model to estimate coarse motion flow owing to its effectiveness in facilitating learning motion. However, we have also proposed a non-prior-based initialization approach. Specifically, we reshape the structure correlation volume to  $h \times w \times (h \times w)$  and denote the reshaped  $C$  as  $C^r$ . We then perform a softmax operation on the final dimension, treating the result as an attention matrix. This matrix is subsequently employed to sum up an identity grid. The non-prior-based initialization is formulated as follows:
$$
\mathcal {Q} _ {S \leftarrow D} (z) = \sum_ {i = 1} ^ {h \times w} i \cdot \operatorname {S o f t m a x} \left(C ^ {r}\right) [ z, i ] \tag {8}
$$
Where  $z$  indexes the driving spatial dimensions and  $i$  denotes the reshaped source dimension. In this way our proposed method becomes totally non-prior-based, we study this initialization in Section 4.
3.3 Image generation
As previously mentioned, we have prepared multi-scale motion flows, occlusion maps, and source features  $\{\mathcal{F}_i,\mathcal{O}_i,f_i\}_{i = 1}^N$ . To better synthesize facial details, we use a Unet-like image generator which effectively combines the multi-scale source features with the refined multi-scale motion flows. In each layer of the image decoder, we integrate information from the current warping and previous generation by multiplying them with the occlusion map and reversed occlusion map. The generation process can be summarized as follows:
$$
\operatorname {o u t} _ {1} = \mathcal {F} _ {1} \left(f _ {1}\right) \cdot \mathcal {O} _ {1} \tag {9}
$$
$$
\operatorname {o u t} _ {i + 1} = \mathcal {F} _ {i + 1} \left(f _ {i + 1}\right) \cdot \mathcal {O} _ {i + 1} + \operatorname {U p B o l c k} \left(\operatorname {R e s B o l c k} \left(\operatorname {o u t} _ {i}\right)\right) \cdot \left(1 - \mathcal {O} _ {i + 1}\right), \tag {10}
$$
where UpBolck defines a upsample-conv operation and ResBolck denotes a 2-layer resnet [8] block. At the last layer, the output is activated by a sigmoid function to obtain the generated image.
Our method is trained end to end. Following previous methods [24, 25] and for the sake of simplicity, we only adopt the perceptual loss and the equivariance loss as our objective functions.
Perceptual loss: We adopt the multi-resolution perceptual loss [13] defined with a pre-trained VGG-19 [26] network. Given the driving image  $D$  with resolution index  $i$ , the generated image  $\tilde{D}$  and the feature extractor  $\phi$  with layer index  $l$ , the perceptual loss can be written as follows:
$$
\mathcal {L} _ {p e r} = \sum_ {i} \sum_ {l} \left\| \phi_ {l} (D _ {i}) - \phi_ {l} (\tilde {D} _ {i}) \right\| _ {1}. \tag {11}
$$
Equivariance loss: Given a random geometric transformation  $\mathbf{T}$  and a driving image  $D$ , the equivariance loss can be written as follows:
$$
\mathcal {L} _ {\text {e q u i}} = \sum_ {k} \left\| \mathbf {T} \left(p _ {D} ^ {k}\right) - p _ {\mathbf {T} (D)} ^ {k} \right\| _ {1}. \tag {12}
$$