Title: Learning Motion Refinement for Unsupervised Face Animation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Face animation

Motion refinement

Structure correlation
Abstract: Unsupervised face animation aims to generate a human face video based on the
appearance of a source image, mimicking the motion from a driving video. Existing
methods typically adopted a prior-based motion model (e.g., the local affine motion
model or the local thin-plate-spline motion model). While it is able to capture
the coarse facial motion, artifacts can often be observed around the tiny motion
in local areas (e.g., lips and eyes), due to the limited ability of these methods
to model the finer facial motions. In this work, we design a new unsupervised
face animation approach to learn simultaneously the coarse and finer motions. In
particular, while exploiting the local affine motion model to learn the global coarse
facial motion, we design a novel motion refinement module to compensate for
the local affine motion model for modeling finer face motions in local areas. The
motion refinement is learned from the dense correlation between the source and
driving images. Specifically, we first construct a structure correlation volume based
on the keypoint features of the source and driving images. Then, we train a model
to generate the tiny facial motions iteratively from low to high resolution. The
learned motion refinements are combined with the coarse motion to generate the
new image. Extensive experiments on widely used benchmarks demonstrate that
our method achieves the best results among state-of-the-art baselines.

Learning Motion Refinement for Unsupervised Face Animation

Abstract

Unsupervised face animation aims to generate a human face video based on the appearance of a source image, mimicking the motion from a driving video. Existing methods typically adopted a prior-based motion model ( e.g ., the local affine motion model or the local thin-plate-spline motion model). While it is able to capture the coarse facial motion, artifacts can often be observed around the tiny motion in local areas ( e.g ., lips and eyes), due to the limited ability of these methods to model the finer facial motions. In this work, we design a new unsupervised face animation approach to learn simultaneously the coarse and finer motions. In particular, while exploiting the local affine motion model to learn the global coarse facial motion, we design a novel motion refinement module to compensate for the local affine motion model for modeling finer face motions in local areas. The motion refinement is learned from the dense correlation between the source and driving images. Specifically, we first construct a structure correlation volume based on the keypoint features of the source and driving images. Then, we train a model to generate the tiny facial motions iteratively from low to high resolution. The learned motion refinements are combined with the coarse motion to generate the new image. Extensive experiments on widely used benchmarks demonstrate that our method achieves the best results among state-of-the-art baselines.

1 Introduction

Face animation aims to generate a human face video based on the appearance of a source image to mimic the motion from a driving video. It has gained increasing attention from the community in recent years, due to its great potential in various applications including face swapping, digital humans, video conferencing, and more. Additionally, the emergence of large-scale face video datasets [18, 44] has further contributed to this growing interest.

Existing works on face animation can generally be divided into two categories: model-based and model-free methods (a.k.a supervised and unsupervised methods). On one hand, model-based methods [7, 33] typically use predefined 2D or 3D facial priors, such as 2D landmarks [20] and 3DMM parameters [2], to provide pose and expression information in the generated videos. While these methods have the advantage of describing accurate face poses, they are limited by overly strict structure priors that are not robust to hair and neck motions, which can heavily affect their performance in real-world applications. On the other hand, model-free animation is a more challenging and interesting problem, which aims to automatically learn the motion patterns without using predefined 2D or 3D facial models, thus being more flexible and robust in real-world scenarios. With the emergence of large-scale face video datasets, model-free methods have become more popular in

Refinement

recent years [24, 25]. Among model-free methods, a local prior motion model is usually assumed to transform the sparse corresponding keypoints to the dense motion flow between the source and driving image. More recently, researchers have explored combining the two types of methods [40], such as using prior motion models to learn motion flow for warping source features while simultaneously employing 3DMM parameters to refine and enhance warped features.

In this work, we focus on the model-free face animation task, considering its flexibility for practical applications. Existing model-free face animation methods typically operate under the assumption of a local prior motion model, that is to assume the local motion between a source and driving image generally follows a parametric model such as affine transformation. For example, Siarohin et al . [24] proposed a local affine motion model, Siarohin et al . [25] and Tao et al . [29, 30] further constrained the learning process of the affine matrix, while others proposed a local thin-plate-spline motion model [43]. While these methods have the advantage of facilitating the process of learning motion and the ability to learn significant keypoints, their limitations are also obvious. Firstly, assuming that the human face is locally rigid may not be appropriate, resulting in sub-optimal motion models that fail to capture tiny motions in local facial areas. Secondly, accurately predicting geometric parameters such as the affine matrix from a single image is not easy. To address these issues, we propose a non-prior-based motion refinement approach to compensate for the inadequacy of existing prior-based motion models as illustrated in Fig. 1. Specifically, inspired by recent advances in optical flow [12, 28, 31], we utilize the keypoint features in building a correlation volume between the source and driving images, which represents the structure correspondence between the two images across all spatial locations. This correlation volume serves as non-prior motion evidence, according to which we employ a model to iteratively generate finer motions that refine the coarse motion predicted by prior motion models. In summary, compared to existing prior-based motion models, our approach exhibits a more powerful motion representation in terms of learning finer motions and thus is capable of achieving more realistic face animations.

We conduct extensive experiments on challenging face video benchmarks, and the superior performance of our method over state-of-the-art baselines highlights that learning motion refinement is advantageous for enhancing the existing prior-based motion models on face animation.

2 Related work

Face animation: Recent works on face animation can be roughly categorized into model-based and model-free methods, and more recently some works [40] tried to combine the two types of methods.

Model-based methods [4, 6, 7, 15, 21, 32, 36, 37, 38, 39] leverage predefined facial prior, such as 2D landmarks and 3D face models, to guide the pose or expression of the generated output. For instance, FSGAN [20] used facial landmarks as input to a generator directly to reenact a source face. Similarly, SAFA [33] used a 3D morphable model to aid in expression transfer. These methods are advantageous in modeling accurate face poses but suffer from the too-strict structure prior, which is not robust to neck and hair motions that are prevalent in practical scenarios.

In contrast, model-free methods [9, 10, 23, 24, 25, 30, 35, 42] tend to have better generalization performance and have become more popular due to the advent of large-scale face video datasets. These methods assume a local prior motion between subjects and learn local keypoints and corresponding motion transformation parameters in an unsupervised manner from videos. FOMM [24] is a pioneer work that proposed a local affine transformation model for image animation. Subsequently, MRAA [25], DAM [30], and MTIA [29] constrained the affine motion model learning process with stronger prior, such as PCA decomposition and cooperative motions. In fact, these methods assume a locally rigid object and mainly apply to human bodies, having limited performance on

human faces due to the easily violated locally rigid assumption. TPSM [42] proposed to adopt local thin-plate-spline motions for general animations, while DaGAN [10] used face depth information to better learn the keypoints and affine matrices. However, these methods still fall into the paradigm of prior-based motion models, which have difficulty in modeling tiny motions in local face areas. There are also some works [17, 34] that extremely abandoned the prior-based motion representation, they either used 3D head rotation supervision or utilized multiple source images for compensating the loss of motion prior. In this paper, we try to push the limit of single-image-based unsupervised face animation and propose a non-prior-based motion refinement approach, which overcomes the inadequacy of existing prior-based motion models by learning the finer motions.

Optical flow: Optical flow estimation has been a longstanding research problem in computer vision [11, 12, 27, 28]. Its objective is to estimate the motion between consecutive video frames. It's worth noting that the key part of unsupervised facial animation is also to estimate the motion flow between two images of the face. The main difference between the two tasks lies in the fact that optical flow deals with frames depicting the same scene, whereas facial animation focuses on transferring the motion between video frames featuring different subjects. This fundamental difference leads to variations in the methods employed for the two tasks. In more detail, optical flow computes the visual feature similarities between two frames. On the other hand, facial animation aims to disentangle the structural and visual appearance features and uses the structural feature to establish motion representations. Our method is inspired by the optical flow technique named RAFT [31]. Its fundamental idea is to develop a 4D correlation volume based on the visual features of two frames and use it to iteratively refine the motion flow. Similar to this concept, we build a structural correlation volume in face animation based on keypoint features; moreover, we introduce a multi-scale refinement mechanism to maximally leverage the source features, which helps to capture finer face motions.

3 Method

Our method consists of four key components: (1) a coarse motion estimation module based on a prior motion model, (2) a structure correlation volume calculator, (3) a motion flow updater, and (4) an image generator. While the first and last components are similar to existing face animation methods [24, 25, 29, 30], the second and third components form our proposed non-prior-based motion refinement module. The overview of our method is presented in Fig. 2. Similar to existing methods [24, 25, 29, 30] for achieving face animation, we first estimate the motion flow between the driving and source images. Then, we use the motion flow to warp the source features which is decoded by the image decoder, and generate the final animation video frame by frame. The difference is that in addition to estimating a coarse motion flow based on a prior motion model, we further estimate the motion refinement by our proposed motion refinement module. The coarse motion and the motion refinement are combined to warp source features. In the following subsections, we will first describe the existing prior-based motion models and then introduce our proposed non-prior-based motion refinement module.

3.1 Prior based Motion Estimation

Existing methods [24, 25, 29, 30, 40, 42] generally employ the affine prior and thin plate spline prior to model local facial motion. For the sake of simplicity, we adopt the local affine motion model as our formulation. This model assumes that a human face is composed of several individual parts and each part's motion is defined by an affine transformation. The parameters of these affine transformations are predicted by a keypoint detector, as shown in Fig. 2(a). The local affine transformations are then transformed into a set of part motion flows. A dense motion network predicts a composition mask, which is used to weigh the set of part motion flows to obtain a single motion flow, and an occlusion map which is used to mask the warped source feature in the occluded area.

Formally, we denote by S and D the source and driving images. A set of keypoints { p k s , p k d } ∈ R 2 × 1 and corresponding affine parameters { A k s , A k d } ∈ R 2 × 2 are predicted by the keypoint detector, where k = 1 , ..., K . These affine parameters predicted from a single image are assumed to be the transformation to an abstract reference image [24]. Additionally, let z ∈ { 0 , 1 , ..., h × w -1 } denote any pixel coordinate of the driving image, where h and w define the resolution of the motion flow which in this stage is set to 4 times downsampled image resolution. The part motion flow between

Keypoint

Detector

⨂

Structure

Correlation

Volume

Dense

Motion

Driving

Structure

Encoder

Source

Structure

Encoder

Flow

Updater

Image

Encoder

Image

Decoder

Source Feature

Flow

Updater

Source Feature

⋯

Flow

Updater

Source Feature

Updated Flow

(a) Prior based coarse motion estimation

(b) Non-prior based motion refinement

(c) Image generation

Updated flow for warping

⋯

Updated Flow

Initial Flow

𝑆

𝐷

$ 𝐷

the driving and source images is derived using the following formula:

Intuitively, Equation (1) represents the flow format of an affine transformation that displays the coordinate correspondence between two images. In addition to the K part motion flows, a motion flow of the background, which is typically considered to be static, is also represented by:

Using the predicted composition mask M ∈ R H × W × ( K +1) , we can obtain the motion flow between the driving and source image as follows:

We treat the Equation (3) as a coarse estimation of the motion flow, and in the following sections, we denote it using F 0 = T S ← D ( z ) without special indication. Additionally, we also denote O 0 as the occlusion map estimated by the prior motion model. Note that we derived F 0 and O 0 by the affine prior, it is also suitable to use the thin plate spline prior for estimating the coarse motion flow.

3.2 Non-prior based Motion Refinement

Prior-based motion models have the advantage of facilitating learning motion in the early stages and can learn meaningful keypoints. However, the prior assumption such as a locally rigid face may not be appropriate, limiting them on learning finer facial motions. In this subsection, we present our non-prior-based motion refinement approach that compensates for the limitations of prior-based motion models.

Our motion refinement process is iterative, using the initial flow estimation obtained from a prior motion model, as described in the previous subsection. First, we construct a structure correlation volume to represent pixel-level structure correspondence between driving and source images. This volume remains fixed throughout later iterations. For each iteration, we use the current flow estimation to access the correlation volume and extract neighboring patch features. The cut correlation values, along with the warped source feature and current flow estimation, are sent to the flow updater to

Warp

Concat

Sum

Source

Feature

Convs

Convs

Updated Flow &

Occlusion

Current Flow

Current Flow & Occlusion

Current Flow

Lookup

Convs

Convs

obtain a refined flow. Note that the iteration proceeds from low to high resolutions, thereby enhancing the ability to capture finer facial movements. Fig. 3 provides a more detailed depiction of a single iteration of our motion flow update procedure. We will now introduce each component and operation.

Structure correlation volume: Inspired by recent advancements in optical flow [31], we propose a similar approach of constructing a structure correlation volume, which is considered to provide non-prior motion evidence for refining a coarse motion flow. We accomplish this by first encoding the source and driving structure features with two separate encoders. The driving structure encoder takes only the driving keypoints as input, while the source structure encoder concatenates the 4 times downsampled source image and keypoints as input. The source image is considered to provide dense visual context information for the sparse keypoints, which helps to densify the source structure. Keypoints are encoded with a fixed variance using Gaussian heatmaps centered on their positions. After encoding, the driving and source structure features are multiplied along the channel dimension to obtain a 4 D structure correlation volume C ∈ R h × w × h × w . The first two dimensions align with the driving coordinate system, while the last two represent the source coordinate system. Each indexed value in the first two dimensions represents the correspondence confidence between the driving pixel position and all source pixels. Note that the calculation of the structure correlation volume is carried out at a resolution that is 4 times lower than that of the original image. We also used a pyramid of correlations, which is similar to the optical flow approach [31]. The pyramid is created by pooling the structure correlation volume repeatedly with a series of scale factors. Further details can be found in the supplementary material.

Correlation lookup: For each iteration of the updates, we utilize the current input motion flow to lookup a { (2 r +1) × (2 r +1) } -patch correspondence values centered at the flow position, where r is the radius of the patch. This patch feature is further flattened to the channel dimension, resulting in a h i × w i × (2 r +1) 2 correlation feature C i , where h i and w i are the flow resolutions of the i 'th iteration. The lookup process is depicted on the left side of Fig. 3 and the correlation features are denoted by the yellow cuboid.

Motion flow update: The motion flow updater is comprised of four convolution modules that share parameters across all iterations. During each iteration, the flow updater takes in the current motion flow, extracted correlation values, and the warped source feature as inputs to produce outputs of updated motion flow and occlusion map.

To be more specific, we represent the current input motion flow as F ↑ i -1 which is upsampled twice from the previous iteration's output. The correlation feature is represented by C i , and the source input feature encoded by the image encoder is represented by f i . Firstly, the residual flow d F i and the residual occlusion map d O i are generated by the flow updater ψ :

where F ↑ i -1 () denotes the warping operation. The update process is then formulated as follows:

We begin the iteration process at the resolution of the image downsampled by 32 times. The initial input motion flow for the first iteration is provided by:

as F 0 is computed at resolution H/ 4 × W/ 4 . The iteration is stopped once the maximal resolution iteration is accomplished, i.e., the image size. It should be noted that the initial flow input, F ↑ 0 , loses a significant amount of information due to the downsampling operation. To ensure preserving as much information from F 0 as possible, we reformulate Equation (5) as:

where the initial flow F 0 is directly resized to the current resolution. The similar rule is also applied to the occlusion map.

To summarize, the non-prior-based motion refinement module yields multi-scale refined motion flows and occlusion maps {F i , O i } N i =1 , which are utilized in the process of image generation for warping source features.

Non-prior-based initialization: As previously discussed, we have been utilizing a prior motion model to estimate coarse motion flow owing to its effectiveness in facilitating learning motion. However, we have also proposed a non-prior-based initialization approach. Specifically, we reshape the structure correlation volume to h × w × ( h × w ) and denote the reshaped C as C r . We then perform a softmax operation on the final dimension, treating the result as an attention matrix. This matrix is subsequently employed to sum up an identity grid. The non-prior-based initialization is formulated as follows:

Where z indexes the driving spatial dimensions and i denotes the reshaped source dimension. In this way our proposed method becomes totally non-prior-based, we study this initialization in Section 4.

3.3 Image generation

As previously mentioned, we have prepared multi-scale motion flows, occlusion maps, and source features {F i , O i , f i } N i =1 . To better synthesize facial details, we use a Unet-like image generator which effectively combines the multi-scale source features with the refined multi-scale motion flows. In each layer of the image decoder, we integrate information from the current warping and previous generation by multiplying them with the occlusion map and reversed occlusion map. The generation process can be summarized as follows:

where UpBolck defines a upsample-conv operation and ResBolck denotes a 2-layer resnet [8] block. At the last layer, the output is activated by a sigmoid function to obtain the generated image.

3.4 Training

Our method is trained end to end. Following previous methods [24, 25] and for the sake of simplicity, we only adopt the perceptual loss and the equivariance loss as our objective functions.

Perceptual loss: We adopt the multi-resolution perceptual loss [13] defined with a pre-trained VGG-19 [26] network. Given the driving image D with resolution index i , the generated image ˜ D , and the feature extractor ϕ with layer index l , the perceptual loss can be written as follows:

Equivariance loss: Given a random geometric transformation T and a driving image D , the equivariance loss can be written as follows:

The overall loss is formulated by the sum of the two without weighting:

4 Experiments

Implementation details: We adopt two Hourglass networks [19] for the keypoint detector and the dense motion network, and a Unet [22] for the image generator similar to previous works [24, 25]. We further employ two hourglass networks for the driving and source structure encoders. The flow updater consists of four convolution blocks that share the parameters across the iterations. We train our method for 100 epochs on four NVIDIA A100 GPU cards or eight NVIDIA 3090 GPU cards. It takes about 24 hours for training. The number of keyoints is set to 10 following previous methods [24, 29, 30, 42]. The patch radius r is set to 3 and the number of iterations is set to 6 . The Adam optimizer [16] is adopted with β 1 = 0 . 5 and β 2 = 0 . 999 , the initial learning rate is set as 2 × 10 -4 and dropped by a factor of 10 at the end of 60 th and 90 th epoch. We leave the architecture details in the supplementary material.

Datasets: We conduct experiments on the widely used Voxceleb1 [18] dataset and the recently collected more challenged CelebV-HQ dataset [44]. Voxceleb1 is a talking head dataset consisting of 20047 videos, among which 19522 are used for training and 525 are used for testing. The CelebV-HQ dataset consists of 35666 video clips, we randomly choose 500 of them for testing. All videos are resized to 256 × 256 for a fair comparison with existing methods.

Metrics: We measure the reconstruction quality using L1, Peak Signal-to-Noise Ratio (PSNR), and LPIPS [41] following [35, 40]. To evaluate the transferred motion quality we adopt Average Keypoint Distance (AKD) following previous methods [24, 25, 30, 42]. The identity quality of the generated videos is measured by Average Euclidean Distance (AED) [24, 25]. We use FID, ARD, and AUH metrics for evaluating cross-identity face animation following [5]. All these metrics are the lower the better except the PSNR.

Baselines: Abunch of baseline methods are compared, including FOMM [24], MRAA [25], LIA [35], DAM [30], DaGAN [10], TPSM [42], MTIA [29] and FNeVR [40]. All reported baseline results are obtained by evaluating the checkpoints from their Github repo. or retrain the provided codes, except that the FNeVR does not provide the training code and thus we do not report their results on the CelebV-HQ dataset. It is worth noting that most of these baseline methods adopt a prior motion model, while DaGAN [10] further explored the face depth information and FNeVR leverage the 3DMM parameters. For comparison with them, we adopt the motion transformer [29] for our prior motion estimation, while we also employ the affine motion model [24] and thin-plate-spline motion model [42] in the section of model ablations.

Quantitative comparison: Table 1 displays the same-identity reconstruction results, which is the standard evaluation setting used in previous methods [10, 24, 25, 29, 30, 40, 42] that reconstruct a driving video based on its first-frame appearance and all-frame motion. As can be seen, our

MRAA

Source

Driving

FOMM

LIA

DAM

DaGAN

TPSM

FNeVR

Ours

method outperforms other methods across all metrics on the two datasets. Specifically, large-margin improvements on the reconstruction metrics (L1, PSNR, LPIPS) demonstrate the advantages of learning motion refinement. What's more, our method achieved the best motion quality (the best score on the AKD metric) particularly on the more challenging CelebV-HQ dataset, highlighting the powerful motion representation capabilities of our non-prior-based motion refinement approach. The superior generated motion quality of our method is further validated by the best ARD and AUH metric in cross-identity experiments in Table 2. The best AED score and FID score also indicate that our method preserves the source face identity well during motion transfer. These significant observations are further supported by the qualitative results presented in Fig. 4, Fig. 5, and Fig. 6.

Qualitative comparison: We conducted experiments on both the same-identity video reconstruction and cross-identity face reenactment similar to previous methods [10, 29, 43], showcasing our qualitative results in Fig. 4, Fig. 5, and Fig. 6. Our method consistently demonstrates greater robustness against large motions (the first row of Fig. 4), detailed face deformations (the third row of Fig. 5 and the fourth row of Fig. 6), occlusions (the second and last row of Fig. 4), and varying light conditions (the fourth row of the Fig. 4), etc. Additionally, our approach outperforms other methods in generating precise facial details such as the eyes and lips, while avoiding common artifacts. To sum up, our methods can effectively learn non-prior-based motion refinement, exhibiting a significant improvement over prior-based motion representation that is limited on learning finer facial motions.

Model ablations: We performed a model ablation study to further evaluate the proposed N onP riorbased M otion R efinement module. As previously discussed in Equation (8), in the non-prior-based initialization process, we initialize the motion flow using the structure correlation volume itself. We

Source

Driving

FOMM

LIA

DAM

DaGAN

TPSM

FNeVR

Ours

MTIA

2元8

Source

Driving

FOMM

LIA

DAM

DaGAN

TPSM

MRAA

Ours

MTIA

DRCESDEPLOYEDATGOVTBULDING

RCES DE

denoted this model variant as NPMR. As shown in Table 3, our NPMR-only model did not yield good results, highlighting the importance of prior-based initial motion estimation for learning finer facial motions and the crucial role it plays in our non-prior-based motion refinement module. When combined with specific prior motion estimations, such as that from the affine and thin plate spline motion models, our method showed promising improvements over the original prior motion models. These combinations with different prior motion models significantly demonstrate the flexibility of the proposed non-prior-based motion refinement module, and its strong ability to promote the enhanced motion representation over existing prior motion models.

Component ablations: We conducted a thorough examination of the inputs and outputs of our non-prior-based motion refinement module through detailed ablations. Specifically, we explored the impacts of the model variants of the source structure encoder without the input of a source image, the flow updater without the input of warped source features, and the flow updater without the output of occlusion or flow (i.e. only updating either occlusion or flow and keeping the other the same as in the initialization process). The results are presented in Table 4. It is noteworthy that both the warped source feature and the source image play an important role in the motion refinement process as demonstrated by the decreased AKD and AED metric values on the w/o warped source feature and w/o source image variants. Importantly, if we solely refine the occlusion map without updating the motion flow (and vice versa), a significant decrease in AKD and AED metrics occurs, validating the significance of our motivation to refine the coarse motion flow.

Source

Driving

Refined motion flow

Motion flow visualizations: To further understand the motion refinement process, we visualize the refined motion flows in different iterations. As can be seen in Fig. 7, the motion flow gets refined from the initial iteration to the final iterations, corresponding to our description in Equation (7).

5 Conclusion

In this paper, we investigate the task of unsupervised face animation and propose a novel motion refinement approach to address the inadequacies of existing prior-based motion models for estimating finer facial motions. Our approach utilizes a structure correlation volume constructed from keypoint features, to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow estimated by a prior motion model. We conducted extensive experiments on challenging benchmarks, and our results demonstrate that our approach enhances the capability of prior-based motion representation through learning motion refinement.

Limitations: Amajor limitation of our approach is that it relies on the quality of the learned keypoints, as we mainly construct our proposed structure correlation volume using these keypoints. While current prior motion models may support the learning of significant keypoints, our method may still be limited by the quality of keypoints that are learned in an unsupervised fashion. To overcome this limitation, one potential solution would be to integrate model-based techniques that leverage facial priors, such as predefined keypoints that are more structured. Another limitation is the identity shift problem in cross-identity face animation, as keypoints often leak the face shape information of a driving video. The issue can be alleviated by employing relative motion transfer as in existing unsupervised animation methods [24, 25, 29]. Moreover, we could also explore prior motion models that can disentangle the global head motion and micro-expression motion, like facevid2vid [34].

Social impact: Our method has the potential to be used in creating deepfakes, which could have negative impacts. Therefore, individuals intending to use our technique for creating deepfakes should obtain authorization to use the respective human face images. Nevertheless, our method can also be utilized in creating imaginative image animations for entertainment purposes.

Acknowledgement: This work is supported by the National Natural Science Foundation of China (No. 62176047), Shenzhen Fundamental Research Program (No. JCYJ20220530164812027), and the Fundamental Research Funds for the Central Universities Grant (No. ZYGX2021YGLH208).

References

Brandon Amos, Bartosz Ludwiczuk, Mahadev Satyanarayanan, et al. Openface: A general-purpose face recognition library with mobile applications. CMU School of Computer Science , 6(2), 2016. 13

Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques , pages 187-194, 1999. 1

Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision , pages 1021-1030, 2017. 13

Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13786-13795, 2020. 2

Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14398-14407, 2021. 7, 13

Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 3d guided fine-grained face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9821-9830, 2019. 2

Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenactment preserving identity of unseen targets. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10893-10900, 2020. 1, 2

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 6

Fa-Ting Hong, Li Shen, and Dan Xu. Dagan++: Depth-aware generative adversarial network for talking head video generation. arXiv preprint arXiv:2305.06225 , 2023. 2

Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3397-3406, 2022. 2, 3, 7, 8

Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence , 17(1-3):185-203, 1981. 3

Junhwa Hur and Stefan Roth. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5754-5763, 2019. 2, 3

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision , pages 694-711. Springer, 2016. 6

Eshin Jolly, Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, Matthew Kenny, and Luke J Chang. Py-feat: Python facial expression analysis toolbox. arXiv preprint arXiv:2104.03509 , 2021. 13

Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt. Deep video portraits. ACM Transactions on Graphics (TOG) , 37(4):163, 2018. 2

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. 7

Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. Implicit warping for animation with image sets. arXiv preprint arXiv:2210.01794 , 2022. 3

Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612 , 2017. 1, 7

Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision , pages 483-499. Springer, 2016. 7

Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7184-7193, 2019. 1, 2

Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francesc Moreno-Noguer. Ganimation: Anatomically-aware facial animation from a single image. In Proceedings of the European conference on computer vision (ECCV) , pages 818-833, 2018. 2

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015. 7

Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2377-2386, 2019. 2, 13

Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances in Neural Information Processing Systems , 2019. 2, 3, 6, 7, 10, 13

Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In CVPR , 2021. 2, 3, 6, 7, 10

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

6

Deqing Sun, Charles Herrmann, Fitsum Reda, Michael Rubinstein, David Fleet, and William T Freeman. What makes raft better than pwc-net? arXiv preprint arXiv:2203.10712 , 2022. 3

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8934-8943, 2018. 2, 3

Jiale Tao, Biao Wang, Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. Motion transformer for unsupervised image animation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI , pages 702-719. Springer, 2022. 2, 3, 7, 8, 10, 13

Jiale Tao, Biao Wang, Borun Xu, Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. Structure-aware motion transfer with deformable anchor model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3637-3646, 2022. 2, 3, 7

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 402-419. Springer, 2020. 2, 3, 5, 13

Soumya Tripathy, Juho Kannala, and Esa Rahtu. Facegan: Facial attribute controllable reenactment gan. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1329-1338, 2021. 2

Qiulin Wang, Lu Zhang, and Bo Li. Safa: Structure aware face animation. In 2021 International Conference on 3D Vision (3DV) , pages 679-688. IEEE, 2021. 1, 2

Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10039-10049, 2021. 3, 10

Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. In International Conference on Learning Representations , 2022. 2, 7

Dongxu Wei, Xiaowei Xu, Haibin Shen, and Kejie Huang. C2f-fwn: Coarse-to-fine flow warping network for spatial-temporal consistent motion transfer. Proceedings of the AAAI Conference on Artificial Intelligence , 35(4):2852-2860, May 2021. 2

Yuxiang Wei, Ming Liu, Haolin Wang, Ruifeng Zhu, Guosheng Hu, and Wangmeng Zuo. Learning flow-based feature warping for face frontalization with illumination inconsistent supervision. In European Conference on Computer Vision , pages 558-574. Springer, 2020. 2

Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV) , pages 670-686, 2018. 2

Guangming Yao, Yi Yuan, Tianjia Shao, Shuang Li, Shanqi Liu, Yong Liu, Mengmeng Wang, and Kun Zhou. One-shot face reenactment using appearance adaptive normalization. Proceedings of the AAAI Conference on Artificial Intelligence , 35:3172-3180, May 2021. 2

Bohan Zeng, Boyu Liu, Hong Li, Xuhui Liu, Jianzhuang Liu, Dapeng Chen, Wei Peng, and Baochang Zhang. FNeVR: Neural volume rendering for face animation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 2, 3, 7

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018. 7

Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3657-3666, 2022. 2, 3, 7

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning to forecast and refine residual motion for image-to-video generation. In Proceedings of the European conference on computer vision (ECCV) , pages 387-403, 2018. 2, 8

Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII , pages 650-667. Springer, 2022. 1, 7

Appendix

In this appendix, we present detailed information on the architectural design, evaluation metrics, correlation pyramid, and parameter analysis on the number of keypoints and the number of iterations.

Architecture details: Our system mainly consists of a keypoint detector, a dense motion module, the source and driving structure encoders, a flow updater, and an image generator. The keypoint detector and dense motion modules are implemented using blocks similar to those used in previous works [23, 29]. In contrast, we provide architecture details of our source and driving structure encoders and image generator in Figure 8. All the modules are included in our provided codes for easy reference and implementation.

Evaluation metrics: We mainly introduce four metrics that utilized third-party models for evaluation.

Average Keypoint Distance (AKD [24]). This metric computes the average keypoint distance between generated and ground-truth images. It is designed to evaluate the pose quality of the generated images. We use existing detectors [3] to extract the facial landmarks.

Average Euclidean Distance (AED [24]). This metric is designed to assess the identity quality of generated images based on specific feature representations, that are extracted from a pre-trained facial identification network [1]. The average Euclidean distance between generated and groundtruth video frames is computed.

Average Rotation Distance (ARD [5]). We use the toolbox py-feat [14] to extract the Euler angles of the head poses, and then compute the average Euler angles distance between the generated and driving images. This metric evaluates the head pose quality.

Action Units Hamming distance (AUH [5]). This metric measures the quality of facial expression, it computes the average Hamming distance between action units of generated and driving images. We use the toolbox py-feat [14] to extract facial action units.

Details of the correlation pyramid: We pool the structure correlation volume C ∈ R h × w × h × w in the last two dimensions to obtain the pyramidal structure correlation volume { C i ∈ R h × w × h/ 2 i × w/ 2 i } P i =0 . In each iteration, we sample patch correlation features on all C i 's in the pyramid, as the pyramid design aims to capture more rich motion features of different scales, which is inspired by the optical flow method RAFT [31]. Both the patch radius and the pyramid level can expand the search space of the structure correlation volume, resulting in the expanded correlation feature dimensions. In the main paper's experiments, we empirically set P to 1 and r to 3 .

Number of keypoints: Following previous methods [24, 29], we configured the number of keypoints to 10 in the main paper. Here we conduct experiments to study this hyperparameter, with FOMM adopted as the prior motion model. As shown in Table 5, in a relatively sparse configuration such as 10-20 keypoints, our non-prior-based motion refinement approach can consistently improve the performance of the prior motion model.

Number of iterations: We set the iteration number according to the image resolutions in the main paper. Specifically, we start the iteration at a lower feature resolution H/ 32 × W/ 32 that is meaningful for motion flow for warping, and end the iteration at the highest resolution H × W , thus the total iterations are set to 6 for a 256-resolution image. By changing the highest resolution to H/ 2 × W/ 2 , H/ 4 × W/ 4 , H/ 8 × W/ 8 , we obtain iteration settings of 5 , 4 , 3 accordingly. We then analyze the performance and run time of different iteration settings. As seen in the table, the FPS goes down as the iteration number increases. At the same time the performance is generally better with higher iterations, especially for the motion-related metric AKD and identity-related metric AED. Overall, the 5-iteration setting can be a good trade-off.

DownBlock-64

DownBlock-128

DownBlock-256

DownBlock-512

DownBlock-512

UpBlock-512

UpBlock-256

UpBlock-128

UpBlock-64

UpBlock-32

Conv

1×1

-256

Input-10

DownBlock-64

DownBlock-128

DownBlock-256

DownBlock-512

DownBlock-512

UpBlock-512

UpBlock-256

UpBlock-128

UpBlock-64

UpBlock-32

Conv

1×1

-256

Input-13

DownBlock-128

DownBlock-256

DownBlock-512

DownBlock-512

DownBlock-512

ResUpBlock-512

ResUpBlock-512

ResUpBlock-256

ResUpBlock-128

ResUpBlock-64

Conv

7×7

-3

Input-3

SameBlock-64

Sigmoid-3

×(1 - 𝑂)

×(1 - 𝑂)

×(1 - 𝑂)

×(1 - 𝑂)

×𝑂

×𝑂

×𝑂

×𝑂

×𝑂

Sum-3

×𝑂

×(1 - 𝑂)

(a) Driving Structure Encoder

(b) Source Structure Encoder

(c) Image Generator

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper anaylizes the limitations of existing face animation methods in capturing the finer facial motions, and hence design a non-prior-based motion refinement approach to achieve finer face motion transfer in local areas. A correlation volume between the source and driving images is constructed as non-prior motion evidence, and a refinement module is introduced to generate the fine facial motions iteratively. Extensive experiments show the effectiveness of this paper.\n\nSTRENGTHS:\n1. The idea of involving a non-prior based motion refinement module is effective to capture the fine motions in local areas.\n2. The manuscript is well written and clearly states the main idea and contribution in face animation.\n3. This paper performs extensive experiments, and the results of the same-identity video reconstruction task outperform many previous methods.\n\nWEAKNESSES:\n1. This paper concentrates on the motion issue, but the video results of this paper are not impressive enough. No obvious improvements can be seen given the video results on the self reconstruction.\n2. This paper does not show the quantitative results in terms of some usual metrics, i.e. CSIM and FID, on the cross-identity reenactment task. I wonder the performance of this work in preserving identity.\n3. The introduction should be improved, as it only concludes limited contributions.\n\nQUESTIONS:\n1. Could the authors give more explations about the meanings of flow in the figures? I can not see the changes after updation. There are obvious improvements on the occlusion map, does the improvement in motions mainly come from the enhanced occlusion map.\n2. Since Face vid2vid [1] shows a promissing performance on the cross-identity reenactment task, it is suggested to give a comparison with Face vid2vid. It is good to provide the video comparisons.\n   \n   [1] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021.\n1. It is better to provide computation cost evaluation at inference, i.e. Flops, memory, and FPS, compared to other SOTA methods.\n\nLIMITATIONS:\nThis paper is effective to refine the motions to obtain better results, but is still limited to the quality of the learned keypoints, as the limitaions in the paper stated, which I think is more urgent to solve for unsupervised face animation. And this paper did not give comprehensive evaluations on the performance in identity preserving, which is important for face animation in practical applications. \nI'm not sure whether it is qualified for the conference.  If the authors could give reasonable response for the problems stated in **Weakness** and **Questions**, I would like to consider improving my rating.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper proposes a method for face animation via learning to refine a coarse motion field. The refinement is performed in a recurrent manner using the previous iteration's motion, occlusion map and structure correlation volume.  Both the qualitative and quantitative results show improvement over prior art.

STRENGTHS:
1) Qualitative results show that the proposed method is clearly model finer facial movement much better than prior art, especially around the eye. FNeVR is very close but the proposed method has fewer uncanny artifacts.

2) Quantitatively as well, the proposed method outperforms prior art.

3) The paper is well written.

WEAKNESSES:
1) Across all the results, I notice a strong identity shift during animation. While this is common (and seemingly worse) in prior art as well, it is important to quantify. One way to measure this is to report the FaceIDLoss or L2-loss as the head is rotated 15, 30, 60 degrees from the original head-pose. I believe this is an important evaluation to include for the sake of completeness.

QUESTIONS:
N/A

LIMITATIONS:
1) An important limitation of this method is that it is sensitive to the scale of the face of the driving video. For example, around the 1:47 of the supplementary video, the face structure of the animated face deviates significantly from the source and is actually closer to that of the target. This is unavoidable due to the use of 2D key points in the method, in fact it is expected. The authors must include this as a limitation of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes to the field of unsupervised face animation by introducing a novel motion refinement method to overcome the limitations of existing prior-based motion models, especially in estimating detailed facial movements.\n\nThe paper's approach introduces a new method which uses a structure correlation volume built from keypoint features. This provides motion information that does not rely on prior data. This information is used to iteratively refine the coarse motion flow estimated by a previous motion model.\n\nThe authors have conducted numerous experiments on challenging benchmarks to test the effectiveness of their approach. The results indicate that this new method enhances the capability of prior-based motion representation by learning how to refine motion. This suggests that their approach can effectively increase the accuracy of unsupervised face animation tasks.\n\nSTRENGTHS:\nIn this research, the authors present a new unsupervised face animation approach that concurrently learns both coarse (global) and finer (local) facial motions. Their method integrates a local affine motion model to learn the global, coarse facial motion and a novel motion refinement module to compensate for the local affine motion model's ability to model more detailed facial motions in local areas.\n\nThe motion refinement process is based on the dense correlation between the source and driving images. To achieve this, a structure correlation volume is first constructed using the keypoint features of the source and driving images. The authors then train a model to iteratively generate minor facial motions from low to high resolution.\n\nThe learned motion refinements are combined with the coarse motion to generate the new image. After performing extensive experiments on widely used benchmarks, the method was found to deliver the best results among existing state-of-the-art methods. The authors have also committed to making the source code for their method publicly available in the future.\n\nWEAKNESSES:\n1. The goal of this article is to learn fine facial motion, but using the VoxCeleb dataset may not be sufficient for this purpose. In my view, finer facial motion refers to the ability to reproduce wrinkles, eyeballs, and micro-expressions at high resolution, which may require higher resolution datasets.\n\n2. As noted by the author, the reliability of keypoints estimation heavily influences the quality of finer facial motion. It is not clear whether the improvement of existing models comes from more accurate keypoints estimation or the proposed module.\n\n3. This paper is very similar to RAFT from ECCV2020 in terms of insight and specifically for the module. A clearer comparison between the two works and their essential differences would be helpful.\n\n4. It is important to note that fine facial motion may not equal to better image quality. Therefore, a more thorough evaluation of the estimated motion would be beneficial. Additionally, conducting this task on higher resolution facial images would provide more convincing results.\n\n5. The author sets the iteration number to 8, which is an important hyperparameter. It would be useful to know why this setting was chosen and whether it significantly affects the model's runtime.\n\nQUESTIONS:\n1. This paper is very similar to RAFT from ECCV2020 in terms of insight and specifically for the module, which raises novelty concerns.\n\n2. Related works [1] are not discussed and compared in this paper.\n\n[1] Tao, Jiale, Biao Wang, Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. \"Motion Transformer for Unsupervised Image Animation.\" In European Conference on Computer Vision, pp. 702-719. Cham: Springer Nature Switzerland, 2022.\n\nLIMITATIONS:\nBoth limitations and social impact are discussed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an unsupervised method for learning to create continuous video animations of faces given a single input image and a driving video of the same or a different subject. The method relies on the optical flow between the driving and source images to warp the features of the source image, which are then passed to a generator to create the final output frames. The main contribution of the proposed method lies in the two-step formulation of the flow estimation, in which starting from a coarse prediction using a known prior-based method (Affine Transforms, Thin Plate Splines) it iteratively produces updates to reach a more fine-grained flow (and occlusion) estimation with more local details. The flow updates are learnt in an unsupervised way without using prior models, but by building a correlation volume between the source and driving images. The paper includes experiments, generated images and supplementary videos that validate the authors' claims for more detailed animations in comparison to similar methods.\n\nSTRENGTHS:\n- The paper is well written overall and it is easy to follow along with the presented concepts and results.\n- The paper includes experimental results that validate the authors' claims, as well as ablations that offer a better understanding about model design choices.\n- The method produces higher-quality generated images/videos than the compared methods for unsupervised learning of face animation.\n- The idea of using motivation from RAFT to built a similar correlation structure for estimating motion flow for face animation is interesting.\n\nWEAKNESSES:\n- Some design choices are not empirically or experimentally justified in the paper. For example, why are specifically 6 iterations used to update the initial coarse estimations? Why does it make sense to sample from the same correlation structure for all iterations, while each iteration operates at a different image scale? \n- Even though the method performs well compared to previous methods, there is still some evident identity shift in cross-subject experiments, meaning that the shape of the face slightly changes from the original to the one of the driving frame. This is evident in Figure 5 (for example 2nd row) as well as in the supplemental video. This effect makes sense because the estimated flow between different subjects possibly includes more deformation information, rather than only deformation because of expressions or rigid motion. Disentangling identity from other motions is a common weakness of cross-identity animation methods, which exists in this method, too.\n\nQUESTIONS:\n- In 3.4 it is mentioned that a perceptual and an equivariance loss are employed for training. Is the model trained with both same and different subjects for source and driving frames? \n- How are C^{i} combined to get the final values from the correlation pyramid? \n- What is the effect of the correlation pyramid to the memory requirements of the method? How does the memory requirements compare with the compared methods?\n\nLIMITATIONS:\nThe authors have included a section discussing the method's limitations and possible negative societal impact. A mentioned limitation (the correlation matrix relying on learned keypoints) might also be the reason for changing the facial shape in cross-subject animations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
1. This work proposed a  new unsupervised face animation approach to learn simultaneously the coarse and finer motions.
2. The results outperformed the state-of-the-art methods on two representative datasets.

STRENGTHS:
1. The target task is important.
2. The method is intuitive and reasonable.
3. The results are good.
4. The paper is well-written.

WEAKNESSES:
1. Compared to prior model. The proposed non-prior models seems like a non-linear version of prior model based on affine transformation. When the number of keypoints increasing and the grid of affine transformation being much finer, will it approach the proposed non-prior model? Another related question is that why the proposed model works better than a local thin-plate-spline motion model (such as [38])?

2. Visualization. 1) In the visualization of comparison in video, the results of the proposed model looks close to FNeVR. Authors could also provide user study in visual quality for comparison. 2) Also, the results on CelebV-HQ is really important, since it is a much challenging dataset. Visualization results on this dataset is necessary, in comparison, ablation study and video. 3) Why didn't report the results of FNeVR on CelebV-HQ on Table 1?

I am inclined to accept this paper, if the concerns can be solved.

QUESTIONS:
None.

LIMITATIONS:
Yes, limitations are discussed in the main paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new approach to generate human face videos based on a source image and a driving video, which simultaneously learning both coarse and finer facial motions. The proposed approach utilizes a structure correlation volume constructed from keypoint features to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow that is estimated by a prior motion model. \n\nThe main contributions of the paper are: \n1. A non-prior-based motion refinement approach to compensate for the inadequacy of existing prior-based motion models.\n2. Utilize the keypoint features to build a structure correlation volume that represents the structure correspondence between the source and driving images across all spatial locations.\n3. Extensive experiments on challenging benchmarks that demonstrate the effectiveness of the proposed approach in enhancing the capability of prior-based motion representation through learning the motion refinement.\n\nSTRENGTHS:\n1. The paper presents a novel approach to generate human face videos that simultaneously learns both coarse and finer facial motions. The proposed approach utilizes a structure correlation volume constructed from keypoint features to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow estimated by a prior motion model. The approach addresses a significant problem in the field of unsupervised face animation.\n\n2. The paper is of good quality, and the method proposed is clear. The authors provide a detailed description of the approach, including the structure correlation volume and the motion refinement module. The experimental results demonstrate the effectiveness of the proposed approach in enhancing the capability of prior-based motion representation through learning the motion refinement.\n\n3. The paper is well-written and easy to follow. The authors provide a clear and concise description of the proposed approach, including the key components and the experimental setup. The paper is well-organized, and the authors provide a clear summary of the contributions and limitations of the proposed approach.\n\n4. The proposed approach has significant implications for the field of unsupervised face animation which addresses an important problem in this field: the inadequacy of existing prior-based motion models to capture detailed facial motions. The proposed approach is effective in enhancing the capability of prior-based motion representation through learning the motion refinement. The approach has potential applications in creating imaginative image animations for entertainment purposes, but it also has the potential to be used in creating deepfakes, which could have negative impacts. The authors acknowledge this limitation and provide recommendations for future work.\n\nWEAKNESSES:\n1.\tThe pictures in Figure 1 are too small, especially the optical flow map is not clear enough, no obvious changes can be seen before and after refinement. Also, is the schematic diagram of affine transformation in Figure 1 drawn based on a real example? Why does the affine transformation change so much after refinement?\n\n2.\tStraightforward combination of existing techniques. The innovation of this paper is not enough, the main innovation point lies in the motion refinement module. However, the method of correlation volume and iteratively refine optical flow used in it is very similar to the correlation matrix and iteratively update in some flow estimation[1,2,3] and correspondence estimation[4,5] methods, and this set of process of first constructing a 4D correlation volume, and then iteratively updating and optimizing optical flow has also been used in some neural style transfer(NST) methods[6,7], but the author did not clarify the differences between the module they used in this paper and related modules in these NST methods.\n\n3.\tThere is an error in line 180 of the article: the author claims that the look up operation on the correlation volume is shown on the left side of Figure 2, but there is no corresponding content in Figure 2. Should Figure 2 be changed to Figure 3? \n\n4.\tIn the comparison video provided in the supplemental material, the video of each method is too small, and it is difficult to see the tiny facial deformation details. It is recommended to arrange the videos of each method in the form of a nine-square grid, and enlarge the size of each video window.\n\n5.\tThe experiments of verifying the effectiveness of the proposed method are insufficient. The framework used by your method is the framework of unsupervised image animation [8,9,10,11], which should be able to generate animation videos on any object category. In addition to human faces, this framework can also be applied to human bodies and animals. Previous unsupervised image animation methods [8,9,10,11] have also been tested on related datasets of different subjects, including TaiChiHD[9], TED-talks[10], and MGif[8]. Your method has only been tested on face-related datasets, but from your overall framework, I don't see any modules that restrict the method to only work on human face. Therefore, I think that the non-prior-based motion refinement module should be applied to datasets of different objects to further test its effectiveness.\n\n6.\tLack of quantitative experiments on the Cross-identity task, such as user study that have been used in previous methods [8,9,10,11].\n\nReferences\n\n[1] Dosovitskiy, Alexey, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. 2015. “FlowNet: Learning Optical Flow with Convolutional Networks.” In 2015 IEEE International Conference on Computer Vision (ICCV). doi:10.1109/iccv.2015.316.\n\n[2] Teed, Zachary, and Jia Deng. 2020. “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow.” In Computer Vision – ECCV 2020, Lecture Notes in Computer Science, 402–19. doi:10.1007/978-3-030-58536-5_24.\n\n[3] Xu, Haofei, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. 2022. “GMFlow: Learning Optical Flow via Global Matching.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr52688.2022.00795.\n\n[4] Kim, Seungryong, Stephen Lin, SangRyul Jeon, Dongbo Min, and Kwanghoon Sohn. 2018. “Recurrent Transformer Networks for Semantic Correspondence.” arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition, October.\n\n[5] Zhang, Pan, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. 2020. “Cross-Domain Correspondence Learning for Exemplar-Based Image Translation.” In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr42600.2020.00519.\n\n[6] Liu, Xiaochang, Xuanyi Li, Ming-Ming Cheng, and Peter Hall. 2020. “Geometric Style Transfer.” Cornell University - arXiv,Cornell University - arXiv, July.\n\n[7] Liu, Xiao-Chang, Yong-Liang Yang, and Peter Hall. 2021. “Learning to Warp for Style Transfer.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.00370.\n\n[8] Siarohin, Aliaksandr, Stephane Lathuiliere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. “Animating Arbitrary Objects via Deep Motion Transfer.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr.2019.00248.\n\n[9] Siarohin, Aliaksandr, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. “First Order Motion Model for Image Animation.” Neural Information Processing Systems,Neural Information Processing Systems, January.\n\n[10] Siarohin, Aliaksandr, Oliver J. Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. 2021. “Motion Representations for Articulated Animation.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.01344.\n\n[11] Zhao, Jian, and Hui Zhang. n.d. “Thin-Plate Spline Motion Model for Image Animation.”\n\nQUESTIONS:\nIt’s suggested that the author should clarify the differences between the module they used in this paper and the related modules in some NST methods [6,7]. Also, experiments on more datasets of different object categories are needed.\n\nLIMITATIONS:\nYes, they have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-18"}