Learning Motion Refinement for Unsupervised Face Animation
Authors: Jiale Tao
Shuhang Gu
Wen Li
Lixin Duan
Venue: NeurIPS 2023 poster (2023)
Keywords: Face animation
Motion refinement
Structure correlation

Abstract: Unsupervised face animation aims to generate a human face video based on the
appearance of a source image, mimicking the motion from a driving video. Existing
methods typically adopted a prior-based motion model (e.g., the local affine motion
model or the local thin-plate-spline motion model). While it is able to capture
the coarse facial motion, artifacts can often be observed around the tiny motion
in local areas (e.g., lips and eyes), due to the limited ability of these methods
to model the finer facial motions. In this work, we design a new unsupervised
face animation approach to learn simultaneously the coarse and finer motions. In
particular, while exploiting the local affine motion model to learn the global coarse
facial motion, we design a novel motion refinement module to compensate for
the local affine motion model for modeling finer face motions in local areas. The
motion refinement is learned from the dense correlation between the source and
driving images. Specifically, we first construct a structure correlation volume based
on the keypoint features of the source and driving images. Then, we train a model
to generate the tiny facial motions iteratively from low to high resolution. The
learned motion refinements are combined with the coarse motion to generate the
new image. Extensive experiments on widely used benchmarks demonstrate that
our method achieves the best results among state-of-the-art baselines.

Learning Motion Refinement for Unsupervised Face Animation
Jiale Tao Shuhang Gu
Wen Li Lixin Duan 1.2
School of Computer Science and Engineering, UESTC! Shenzhen Institute for Advanced Study, UESTC² (jialetao.std, shuhanggu, liwenbnu, Ixduan)@gmail.com
Abstract
Unsupervised face animation aims generate human face video based on the appearance of source image, mimicking the motion from driving video. Existing methods typically adopted prior-based motion model (e.g.. the local affine motion model or the local thin- plat-plat--pplnee motion model). While it is able to capture the coarse facial motion, artifacts can often be observed around the tiny motion in local areas (e.g., lips and eyes), due to the limited ability of these methods model the finer facial motions. In this work, design face animation approach to learn simultaneously the coarse and finer motions. partieular, while exploiting the local affine motion model to learn the global coarse facial motion, we design a novel motion refinement module to compensate for the local affine motion model for modeling finer face motions in local areas. The motion refinement is learned from the dense correlation between the source and driving images. Specifically,
unsupervised
first construct a structure correlation volume based on the keypoint features of the source and driving images. Then, we train a model generate the tiny facial motions iteratively from low to high resolution. The learned motion refinements are combined with the coarse motion to generate the new image. Extensive experiments on widely used benchmarks demonstrate that our method achieves the best results among state-of of-the-art baselines.
Introduction
Face animation aims to generate a human face video based on the appearance of a source image to mimic the motion from driving video. has gained increasing attention from the community in recent years, due to its great potential in various applications including face swapping, digital humans, video conferencing, and more. Additionally, the emergence large-scale face video datasets [18, 44] has further contributed to this growing interest.
Existing works on face animation can generally be divided into two categories: model-based and model-free methods (a.k.a supervised and unsupervised methods). On one hand, model-based methods [7, 33] typically use predefined 2D or 3D facial priors, such as 2D landmarks [20] and 3DMM parameters [2], to provide pose and expression information in the generated videos. While these methods have the advantage of describing accurate face poses, they are limited by overly strict structure priors that are not robust to hair and neck motions, which can heavily affect their performance more challenging and
real-world applications. On the other hand, model-free animation interesting problem, which aims to automatically learn the motion patterns without using predefined 2D or 3D facial models, thus being more flexible and robust in real-world scenarios. With the emergence of large-scale face video datasets, model-free methods have become more popular in
"Codes will be available at http://giinabcoo//aaeeTaaaMRRrR
Thee Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Refinement
Figure 1: Illustration of the non-prior based motion refinement. Initially, prior motion such as an affine transformation defines motion flow. However, after the refinment, this flow can be matched to new deformation that more realistically models underlying face deformations.
recent years [24, 25]. Among model-free methods, a local prior motion model is usually assumed to transform the sparse corresponding keypoints the dense motion flow between the source and driving image. More recently, researchers have explored combining the two types of methods [40], such as using prior motion models learn motion flow for warping source features while simultaneously employing 3DMM parameters to refine and enhance warped features.
In this work, focus the model-free face animation task, considering its flexibility for practical applications. Existing model-free face animation methods typically operate under the assumption of local prior motion model, that is to assume the local motion between source and driving image generally follows a parametric model such as affine transformation. For example, Siarohin et al. [24] proposed local affine motion model, Siarohin al. [25] and Tao et al. [29, 30] further constrained the learning process of the affine matrix, while others proposed a local thin-plate-spe model [43]. While these methods have the advantage of facilitating the process of learning motion
motion
and the ability to learn significant keypoints, their limitations are also obvious. Firstly, assuming that the human face is locally rigid may not be appropriate, resulting in sub-optimal motion models that fail to capture tiny motions in local facial areas. Secondly, accurately predicting geometric parameters such as the affine matrix from a single image not easy. To address these issues, we propose a non non-priors con-prior-basee motion refinement approach to compensate for the inadequacy of existing prior-based motion models as illustrated in Fig. Specifically, inspired by recent advances in optical flow [12, 28, 31], we utilize the keypoint features in building correlation volume between the source and driving images, which represents the structure correspondence between the two images across all spatial locations. This correlation volume serves as non non-prior motion evidence, according to which we employ a model to iteratively generate finer motions that refine the coarse motion predicted by prior motion models. In summary, compared to existing prior-based motion models, our approach exhibits a more powerful motion representation terms of learning finer motions and thus is capable of achieving more realistic face animations.
We conduct extensive experiments on challenging face video benchmarks, and the superior perfor- mance of our method over sfate-of-the-arr art baselines highlights that learning motion refinement is advantageous for enhancing the existing prior-based motion models on face animation.
2 Related work
Face animation: Recent works on face animation can be roughly categorized into model-based and model-free methods, and more recently some works [40] tried to combine the two types of methods.
Model-based methods [4, 6, 7, 15, 21, 32, 36, 37, 38, 39] leverage predefined facial prior, such as 2D landmarks and 3D face models, to guide the pose expression of the generated output. For instance, FSGAN [20] used facial landmarks as input to a generator directly to reenact source face. Similarly, SAFA [33] used 3D morphable model to aid in expression transfer. These methods are advantagcous in modeling accurate face poses but suffer from the too-strict structure prior, which is not robust to neck and hair motions that are prevalent in practical scenarios.
In contrast, model-free methods [9, 10, 23, 24, 25, 30, 35, 42] tend to have better generalization performance and have become more popular due to the advent of large-scale face video datasets. These methods assume local prior motion between subjects and learn local keypoints and corre- sponding motion transformation parameters in an unsupervised manner from videos. FOMM [24] is a pioneer work that proposed a local affine transformation model for image animation. Subsequently, MRAA [25]. DAM [30], and MTIA [29] constrained the affine motion model learning process with stronger prior, such as PCA decomposition and cooperative motions. In fact, these methods assume a locally rigid object and mainly apply to human bodies, having limited performance

---

human faces due to the easily violated locally rigid assumption. TPSM [42] proposed to adopt local can--laat-sppine motions for general animations, while DaGAN [10] used face depth information to better learn the keypoints and affine matrices. However, these methods still fall into the paradigm of prior-based motion models, which have difficulty in modeling tiny motions local face areas. There are also some works [17, 34] that extremely abandoned the prior-based motion representation, they either used 3D head rotation supervision or utilized multiple source images for compensating the loss of motion prior. In this paper, we try to push the Jimit of single-image-ber basee unsupervised face animation and propose a non-prior-basee
motion refinement approach, which overcomes the inadequacy of existing prior- based motion models by learning the finer motions.
Optical flow: Optical flow estimation has been longstanding research problem in computer vision [11, 12, 27, 28]. Its objective is to estimate the motion between consecutive video frames. It's worth noting that the key part of unsupervised facial animation is also to estimate the motion flow between two images of the face. The main difference between the two tasks lies in the fact that optical flow deals with frames depicting the same scene, whereas facial animation focuses on transferring the motion between video frames featuring different subjects. This fundamental difference leads to variations in the methods employed for the two tasks. In more detail, optical flow computes the visual feature similarities between two frames. On the other hand, facial animation aims to disentangle the structural and visual appearance features and uses the structural feature to establish motion representations. Our method is inspired by the optical flow technique named RAFT [31]. Its fundamental idea is to develop a 4D correlation volume based on the visual features of two frames and use to iteratively refine the motion flow. Similar to this concept, we build structural correlation volume in face animation based on keypoint features; moreover, we introduce a multi-scale refinement mechanism to maximally leverage the source features, which helps capture finer face motions.
3 Method
Our method consists of four key components: (1) a coarse motion estimation module based on a prior motion model, (2) structure correlation volume calculator, (3) a motion flow updater, and (4) an image generator. While the first and last components are similar to existing face animation methods [24, 25, 29, 30], the second and third components form our proposed non-prior- bradd motion refinement module. The overview of our method is presented in Fig. 2. Similar to existing methods [24, 25, 29, 30] for achieving face animation, we first estimate the motion flow between the driving and source images. Then, we use the motion flow to warp the source features which is decoded by the image decoder, and generate the final animation video frame by frame. The difference is that in addition to estimating a coarse motion flow based on prior motion model, we further estimate the motion refinement by our proposed motion refinement module. The coarse motion and the motion refinement are combined to warp source features. In the following subsections, we will first describe the existing prior-bace based motion models and then introduce our proposed non-prior-bant based motion refinement module.
3.1 Prior based Motion Estimation
Existing methods [24, 25, 29, 30, 40, 42] generally employ the affine prior and thin plate spline prior to model local facial motion. For the sake of simplicity, we adopt the local affine motion model as our formulation. This model assumes that a human face is composed of several individual parts and each part's motion defined by an affine transformation. The parameters of these affine transformations are predicted by keypoint detector, as shown in Fig. 2(a). The local affine transformations are then transformed into a set of part motion flows. A dense motion network predicts a composition mask, which is used to weigh the set of part motion flows to obtain single motion flow, and an occlusion map which is used to mask the warped source feature in the occluded area.
Formally, we denote by S and D the source and driving images. A set of keypoints {p..r. r } E 2 and corresponding affine parameters { A5) E R2*2 are predicted by the keypoint detector, where & 1,..., K. These affine parameters predicted from a single image are assumed to be the transformation to abstract reference image [24]. Additionally, let E 10. h x w 1} denote any pixel coordinate of the driving image, where h and w define the resolution of the motion flow which in this stage is set to 4 times downsampled image resolution. The part motion flow between
3

---

( Prior based coarse motion estimation
Image generation
Keypoint Detector
Image Encoder
Image Decoder
Motion
Updated flow warping
(b) Non-prior based motion refinement
Driving
Correlation Volume
Inittal Flow
Updated Flow
Updated Flow
Stracture
Encoder
Flow Updater
Flow Updater
Flow Updater
Structure
Encoder
Souree Feature
Souree Feature
Source Feature
Figure 2: Overview of the pipeline. Our method consists three modules: (a) Prior-based coarse motion estimation module first estimates coarse motion flow based prior motion model. (b) Our proposed non-prior based motion refinement module constructs 4D structure correlation volume that provides the non-prior motion evidence, based which the coarse motion flow is iteratively refined. (c) The image generation module for both encoding multi-scale source features and decoding multi-scale warped source features.
the driving and source images is derived using the following formula:
(z) = p (A5)
Intuitively, Equation (1) represents the flow format of an affine transformation that displays the coordinate correspondence between images. In addition the K part motion flows, motion flow of the background, which typically considered to be static, is also represented by:
T TT
Using the predicted composition mask M R the driving and source image as follows:
obtain the motion flow between
K Σ M²(2) Th Ts))(=), k=
Ts)
We treat the Equation (3)
estimation of the motion flow, and in the following sections, denote it using Fo Ts-ppi D without special indication. Additionally, we also denote O₂ as the occlusion map estimated by the prior motion model. Note that derived Fo and Og by the affine prior, it is also suitable to use the thin plate spline prior for estimating the coarse motion flow.
3.2 Non-prior based Motion Refinement
Prior-based motion models have the advantage of facilitating learning motion in the early stages and can learn meaningful keypoints. However, the prior assumption such as a locally rigid face may not be appropriate, limiting them learning finer facial motions. In this subsection, our non non-priore baser motion refinement approach that compensates for the limitations of prior-based motion models.
present
Our motion refinement process is iterative, using the initial flow estimation obtained from prior motion model, as described in the previous subsection. First, we construct a structure correlation volume to represent pixel-level structure correspondence between driving and source images. This volume remains fixed throughout later iterations. For each iteration, use the current flow estimation to access the correlation volume and extract neighboring patch features. The cut correlation values, along with the warped source feature and current flow estimation, sent to the flow updater to

---

Convs
Comvs
Concat
Convs
Sum
Lookup
Updated Flow & Occlusion
Source
Convs
Warp
Feature
Current Flow
Current Flow
Current Flow & Occlasioon
Figure 3: Details of the motion flow updater. It outputs the updated motion flow and occlusion map by processing three types of input features: warped source image feature, correlation feature extracted from the structure correlation volume using lookup operation, and the current motion flow. The correlation feature and input motion flow are passed through two separate convolution modules, and their results concatenated to yield an intermediate motion feature. This intermediate feature, together with the imput motion flow and the warped source feature, is combined and input to convolution module, which produces the residual flow and occlusion map.
obtain a refined flow. Note that the iteration proceeds from low high resolutions, thereby enhancing the ability to capture finer facial movements. Fig. 3 provides more detailed depiction of a single iteration of our motion flow update procedure. We will now introduce each component and operation.
Structure correlation volume: Inspired by recent advancements in optical flow [31], we propose similar approach of constructing structure correlation volume, which is considered to provide non-prior motion evidence for refining coarse motion flow. We accomplish this by first encoding the source and driving structure features with two separate encoders. The driving structure encoder takes only the driving keypoints input, while the source structure encoder concatenates the times downsampled source image and keypoints as input. The source image is considered to provide dense visual context information for the sparse keypoints, which helps to densify the source structure. Keypoints are encoded with fixed variance using Gaussian heatmaps centered on their positions. After encoding, the driving and source structure features are multiplied along the channel dimension to obtain a 4D structure correlation volume C' Rx xhxxeeee x The first two dimensions align with the
driving coordinate system, while the last two represent the source coordinate system. Each indexed value in the first two dimensions represents the correspondence confidence between the driving pixel position and all source pixels. Note that the calculation of the structure correlation volume is carried out at resolution that is 4 times lower than that of the original image. We also used a pyramid of correlations, which is similar the optical flow approach [31]. The pyramid is created by pooling the structure correlation volume repeatedly with series of scale factors. Further details can be found in the supplementary material.
Correlation lookup: For each iteration the updates, we utilize the current input motion flow to lookup a {(2r + 1) x (2r 1at-h correspondence values centered at the flow position, where the radius of the patch. This patch feature is further flattened to the channel dimension, resulting in h2 w; x (2r + 2 correlation feature C, where h; and w; the flow resolutions of the i'th iteration. The lookup process is depicted on the left side of Fig. 3 and the correlation features are denoted by the yellow cuboid.
Motion flow update: The motion flow updater is comprised of four convolution modules that share parameters across all iterations. During each iteration, the flow updater takes in the current motion flow, extracted correlation values, and the warped source feature as inputs to produce outputs of updated motion flow and occlusion map.
To be more specific, we represent the current input motion flow as FL_ which is upsampled twice from the previous iteration's output. The correlation feature is represented by C, and the source input feature encoded by the image encoder is represented by fi. Firstly, the residual flow dF, and the residual occlusion map dO, are generated by the flow updater tb:
dF, C,
(4)
where F(-0 denotes the warping operation. The update process is then formulated as follows:
F F== + dF, O, do,,
(5)

---

We begin the iteration process at the resolution of the image downsampled by 32 times. The initial input motion flow for the first iteration is provided by:
Fi Downsample(Fe. 8),
(6)
as Fo is computed at resolution H/4 x W/4. The iteration is stopped once the maximal resolution iteration is accomplished, i.c., the image size. It should be noted that the initial flow input, FJ. loses significant amount of information due to the downsampling operation. To ensure preserving as much information from Fo as possible, we reformulate Equation (5) as:
Fi = Resize(FF) + dF, + d + d +...
(7)
where the initial flow Fo is directly resized the current resolution. The similar rule is also applied to the occlusion map.
To summarize, the non-prior-based motion refinement module yields multi-scale refined motion flows and occlusion maps (F,, 1 which are utilized the process of image generation for warping source features.
Non-prior-based initialization: As previously discussed, have been utilizing a prior motion model to estimate coarse motion flow owing to its effectiveness in facilitating learning motion. However, we have also proposed a non-prior con-prior-based based initialization approach. Specifically, we reshape the structure correlation volume to h x w x (h x w) and denote the reshaped C' as C". We then perform softmax operation on the final dimension, treating the result as an attention matrix. This matrix is subsequently employed to sum up an identity grid. The non non-prior-boe prior-based initialization is formulated as follows:
Ax
2se (z)
Σ i Softmax (C")
(8)
Where indexes the driving spatial dimensions and denotes the reshaped source dimension. In this way our proposed method becomes totally non-prior-based, we study this initialization in Section 4.
3.3 Image generation
As previously mentioned, we have prepared multi-scale motion flows, occlusion maps, and source features [F,, O, fi J To better synthesize facial details, we use a Unet-like image generator which effectively combines the multi-scale source features with the refined multi-scale motion flows. In each layer of the image decoder, we integrate information from the current warping and previous generation by multiplying them with the occlusion map and reversed occlusion map. The generation process can be summarized as follows:
out (ii) O
out,+1 = F(-((())) 0++1 + UpBolck( (oooBookcoot)))) (1 O++1),
(10)
where UpBolck defines a upsample-conv operation and ResBolck denotes a 2-layer resnet [8] block. At the last layer, the output is activated by a sigmoid function to obtain the generated image.
3.4 Training
Our method is trained end end. Following previous methods [24, 25] and for the sake of simplicity, we only adopt the perceptual loss and the equivariance loss as our objective functions.
Perceptual loss: We adopt the multi-resolution perceptual loss [13] defined with a pre-trained VGG-19 [26] network. Given the driving image D with resolution index i, the generated image D, and the feature extractor with layer index 1, the perceptual loss can be written as follows:
Per ΣΣ
oo (D.) (((D))),
(11)
Equivariance loss: Given random geometric transformation T and driving image D, the equivariance loss can be written as follows:
Lequi
Σ ((rb)-ri(
(12)
The overall loss is formulated by the sum of the two without weighting:
L Eper + Lequi
(13)

---

Table I: Quantitative comparisons with state-of-the-art methods the video self reffssccontttitionn task. We present results on the Voxcelebl and Ce datasets. Our method generally achieves the best performance on all evaluation metrics.
Voxcelebl
CelebV-HQ LPIPS
L1
PSNR
LPIPS
AKD
AED
L1
PSNR
AKD
AED
FOMM
0.0412
23.85
0.171
1.284
0.135
0.0531
22.95
0.204
3,491
0.219
MRAA
0.0394
24.47
0.166
1.274
0.132
0.0433
24.65
0.173
1.852
0.167
LIA
0.0425
23.64
0.212
1.457
0.138
0.0507
22.95
0.235
1.982
0.179
DAM
0.0395
24.51
0.165
1.242
0.124
0.0496
23.58
0.189
1.899
0.180
DaGAN
0.0422
24.03
0.168
1.265
0.125
0.0652
21.37
0.249
8.704
0.307
TPSM
0.0396
24.84
0.160
1,203
0.122
0.0412
25.29
0.160
1.663
0.156
MTIA
0.0370
25.09
0.159
1.190
0.120
0.0405
25.63
0.157
1.532
0.153
FN
0.0404
24.36
0.165
1.238
0.126
Ours
0.0354
25.57
0.151
1.155
0.108
0.0376
26.11
0.147
1.424
0.137
Table 2:
Cross-idente evaluation on the Voxceleb1 dataset.
FOMM
MRAA
LIA
DAM
DaGAN
TPSM
MTTA
FNeVR
Ours
ARD
3.122
2.678
3.883
2.669
3.090
2.724
2.794
2.755
2.399
AUH
0.850
0.729
0.772
0.717
0.751
0.668
0.676
0.691
0.647
FID
70.00
69.29
71.01
69.16
68.50
68.67
66.65
66.48
64.68
4 Experiments
Implementation details: We adopt two Hourglass networks [19] for the keypoint detector and the dense motion network, and Unet [22] for the image generator similar to previous works [24, 25]. We further employ two hourglass networks for the driving and source structure encoders. The flow updater consists of four convolution blocks that share the parameters across the iterations. We train our method for 100 epochs four NVIDIA A A100 GPU cards or eight NVIDIA 3090 GPU cards. It takes about 24 hours for training. The number of keyoints is set to 10 following previous methods [24, 29, 30, 42]. The patch radius r is
3 and the number of iterations is set to 6. The Adam optimizer [16] is adopted with 3 = 0.5 and 32 = 0.999, the initial learning rate is set as 2 x 10 4 and dropped by factor of 10 the end of 60th and 90th epoch. We leave the architecture details in the supplementary material.
Datasets: We conduct experiments the widely used Voxceleb1 [18] dataset and the recently collected more challenged CelebV-HQ dataset [44]. Voxcelebl is a talking head dataset consisting of 20047 videos, among which 19522 are used for training and 525 are used for testing. The Ce dataset consists of 35666 video clips, we randomly choose 500 of them for testing. All videos are resized 256 256 for fair comparison with existing methods.
We measure the reconstruction quality using L1, Peak Signal-to-Noise Ratio (PSNR), and LPIPS [41] following [35, 40J. To evaluate the transferred motion quality adopt Average Keypoint Distance (AKD) following previous methods [24, 25, 30, 42]. The identity quality of the generated videos is measured by Average Euclidean Distance (AED) [24, 25]. We use FID, ARD, and AUH metrics for evaluating coossiidenttyy face animation following [5]. All these metrics are the lower the better except the PSNR.
Metrics:
Baselines: A bunch baseline methods are compared, including FOMM [24], MRAA [25], LIA [35], DAM [30], DaGAN [10], TPSM [42], MTIA [29] and FNeVR [40]. All reported baseline results are obtained by evaluating the checkpoints from their Github repo. or retrain the provided codes, except that the VN does provide the training code and thus we do not report their results the V-H dataset. It is worth noting that most of these baseline methods adopt a prior motion model, while DaGAN [10] further explored the face depth information and FNeVR leverage the 3DMM parameters. For comparison with them, we adopt the motion transformer [29] for our prior motion estimation, while we also employ the affine motion model [24] and thin main-plate-spiig motion model [42] in the section of model ablations.
Quantitative comparison: Table I displays the same-identity reconstruction results, which is the standard evaluation setting used in previous methods [10, 24, 25, 29, 30, 40, 42] that reconstruct driving video based on its first-frame appearance and all-frame motion. As can be seen, our
7

---

Table 3: Model ablations the proposed non- non-prior based motion refinement module. We present results on the Voxcelebl dataset. Baseline methods are all based on our implementation. which differentiates from original methods with multi-scale feature fusion mechanism as described in Equation (10).
L1
PSNR
LPIPS
AKD
AED
NPMR
0.0393
24.41
0.164
1.328
0.133
FOMM
0.0386
24.62
0.164
1.254
0.124
FOMM NPMR
0.0367
25.15
0.156
1.224
0.114
TPSM
0.0370
25.09
0.159
1.231
0.120
TPSM NPMR
0.0353
25.51
0.152
1.176
0.107
MTIA
0.0370
23.09
0.159
1.190
0.120
MTIA NPMR
0.0354
25.57
0.151
1.155
0.108
Source
Driving
FOMM
MRAA
LIA
DAM
DxGAN
TPSM
ENeVR
Ours
Same-identity video reconstruction on the Voxceleb1 dataset. Our method generally performs the best among a variety baselines.
Figure 4:
method outperforms other methods across all metrics on the two datasets. Specifically, large-margin improvements on the reconstruction metrics (L.1, PSNR, LPIPS) demonstrate the advantages of learning motion refinement. What's more, method achieved the best motion quality (the best score on the AKD metric) particularly on the more challenging CelebV-HQ dataset, highlighting the powerful motion representation capabilities of our non non-prior- an---rrrr-bsedd motion refinement approach. The superior generated motion quality our method is further validated by the best ARD and AUH metric in cross cross-identity experiments in Table 2. The best AED score and FID score also indicate that our method preserves the source face identity well during motion transfer. These significant observations are further supported by the qualitative results presented in Fig. 4, Fig. 5, and Fig. 6.
Qualitative comparison: We conducted experiments on both the same-identity video reconstruction and cross-identity face reenactment similar to previous methods [10, 29, 43), showcasing our qualita- tive results in Fig. 4, Fig. 5, and Fig. 6. Our method consistently demonstrates greater robustness against large motions (the first row of Fig. 4), detailed face deformations (the third row of Fig. and the fourth row of Fig. 6), occlusions (the second and last row of Fig. 4), and varying light approach outperforms other methods
conditions (the fourth row of the Fig. 4), etc. Additionally, in generating precise facial details such as the eyes and lips, while avoiding common artifacts. To sum up, methods effectively learn non-prior priior-bosedd motion refinement, exhibiting significant improvement over prior- based motion representation that is limited on learning finer facial motions.
We performed model ablation study to further evaluate the proposed Non-Prion based Motion Refinement module. previously discussed in Equation (8), in the non-prior-bee based initialization process, initialize the motion flow using the structure correlation volume itself. We
Model ablations:

---

Driving
FOMM
DaGAN
FN
Figure 5: Cross-identity face reenactment on the Voxcelebl dataset, our method consistently demon- strates superior performance in generating high-fidelity results compared to existing methods.
Driving
Figure Cross-identity face reenactment on the CelebV-HQ dataset, our method consistently demonstrates superior performance in generating high-fidelity results compared to existing methods.
denoted this model variant as NPMR. As shown in Table 3, our NPMR -only model did not yield good results, highlighting the importance of prior-based initial motion estimation for learning finer facial motions and the crucial role it plays in our non non-prior-basee combined with specific prior motion estimations, such as that from the affine and thin plate spline motion models, our method showed promising improvements over the original prior motion models. These combinations with different prior motion models significantly demonstrate the flexibility of the proposed non-prior-based motion refinement module, and its strong ability to promote the enhanced motion representation over existing prior motion models.
motion refinement module. When
Component ablations: We conducted a thorough examination of the inputs and outputs of our non-prior-base based motion refinement module through detailed ablations. Specifically, we explored the impacts of the model variants of the source structure encoder without the input of a source image, the flow updater without the input of warped source features, and the flow updater without the output of occlusion or flow (i.e. only updating either occlusion or flow and keeping the other the same as in the initialization process). The results are presented in Table 4. It is noteworthy that both the warped source feature and the source image play an important role in the motion refinement process as demonstrated by the decreased AKD and AED metric values on w/o warped source feature and w/o source image variants. Importantly, if we solely refine the occlusion map without updating the motion flow (and vice versa), a significant decrease in AKD and AED metries occurs, validating the significance of our motivation to refine the coarse motion flow.

---

Table 4: Component ablations of the inputs and outputs of the proposed non-prior-based motion
refinement module. We present results on the Voxceleb1 dataset.
LI
PSNR
LPIPS
AKD
AED
w/o o flow
0.0362
25.48
0.155
1.172
0.112
w/ occlusion
0.0356
25.48
0.152
1.175
0.112
w/ warped source feature
0.0357
25.52
0.153
1.160
0.110
w/ souree image
0.0354
25.45
0.152
1.165
0.113
Ours full
0.0354
25.57
0.151
1.155
0.108
Source
Driving
Refined motion flow
Figure 7: Motion flow visualization of the refining process, show refined motion flows of all iterations. Note that flows of different resolutions are resized to the image resolution.
Motion flow visualizations: To further understand the motion refinement process, we visualize the refined motion flows in different iterations. As can be seen in Fig. 7, the motion flow gets refined from the initial iteration to the final iterations, corresponding to our description in Equation (7).
5 Conclusion
In this paper, we investigate the task of unsupervised face animation and propose a novel motion refinement approach address the inadequacies of existing prior-based motion models for estimating finer facial motions. Our approach utilizes a structure correlation volume constructed from keypoint features, provide non-prior-based motion information, which used to iteratively refine the coarse motion flow estimated by a prior motion model. We conducted extensive experiments on challenging benchmarks, and results demonstrate that approach enhances the capability of prior-based motion representation through learning motion refinement.
Limitations: A major limitation of our approach is that it relies the quality of the learned keypoints, as we mainly construct our proposed structure correlation volume using these keypoints. While current prior motion models may support the learning of significant keypoints, method may still be limited by the quality of keypoints that are learned an unsupervised fashion. To overcome this limitation, one potential solution would be integrate model-based techniques that leverage facial priors, such as predefined keypoints that are more structured. Another limitation is the identity shift problem in cross-identity face animation, keypoints often leak the face shape information of a driving video. The issue can be alleviated by employing relative motion transfer as in existing unsupervised animation methods [24, 25, 29]. Moreover, could also explore prior motion models that can disentangle the global head motion and micro-expression motion, like facevid2vid [34].
Social impact: Our method has the potential to be used in creating deepfakes, which could have negative impacts. Therefore, individuals intending to use our technique for creating deepfakes should obtain authorization
the respective human face images. Nevertheless, method can also be utilized in creating imaginative image animations for entertainment purposes.
Acknowledgement: This work is supported by the National Natural Science Foundation of China (No. 621 62176047), Shenzhen Fundamental Research Program (No. JCYI200200520114411200777 and the Fundamental Research Funds for the Central Universities Grant (No. ZYGX2021YGLH208)
10

---

References
[1] Brandon Amos, Bartosz Ludwiczuk, Mahadev Satyanarayanan, et al. Openface: A general-purpost face
recognition library with mobile applications, CMU School of Computer Science, 6(2). 2016. 13
[2] Volker Blanz and Thomas Vetter. A morphable model for the symthesis of 3d faces. In Proceedings of the
26th annual conference on Computer graphics and interactive techniques, pages 187-194, 1999.
[3] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem??and a dataset of 230,000 3d facial landmarks). In Proceedings of the IEEE International
Conference on Computer Vision, pages 1021-1030, 2017. 13
[4] Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13786-13795, 2020. 2
[5] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriaa Sharmanska. Headgan: Ohe-shot neural head synthesis and editing. In Proceedings of the IEEF/CVF International Conference on Computer Vision,
pages 14398-14407, 2021. 7, 13
[6] Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 3d guided fine-grained face manipulation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9821-9838 2019, 2 [7] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenactment preserving identity of unseen targets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10893-10900, 2020. 1, 1,2
[8] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[9] Fa- Fa-Ting Hong. Li Shen, and Dan Xu. Dagan++; Depth-aware generative adversarial network for talking head video generation. arXiv preprini arXiv:2305. 00225.622.... 2023.
[10] Fa-Ting Hong. Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 3397-3406, 2022. 2, 2,3,77 7, 8
[11] Berthold KP Horn and Brian G Schunck. Deternining optical flow. Artificial intelligence, 17(1-3):185-25)
1981. 3
[12] Junhwa Hur and Stefan Roth. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5754-5763,
2019. 3.
[13] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pages 694-711. Springer, 2016. 6
[14] Eshin Jolly. Jin Hyun Cheong. Tiankang Xie, Sophie Byrne Matthew Kenny. and Luke J Chang. Py-feat: Python facial expression analysis toolbox. arXiv preprint arXiv:2104, 03509, 2021. 13
[15] Hycongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thics, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt. Deep video portraits. ACM Transactions on Graphics (TOG), (7()):163, 2018.
[16] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego,
CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[17] Arun Mallya, Ting-Chun Wang, and Ming- Yu Liu. Implicit warping for animation with image sets. arXiv preprint arXiv:2210.01770
2022. 3
[18] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identification dataset. arXiv preprint /7Xiv:/706. 08612, 22, 2017. 1.7
[19] Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In
European conference computer vision, pages 483-499. Sprineer 2016. 7
[20] Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2184-7193, 2019. 1,2
[21] Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francese Moreno-Noguer Ganimation: Anatomically-awe aware facial animation from single image. In Proceedings of the European
conference on computer vision (ECCV), pages 818- 833, 2018. 2
[22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and compuder-assister intervention, pages 234-241. Springer, 2015. 7
[23] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pages 2377-2386. 2019. 2, 13
[24] Aliaksandr Siarohin. Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances Neural Information Processing Systems, 2019. 2, 3, 6, 7, 10.
13
[25] Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representa- tions for articulated animation. In CVPR, 2021. 2, 3, 6, 7, 10
[26] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recog- nition. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Rep- resentations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
11

---

6
[27] Deqing Sun, Charles Herrmann, Fitsum Reda, Michael Rubinstein, David Fleet, and William T Freeman. What makes raft better than pwe-net? arXiv preprint 1xXi:2203.117111 2022.
[28]
Deqing Sun, Xiandong Yang. Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference computer vision and pattern
recogmition. pages 8934-8943, 2018. 2.3
[29] Jiale Tao, Biao Wang. Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. Motion transformer for unsupervised image animation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI. pages 702-719. Springer, 2022. 2, 3, 7, 8, 10, 13 [30] Jiale Tao, Biao Wang, Borun Xu, Tiezheng Ge, Yuning Jiang. Wen Li. and Lixin Duan. Structure-awar motion transfer with deformable anchor model. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pottern Recognition, pages 3637-3644 3646, 2022. 2. 3.7
[31] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow. UK. August 23-28, 2020, Proceedings, Part II
16. pages 402-419. Springer, 2020. 2, 3, 5. 13
[32] Soumya Tripathy, Juho Kannala, and Esa Rahtu. Facegan: Facial attribute controllable reenactment gan. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1329-1338.
2021. 2
[33] Qiulin Wang. Lu Zhang. and Bo Li. Safa: Structure aware face animation. In 2021 International Conference
on 3D Vision (3DV), pages 679-688. IEEE, 2021. 1,2
[34] Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot frec-view neural talking- synthesis for video conferencing. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10039-10049, 2021. 10
[35] Yaohui Wang. Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning animate images via latent space navigation. In International Conference on Learning Representations,
2022. 2. 7
[36] Dongxu Wei, Xiaowei Xu, Haibin Shen, and Kejie Huang. C2[-fwn: Coarse-to-fine flow warping network for spatial-temports consistent motion transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4))2852-286) May 2021. 2
[37] Yuxiang Wei, Ming Liu, Haolin Wang, Ruifeng Zhu, Guosheng Hu, and Wangmeng Zuo. Learning flow-based feature warping for face frontalization with illumination inconsistent supervision. In European
Conference on Computer Vision, pages 558-574. Springer, 2020. 2
[38] Olivia Wiles. A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV), pages 670-686, 2018. 2
[39] Guangming Yao, Yi Yuan, Tianjia Shao, Shuang Li, Shanqi Liu, Yong Liu, Mengmeng Wang, and Kun Zhou. Onc-shot face recnactment using appearance adaptive normalization. Proceedings of the AAAI
Conference on Artificial Intelligence, 15:3172-3180 May 2021. 2
[40] Bohan Zeng, Bayu Liu, Hong Li, Xuhui Liu, Jianzhuang Liu, Dapeng Chen, Wei Peng, and Baochang Zhang. FNeVR: Neural volume rendering for face animation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave. and Kyunghyun Cho, editors. Advances Neural Information Processing Systems, 2022. 2. 3, 7 [41] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep fcatures as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[42] Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3657-3666 2022. 2, 3,
[43] Long Zhao, Xi Peng. Yu Tian, Mubhasir Kapadia, and Dimitris Metaxas. learning to forecast and refine residual motion for image-to-video generation. In Proceedings of the European conference on computer
vision (ECCV), pages 387-403, 2018. 2,
[44] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale vidco facial attributes dataset. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part VII, pages 650 667. Springer, 2022.
1, 7
12

---

Appendix
In this appendix, we present detailed information on the architectural design, evaluation metrics, correlation pyramid, and parameter analysis on the number of keypoints and the number of iterations.
Architecture details: Our system mainly consists of keypoint detector, dense motion module, the source and driving structure encoders, a flow updater, and an image generator. The keypoint detector and dense motion modules are implemented using blocks similar those used in previous works [23, 29]. In contrast, we provide architecture details of our source and driving structure encoders and image generator in Figure 8. All the modules are included in our provided codes for easy reference and implementation.
Evaluation metrics: We mainly introduce four metrics that utilized third-party partt models for evaluation.
Average Keypoint Distance (AKD [24]). This metric computes the average keypoint distance between generated and ground-truth images. It is designed to evaluate the pose quality of the generated images. We use existing detectors [3] to extract the facial landmarks.
Average Euclidean Distance (AED [24]). This metrie is designed assess the identity quality of generated images based on specific feature representations, that are extracted from pre-trained facial identification network [1]. The average Euclidean distance between generated and ground- truth video frames is computed.
Average Rotation Distance (ARD [5]. We use the toolbox py-feat [14] to extract the Euler angles of the head poses, and then compute the average Euler angles distance between the generated and driving images. This metric evaluates the head pose quality.
Action Units Hamming distance (AUH [5]). This metric measures the quality of facial expression, it computes the average Hamming distance between action units of generated and driving images. We use the toolbox py-feat [14] to extract facial action units.
Details of the correlation pyramid: We pool the structure correlation volume C' E Rhxuxxxan in the last two dimensions to obtain the pyramidal structure correlation volume {Ci E sample patch correlation features on all C in the pyramid, as the pyramid design aims to capture more rich motion features different scales, which is inspired by the optical flow method RAFT [31]. Both the patch radius and the pyramid level can expand the search space of the structure correlation volume, resulting in the expanded correlation feature dimensions. In the main paper's experiments, empirically set P to 1 and r to 3.
R/xxxk/2'xxx/ x ) In each iteration,
Number of keypoints: Following previous methods [24, 29], we configured the number of keypoints to 10 in the main paper. Here we conduct experiments to study this hyperparameter, with FOMM adopted as the prior motion model. As shown in Table 5, in relatively sparse configuration such as 10-20 keypoints, our non-prior-based motion refinement approach can consistently improve the performance of the prior motion model.
Number of iterations: We set the iteration number according to the image resolutions in the main paper. Specifically, we start the iteration at a lower feature resolution H/32 W/32 that is meaningful for motion flow for warping, and end the iteration at the highest resolution H x W, thus the total iterations are set to for 256-resolution image. By changing the highest resolution to H/2 x W/2, H H/4x x W/4, H /8 x W/8, we obtain iteration settings of 5, 4, accordingly. We then analyze the performance and run time of different iteration settings. goes down as the iteration number increases. At the same time the performance is generally better with higher iterations, especially for the motion-related metric AKD and identity-relatt metric AED. Overall, the 5-iteration setting can be good trade-off.
seen in the table, the FPS
13

---

Table 5: Parameter analysis on the number keypoints. We present results on the Voxceleb1 dataset.
L1
PSNR
LPIPS
AKI)
AED
FOMM-Kp10
0.0386
24.62
0.164
1.254
0.124
NPMR-FOMMMKppL 10
0.0367
25.15
0.156
1.224
0.114
FOMM-Kp15
0.0376
24.91
0.161
1.245
0.123
NPMR+FOMM-KIH
0.0360
25.34
0.154
1.199
0.108
FOMM-Kp20
0.0376
24.97
0.160
1.222
0.118
FFMM-K0
0.0356
25.49
0.152
1.194
0.107
Table 6: Parameter analysis on the number of iterations. We present results on the Voxceleb1 dataset.
Iteration Number
LI
PSNR
LPIPS
AKD
AED
FPS
3
0.0355
25.52
0.152
1.164
0.109
21.30
0.0353
25.57
0.151
1.156
0.107
20.45
5
0.0354
25.57
0.151
1.151
0.108
19.56
0.0354
25.57
0.151
1.155
0.108
18.57
Isput-3
SameHlack-64
DownRlock-12
x0
Impos-10
Inpur-13
DownBlock-26
k0
DownBlock-64
DownBlock-44
DownBlock-128
DownBlock-125
DownBlocck512
x0
DownBlock-256
DownBlock-256
DownBlock-512
DownBBock-512
DownBoock-512
xx0
ResUpblack:12
DuwnBlluck
DownBlock-512
x(1
RelUppHockk512
UptBlock-522
UpBBockk512 512
<(1 x(1-0)
UpBllek-2266
UpBlooli-256
ResUpBlack
UpBBlck-122 128
UpBBook--22 128
x(1 1 0)
ResUpBlock-12
Up
UpBioceis64l 64
x(1
UpBlook-32
UpHlock-32
Rest Relock44
Conv1x1-256
Conv
Comn 7xC7-3
Driving Structure
(b) Structure Encader
Sigmoid-3
x(1
Image Generator
Figure Detailed architectures of the driving structure encoder, the source structure encoder, and the image generator.
14


Reviews:
[{"review_id": "Yqs1U5853r", "reviewer": "Reviewer_ELv3", "summary": "The paper presents a new approach to generate human face videos based on a source image and a driving video, which simultaneously learning both coarse and finer facial motions. The proposed approach utilizes a structure correlation volume constructed from keypoint features to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow that is estimated by a prior motion model. \n\nThe main contributions of the paper are: \n1. A non-prior-based motion refinement approach to compensate for the inadequacy of existing prior-based motion models.\n2. Utilize the keypoint features to build a structure correlation volume that represents the structure correspondence between the source and driving images across all spatial locations.\n3. Extensive experiments on challenging benchmarks that demonstrate the effectiveness of the proposed approach in enhancing the capability of prior-based motion representation through learning the motion refinement.\n", "questions": "It’s suggested that the author should clarify the differences between the module they used in this paper and the related modules in some NST methods [6,7]. Also, experiments on more datasets of different object categories are needed.", "limitations": "Yes, they have adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a novel approach to generate human face videos that simultaneously learns both coarse and finer facial motions. The proposed approach utilizes a structure correlation volume constructed from keypoint features to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow estimated by a prior motion model. The approach addresses a significant problem in the field of unsupervised face animation.\n\n2. The paper is of good quality, and the method proposed is clear. The authors provide a detailed description of the approach, including the structure correlation volume and the motion refinement module. The experimental results demonstrate the effectiveness of the proposed approach in enhancing the capability of prior-based motion representation through learning the motion refinement.\n\n3. The paper is well-written and easy to follow. The authors provide a clear and concise description of the proposed approach, including the key components and the experimental setup. The paper is well-organized, and the authors provide a clear summary of the contributions and limitations of the proposed approach.\n\n4. The proposed approach has significant implications for the field of unsupervised face animation which addresses an important problem in this field: the inadequacy of existing prior-based motion models to capture detailed facial motions. The proposed approach is effective in enhancing the capability of prior-based motion representation through learning the motion refinement. The approach has potential applications in creating imaginative image animations for entertainment purposes, but it also has the potential to be used in creating deepfakes, which could have negative impacts. The authors acknowledge this limitation and provide recommendations for future work. \n", "weaknesses": "1.\tThe pictures in Figure 1 are too small, especially the optical flow map is not clear enough, no obvious changes can be seen before and after refinement. Also, is the schematic diagram of affine transformation in Figure 1 drawn based on a real example? Why does the affine transformation change so much after refinement?\n\n2.\tStraightforward combination of existing techniques. The innovation of this paper is not enough, the main innovation point lies in the motion refinement module. However, the method of correlation volume and iteratively refine optical flow used in it is very similar to the correlation matrix and iteratively update in some flow estimation[1,2,3] and correspondence estimation[4,5] methods, and this set of process of first constructing a 4D correlation volume, and then iteratively updating and optimizing optical flow has also been used in some neural style transfer(NST) methods[6,7], but the author did not clarify the differences between the module they used in this paper and related modules in these NST methods.\n\n3.\tThere is an error in line 180 of the article: the author claims that the look up operation on the correlation volume is shown on the left side of Figure 2, but there is no corresponding content in Figure 2. Should Figure 2 be changed to Figure 3? \n\n4.\tIn the comparison video provided in the supplemental material, the video of each method is too small, and it is difficult to see the tiny facial deformation details. It is recommended to arrange the videos of each method in the form of a nine-square grid, and enlarge the size of each video window.\n\n5.\tThe experiments of verifying the effectiveness of the proposed method are insufficient. The framework used by your method is the framework of unsupervised image animation [8,9,10,11], which should be able to generate animation videos on any object category. In addition to human faces, this framework can also be applied to human bodies and animals. Previous unsupervised image animation methods [8,9,10,11] have also been tested on related datasets of different subjects, including TaiChiHD[9], TED-talks[10], and MGif[8]. Your method has only been tested on face-related datasets, but from your overall framework, I don't see any modules that restrict the method to only work on human face. Therefore, I think that the non-prior-based motion refinement module should be applied to datasets of different objects to further test its effectiveness.\n\n6.\tLack of quantitative experiments on the Cross-identity task, such as user study that have been used in previous methods [8,9,10,11].\n\nReferences\n\n[1] Dosovitskiy, Alexey, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. 2015. “FlowNet: Learning Optical Flow with Convolutional Networks.” In 2015 IEEE International Conference on Computer Vision (ICCV). doi:10.1109/iccv.2015.316.\n\n[2] Teed, Zachary, and Jia Deng. 2020. “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow.” In Computer Vision – ECCV 2020, Lecture Notes in Computer Science, 402–19. doi:10.1007/978-3-030-58536-5_24.\n\n[3] Xu, Haofei, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. 2022. “GMFlow: Learning Optical Flow via Global Matching.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr52688.2022.00795.\n\n[4] Kim, Seungryong, Stephen Lin, SangRyul Jeon, Dongbo Min, and Kwanghoon Sohn. 2018. “Recurrent Transformer Networks for Semantic Correspondence.” arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition, October.\n\n[5] Zhang, Pan, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. 2020. “Cross-Domain Correspondence Learning for Exemplar-Based Image Translation.” In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr42600.2020.00519.\n\n[6] Liu, Xiaochang, Xuanyi Li, Ming-Ming Cheng, and Peter Hall. 2020. “Geometric Style Transfer.” Cornell University - arXiv,Cornell University - arXiv, July.\n\n[7] Liu, Xiao-Chang, Yong-Liang Yang, and Peter Hall. 2021. “Learning to Warp for Style Transfer.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.00370.\n\n[8] Siarohin, Aliaksandr, Stephane Lathuiliere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. “Animating Arbitrary Objects via Deep Motion Transfer.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr.2019.00248.\n\n[9] Siarohin, Aliaksandr, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. “First Order Motion Model for Image Animation.” Neural Information Processing Systems,Neural Information Processing Systems, January.\n\n[10] Siarohin, Aliaksandr, Oliver J. Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. 2021. “Motion Representations for Articulated Animation.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.01344.\n\n[11] Zhao, Jian, and Hui Zhang. n.d. “Thin-Plate Spline Motion Model for Image Animation.”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ELv3\n\n### Q1: Presentation of Fig.1, cross reference of Fig.3 (line 180), composing type of supplementary videos.\n\n* Sorry for the confusion. The affine transformation is for illustration purposes, not from real examples. But the two motion flows are influenced by a pair of real source and driving images. The motion flow is actually obtained by compositing a set of local affine transformations in FOMM, while we use a single global affine transformation for the purpose of the simple introduction. For better understanding,  we provide the visualization of refined motion flows across all iterations in the video (the last few seconds). We will improve Fig.1 in the later version. \n* You are right, Figure 2 should be changed to Figure 3, we will correct this in the later version.\n* Thanks for the suggestion! We will rearrange the videos of each method to form a nine-square result for better comparison. Benefiting from your suggestion, our newly updated video results are clearer than before. \n\n\n### Q2: The innovation of this paper, difference with optical flow methods and neural style transfer methods.\n\n* Thank you for the valuable thoughts. We addressed these concerns in the **common response**. And we sincerely urge your reconsideration on this point.\n\n\n### Q3: The experiments of verifying the effectiveness of the proposed method are insufficient, other object animation experiments are needed.\n\nSimilar to DaGAN and FNeVR which explored facial specific information such as facial depth and 3DMM information, for helping or enhancing learning the local affine motion representation of faces, we would like to emphasize that our method is better applied to face animation than human body animation. The reason is that the locally rigid assumption is reasonable and robust for human bodies, and thus our non-prior based motion refinement approach may bring no significant improvements. However, this assumption can be easily violated for the human face, since non-rigid deformation is very normal in a local face area, and this special attribute of face motion motivates us to investigate the non-prior based motion refinement. Existing methods such as FOMM, MRAA, DAM and TPSM generally focused on finding better parametric motion models, while bringing considerable improvements on the human body animation, it ignores the importance of non-prior motion for face animation. Therefore, our method is more specially designed for face animation, and we believe the experiments on face animation are sufficient to validate its effectiveness. \n\nAs a thought, our method may help the deformation modeling like clothes of the human body, which may need the finer motion modeling compared to the body itself. However, currently the more urgent problem in body animation is to more accurately find the body structures and then it can be better transferred.\n\nWe here simply conduct additional experiments on the TEDTalks dataset. As it shows, our method can improve the earlier baseline FOMM by a considerable margin, however, it didn't improve the sota baseline TPSM much especially for the motion related metric AKD.\n\n\n|            | L1         |    AKD    |   AED |\n| :--------: | :--------: | :--------:| :----:|\n| FOMM       | 0.0281     | 4.413     | 0.129 |\n| SCORR+FOMM | 0.0269     | 3.535     | 0.119 |\n| TPSM       | 0.0271     | 3.501     | 0.126 |\n| SCORR+TPSM | 0.0265     | 3.499     | 0.118 |\n\n\n\n### Q4: Lack of quantitative experiments on the Cross-identity task, such as user study that have been used in previous methods.\n\n* Thanks for the suggestion. Initially, we have conducted a cross-identity task in the supplementary, evaluating the pose and expression quality of generated videos. And we here further present a comparison on the CSIM and FID metrics. We addressed this concern in the **common response**.\n"}, {"review_id": "HPQW0AKSlJ", "reviewer": "Reviewer_2aeH", "summary": "1. This work proposed a  new unsupervised face animation approach to learn simultaneously the coarse and finer motions.\n2. The results outperformed the state-of-the-art methods on two representative datasets.", "questions": "None.", "limitations": "Yes, limitations are discussed in the main paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The target task is important.\n2. The method is intuitive and reasonable.\n3. The results are good.\n4. The paper is well-written.", "weaknesses": "1. Compared to prior model. The proposed non-prior models seems like a non-linear version of prior model based on affine transformation. When the number of keypoints increasing and the grid of affine transformation being much finer, will it approach the proposed non-prior model? Another related question is that why the proposed model works better than a local thin-plate-spline motion model (such as [38])?\n\n2. Visualization. 1) In the visualization of comparison in video, the results of the proposed model looks close to FNeVR. Authors could also provide user study in visual quality for comparison. 2) Also, the results on CelebV-HQ is really important, since it is a much challenging dataset. Visualization results on this dataset is necessary, in comparison, ablation study and video. 3) Why didn't report the results of FNeVR on CelebV-HQ on Table 1?\n\nI am inclined to accept this paper, if the concerns can be solved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n## Response to Reviewer 2aeH\n### Q1: Will the increasing number of keypoints of the local affine motion model approach the non-prior based motion model? And why the proposed model works better than a local thin-plate-spline motion model?\n\n* Thanks for the good question. If one can learn the dense human face keypoints that are meaningful and structured, the local affine motion model will approach the non-prior based motion, as many local linear models will approach a good non-linear model. However, it is very hard to learn dense structured keypoints in unsupervised face animation. We conduct an experiment that configure the FOMM with 100 keypoints, the result in the table is denoted as \"FOMM-kp100\". It should be noted that the motion related metric AKD even goes through a decrease compared to 10 keypoint configuration. We found that many keypoints are non-meaningful and overlapped with other keypoints or located in the background. On the other hand, in a relatively sparse configuration such as 10-20 keypoints, our non-prior based motion refinement approach can consistently improve the performance.\n* The reasons why the proposed model works better than a local thin-plate-spline motion model are two folds. On one hand, though nonlinear, the local thin-plate-spline motion model is still a parametric (prior-based) motion model, thus the non-prior based motion still has the stronger ability to model complex deformations. On the other hand, in TPSM[37], it needs 5 geometric consistent keypoints to determine a local thin-plate-spline transformation, and 10 local transformations require 50 geometric consistent keypoints which heavily increases the learning difficulty. Though it used the dropout strategy in learning transformations, the learning difficulty may still limit their method in reaching the representation ability of local thin-plate-spline transformations.\n\n|  | L1     | PSNR  | LPIPS | AKD   | AED   |\n|:---------------------:|:------:|:-----:|:-----:|:-----:|:-----:|\n| FOMM-kp100            | 0.0371 | 25.15 | 0.162 | 1.288 | 0.116 |\n| FOMM-kp10             | 0.0386 | 24.62 | 0.164 | 1.254 | 0.124 |\n| SCORR+FOMM-kp10       | **0.0367** | **25.15** | **0.156** | **1.224** | **0.114** |\n| FOMM-kp15             | 0.0376 | 24.91 | 0.161 | 1.245 | 0.123 |\n| SCORR+FOMM-kp15       | **0.0360** | **25.34** | **0.154** | **1.199**| **0.108** |\n| FOMM-kp20             | 0.0376 | 24.97 | 0.160 | 1.222 | 0.118 |\n| SCORR+FOMM-kp20       | **0.0356** | **25.49** | **0.152** | **1.194** | **0.107** |\n\n\n### Q2: User study, visualization of the CelebV-HQ dataset, and why didn't report the results of FNeVR on CelebV-HQ on Table 1\n\n* Thanks for the suggestion. We provide the user study in the **common response**.\n* Thanks for the suggestion. We have provided the anonymous link to video comparison results on the CelebV-HQ dataset, it can be accessed from the Area Chair. As can be seen in the provided video, on this more challenging dataset, our method generally performs the best in terms of learning finer motions, while other methods often produce blurry results, indicating that motion learned by their methods is blurrier than ours. \n* As claimed in lines 255-257 in the main paper, the authors of FNeVR didn't release their training codes and it didn't conduct experiments on this dataset. We had trouble in reproducing their paper due to the unclearness of the use of 3DMM parameters, so we didn't report their results on this dataset.\n"}, {"review_id": "4L8a45mDVZ", "reviewer": "Reviewer_mUhK", "summary": "The paper presents an unsupervised method for learning to create continuous video animations of faces given a single input image and a driving video of the same or a different subject. The method relies on the optical flow between the driving and source images to warp the features of the source image, which are then passed to a generator to create the final output frames. The main contribution of the proposed method lies in the two-step formulation of the flow estimation, in which starting from a coarse prediction using a known prior-based method (Affine Transforms, Thin Plate Splines) it iteratively produces updates to reach a more fine-grained flow (and occlusion) estimation with more local details. The flow updates are learnt in an unsupervised way without using prior models, but by building a correlation volume between the source and driving images. The paper includes experiments, generated images and supplementary videos that validate the authors' claims for more detailed animations in comparison to similar methods. \n", "questions": "- In 3.4 it is mentioned that a perceptual and an equivariance loss are employed for training. Is the model trained with both same and different subjects for source and driving frames? \n- How are C^{i} combined to get the final values from the correlation pyramid? \n- What is the effect of the correlation pyramid to the memory requirements of the method? How does the memory requirements compare with the compared methods?  ", "limitations": "The authors have included a section discussing the method's limitations and possible negative societal impact. A mentioned limitation (the correlation matrix relying on learned keypoints) might also be the reason for changing the facial shape in cross-subject animations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written overall and it is easy to follow along with the presented concepts and results.\n- The paper includes experimental results that validate the authors' claims, as well as ablations that offer a better understanding about model design choices.\n- The method produces higher-quality generated images/videos than the compared methods for unsupervised learning of face animation.\n- The idea of using motivation from RAFT to built a similar correlation structure for estimating motion flow for face animation is interesting.", "weaknesses": "- Some design choices are not empirically or experimentally justified in the paper. For example, why are specifically 6 iterations used to update the initial coarse estimations? Why does it make sense to sample from the same correlation structure for all iterations, while each iteration operates at a different image scale? \n- Even though the method performs well compared to previous methods, there is still some evident identity shift in cross-subject experiments, meaning that the shape of the face slightly changes from the original to the one of the driving frame. This is evident in Figure 5 (for example 2nd row) as well as in the supplemental video. This effect makes sense because the estimated flow between different subjects possibly includes more deformation information, rather than only deformation because of expressions or rigid motion. Disentangling identity from other motions is a common weakness of cross-identity animation methods, which exists in this method, too. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer mUhK\n### Q1: Some design choices are not empirically or experimentally justified in the paper. For example, why are specifically 6 iterations used to update the initial coarse estimations!\nThanks for the careful review and sorry for the confused understanding. We set the iteration number according to the image resolutions. Specifically, we start the iteration at a lower feature resolution $H/32\\times W/32$ that is meaningful for motion flow for warping, and end the iteration at the highest resolution $H\\times W$, thus the total iterations are set to 6 for a 256-resolution image. By changing the highest resolution to $H/2\\times W/2, H/4\\times W/4, H/8\\times W/8$, we obtain iteration settings of $5,4,3$ accordingly. We then analyze the performance and run time of different iteration settings. As seen in the table, the FPS goes down as the iteration number increases, while the performance is generally better with higher iterations especially for the motion-related metric AKD and identity-related metric AED. Overall, the 5-iteration setting can be a good trade-off. \n\n\n|  | L1              | PSNR           | LPIPS          | AKD            | AED            | FLOPs (G)      | FPS            |\n|:--:|:------:|:-------:|:-----:|:-----:|:-------:|:------:|:-------:|\n| 3                     | 0.0357          | 25.37          | 0.153          | 1.194          | 0.112          | **89.86** | **21.30** |\n| 4                     | 0.0354          | **25.54** | 0.152          | 1.188          | 0.113          | 91.20          | 20.45          |\n| 5                     | **0.0353** | **25.54** | **0.151** | 1.177          | 0.108          | 96.34          | 19.56          |\n| 6                     | **0.0353** | 25.51          | 0.152          | **1.176** | **0.107** | 116.5          | 18.57          |\n\n\n### Q2: The identity shift in cross-subject experiment.\nThanks for pointing out this. We addressed this concern in the **common response**. Though our method performs slightly better on the identity-preservation metric, it is indeed a limitation of our method, and in the future, we would like to explore leveraging 3D face information for helping alleviate this problem.\n\n### Q3: Why does it make sense to sample from the same correlation structure for all iterations, while each iteration operates at a different image scale?\nThanks for the detailed question. The correlation matrix is of size $h\\times w\\times h\\times w$, where $h=H/4, w=W/4$, $H,W$ define the image size. The indexes of the first two dimensions are corresponded to the driving frame coordinate while that of the last two dimensions denote the source coordinate. By a simple downsampling or upsampling operation on the first two dimensions, we obtain the different-scale correlation matrixes which are used in different iterations. The reason why it makes sense to sample from the same correlation structure for all iterations is that, the warped source features at different resolution should share similar spatial structures, thus we can downsample or upsample the correlation matrix to the different resolution, and then use it to obtain refined motion flow at different resolution for warping different-scale source features.\n### Q4: Is the model trained with both same and different subjects for source and driving frames\nNo, similar to FOMM, the model is only trained on the same subjects for source and driving frames.\n\n### Q5: How are $C^{i}$ combined to get the final values from the correlation pyramid? What is the effect of the correlation pyramid to the memory requirements of the method?\n* For each $C^{i}$, as described in the line 177-179 of the main paper, we lookup a $(2r+1)\\times (2r+1)$ patch correlation feature. We then concatenate all correlation features in the pyramid, result in a $P\\times (2r+1)^2$-channel correlation feature, where $P$ denotes the pyramid level. \n\n* Actually, the correlation pyramid brings few additional memory requirements, as the concatenated correlation feature will be send to a convolution (with kernel size equal to 1) to get a fixed 128-channel feature. That means, the only increased memory requirements come from the increased convolution parameters of size $(P-1)\\times (2r+1)^2\\times 128$. In practice, we set $P=2, r=3$, and the increased parameter size is equal to $6272=0.006$M, leading to few additional memory requirements.\n\n### Q6: How does the memory requirements compare with the compared methods?\nThanks for the question. We addressed this concern in the **common response**."}, {"review_id": "jWhnDoaDxL", "reviewer": "Reviewer_pDRj", "summary": "This paper contributes to the field of unsupervised face animation by introducing a novel motion refinement method to overcome the limitations of existing prior-based motion models, especially in estimating detailed facial movements.\n\nThe paper's approach introduces a new method which uses a structure correlation volume built from keypoint features. This provides motion information that does not rely on prior data. This information is used to iteratively refine the coarse motion flow estimated by a previous motion model.\n\nThe authors have conducted numerous experiments on challenging benchmarks to test the effectiveness of their approach. The results indicate that this new method enhances the capability of prior-based motion representation by learning how to refine motion. This suggests that their approach can effectively increase the accuracy of unsupervised face animation tasks.", "questions": "1. This paper is very similar to RAFT from ECCV2020 in terms of insight and specifically for the module, which raises novelty concerns.\n\n2. Related works [1] are not discussed and compared in this paper.\n\n[1] Tao, Jiale, Biao Wang, Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. \"Motion Transformer for Unsupervised Image Animation.\" In European Conference on Computer Vision, pp. 702-719. Cham: Springer Nature Switzerland, 2022.", "limitations": "Both limitations and social impact are discussed in this paper.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In this research, the authors present a new unsupervised face animation approach that concurrently learns both coarse (global) and finer (local) facial motions. Their method integrates a local affine motion model to learn the global, coarse facial motion and a novel motion refinement module to compensate for the local affine motion model's ability to model more detailed facial motions in local areas.\n\nThe motion refinement process is based on the dense correlation between the source and driving images. To achieve this, a structure correlation volume is first constructed using the keypoint features of the source and driving images. The authors then train a model to iteratively generate minor facial motions from low to high resolution.\n\nThe learned motion refinements are combined with the coarse motion to generate the new image. After performing extensive experiments on widely used benchmarks, the method was found to deliver the best results among existing state-of-the-art methods. The authors have also committed to making the source code for their method publicly available in the future.", "weaknesses": "1. The goal of this article is to learn fine facial motion, but using the VoxCeleb dataset may not be sufficient for this purpose. In my view, finer facial motion refers to the ability to reproduce wrinkles, eyeballs, and micro-expressions at high resolution, which may require higher resolution datasets.\n\n2. As noted by the author, the reliability of keypoints estimation heavily influences the quality of finer facial motion. It is not clear whether the improvement of existing models comes from more accurate keypoints estimation or the proposed module.\n\n3. This paper is very similar to RAFT from ECCV2020 in terms of insight and specifically for the module. A clearer comparison between the two works and their essential differences would be helpful.\n\n4. It is important to note that fine facial motion may not equal to better image quality. Therefore, a more thorough evaluation of the estimated motion would be beneficial. Additionally, conducting this task on higher resolution facial images would provide more convincing results.\n\n5. The author sets the iteration number to 8, which is an important hyperparameter. It would be useful to know why this setting was chosen and whether it significantly affects the model's runtime.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer pDRj\n\n### Q1:  VoxCeleb dataset may not be sufficient for evaluating finer facial motion and higher resolution dataset may be more appropriate.\n\nThanks for the suggestion, but we may not agree that the VoxCeleb dataset may not be sufficient for evaluating finer facial motion, as our experiments in the main paper had proved this (Figure 4, Figure 5 and Table 1) and results from other methods generally failed in capturing finer face motions. We here also perform experiments on the 512-resolution CelebVHQ dataset. As seen in the table, our method can still improve existing SOTA methods on all metrics on the high resolution setting. Since it is time-consuming to train on the 512-resolution dataset, we are not able to reproduce other baselines at this time, and we will include them in a few days. Thanks for the suggestion again.\n\n|                       | L1  | PSNR | LPIPS | AKD | AED |\n|:---------------------:|:---:|:----:|:-----:|:---:|:---:|\n| TPSM                  | 0.0435 | 24.91  | 0.193   | 3.035 | 0.158 |\n| SCORR+TPSM            | **0.0418** | **25.24**  | **0.185**   | **2.911** | **0.149** |\n\n\n### Q2: Whether the improvement of existing models comes from more accurate keypoints estimation or the proposed module\n\nThanks for the good question. If I'm not mis-understanding, so the question is whether the improvements come from that the proposed module promotes estimating more accurate keypoints, or come from that the proposed module did learn finer motions. We explain this in the following.\n\nSince our structure correlation volume is built on keypoints, the gradient on the structure correlation volume will back-propagate to the keypoint detector, thus it may affect the learning process of keypoints. We validate this by a simple but well-designed experiment on the Voxceleb dataset, that is to stop the gradient flow from our structure encoders to the keypoint detector, by simply detach the keypoints from previous gradient graph before sending them to the structure encoders. The operation is denoted by Ours-$sg$ in the Table. As seen, this operation didn't affect the final performance at all, and the stop gradient operation even performs slightly better. So we believe that the improvements come from that our proposed module did learn finer motions.\n|                       | L1     | PSNR  | LPIPS | AKD   | AED   |\n|:---------------------:|:------:|:-----:|:-----:|:-----:|:-----:|\n| Ours                  | **0.0353**| 25.51 | 0.152 | 1.176 | 0.107 |\n| Ours-$sg$             | **0.0353** | **25.54** | **0.150** | **1.175** | **0.105** |\n\n\n### Q3: A clearer comparison and essential differences between the RAFT and our method.\n\nWe initially discussed this in the related work of the main paper, and we here make a more detailed and clearer discussion in the **common response**. We sincerely hope you could reconsider the novelty of our method in leaning non-prior based motion in face animation.\n\n### Q4: Fine facial motion may not equal to better image quality. A more thorough evaluation of the estimated motion would be beneficial. Additionally, conducting this task on higher resolution facial images would provide more convincing results.\n\nThanks for the thoughtful question. In general, the unsupervised face animation methods (both existing and our method) consist of two stages: **motion estimation** and **image generation**. One can investigate better image generation modules to improve the image quality, such as FNeVR which adopted the SPADE[1] module as their image generator and considerably improves the generation performance as seen in their ablation studies. But we think finding better motion estimation is more meaningful and urgent in this area. So we made our motivation fall into learning motion refinements and our experiments did validate this motivation. For evaluating the motion quality, we think the metric **AKD** in the self reconstruction task, and the **ARD** and **AUH** metrics in cross-identity experiments that we present in Table A1 of the supplementary material, are sufficient for the purpose. We also conduct experiments on the high-resolution dataset in Q1. We sincerely hope that you could reconsider this point.\n\n[1] Semantic Image Synthesis with Spatially-Adaptive Normalization, CVPR 2019.\n\n### Q5: Why set the current iteration number.\n\nThanks for the question. Due to the length limitation of characters, we addressed this similar concern in the **Q1 of Reviewer mUhK**, please refer to that.\n\n\n\n### Q6: Comparison and discussion with existing method [1].\n\nThanks for suggesting the paper. This method is also prior-based animation methods, which employs a vision transformer for better learning affine motion relationship. So our non-prior based motion refinement module would still be effective for improving it. We thus make a direct comparison with it on the Voxceleb dataset, by applying our motion refinement module on the top of the motion transformer. As seen, the performance even surpass our original result in the main paper, which reflects the robustness of our non-prior based motion refinement module to different prior-based motion models. We will discuss this in the later version.\n\n|  | L1     | PSNR  | LPIPS | AKD   | AED   |\n|:---------------------:|:------:|:-----:|:-----:|:-----:|:-----:|\n| MoTrans               | 0.0370 | 25.08 | 0.160 | 1.191 | 0.120 |\n| SCORR+MoTrans         | **0.0352** | **25.64** | **0.150** | **1.164** | **0.108** |\n\n[1] Motion Transformer for Unsupervised Image Animation, ECCV 2022.\n"}, {"review_id": "UcTOm4AtSa", "reviewer": "Reviewer_SbHP", "summary": "This paper proposes a method for face animation via learning to refine a coarse motion field. The refinement is performed in a recurrent manner using the previous iteration's motion, occlusion map and structure correlation volume.  Both the qualitative and quantitative results show improvement over prior art.", "questions": "N/A", "limitations": "1) An important limitation of this method is that it is sensitive to the scale of the face of the driving video. For example, around the 1:47 of the supplementary video, the face structure of the animated face deviates significantly from the source and is actually closer to that of the target. This is unavoidable due to the use of 2D key points in the method, in fact it is expected. The authors must include this as a limitation of their work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) Qualitative results show that the proposed method is clearly model finer facial movement much better than prior art, especially around the eye. FNeVR is very close but the proposed method has fewer uncanny artifacts.\n\n2) Quantitatively as well, the proposed method outperforms prior art.\n\n3) The paper is well written.", "weaknesses": "1) Across all the results, I notice a strong identity shift during animation. While this is common (and seemingly worse) in prior art as well, it is important to quantify. One way to measure this is to report the FaceIDLoss or L2-loss as the head is rotated 15, 30, 60 degrees from the original head-pose. I believe this is an important evaluation to include for the sake of completeness. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer SbHP\n\n### Q1: Identity shift during animation, and FaceIDLoss\n\nThanks for the good suggestion. We have compared the common face identity preservation metric CSIM in the **common response**, and our method performs slightly better on this metric while much better on the motion related metrics.\n\n### Q2: Limitation on the face scale\n\nThanks for the suggestion. Indeed this is a limitation of our method, and we will include this point in the later version.\n\n"}, {"review_id": "Gl4HrDVpLp", "reviewer": "Reviewer_zuqN", "summary": "This paper anaylizes the limitations of existing face animation methods in capturing the finer facial motions, and hence design a non-prior-based motion refinement approach to achieve finer face motion transfer in local areas. A correlation volume between the source and driving images is constructed as non-prior motion evidence, and a refinement module is introduced to generate the fine facial motions iteratively. Extensive experiments show the effectiveness of this paper.", "questions": "1. Could the authors give more explations about the meanings of flow in the figures? I can not see the changes after updation. There are obvious improvements on the occlusion map, does the improvement in motions mainly come from the enhanced occlusion map.\n2. Since Face vid2vid [1] shows a promissing performance on the cross-identity reenactment task, it is suggested to give a comparison with Face vid2vid. It is good to provide the video comparisons.\n   \n   [1] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021.\n1. It is better to provide computation cost evaluation at inference, i.e. Flops, memory, and FPS, compared to other SOTA methods.", "limitations": "This paper is effective to refine the motions to obtain better results, but is still limited to the quality of the learned keypoints, as the limitaions in the paper stated, which I think is more urgent to solve for unsupervised face animation. And this paper did not give comprehensive evaluations on the performance in identity preserving, which is important for face animation in practical applications. \nI'm not sure whether it is qualified for the conference.  If the authors could give reasonable response for the problems stated in **Weakness** and **Questions**, I would like to consider improving my rating. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of involving a non-prior based motion refinement module is effective to capture the fine motions in local areas.\n2. The manuscript is well written and clearly states the main idea and contribution in face animation.\n3. This paper performs extensive experiments, and the results of the same-identity video reconstruction task outperform many previous methods. ", "weaknesses": "1. This paper concentrates on the motion issue, but the video results of this paper are not impressive enough. No obvious improvements can be seen given the video results on the self reconstruction.\n2. This paper does not show the quantitative results in terms of some usual metrics, i.e. CSIM and FID, on the cross-identity reenactment task. I wonder the performance of this work in preserving identity.\n3. The introduction should be improved, as it only concludes limited contributions. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer zuqN\n\n### Q1: No obvious improvements can be seen given the video results on the self reconstruction.\n\nOn one hand, we will rearrange the video results to form a nine-square presentation for better comparison. On the other hand, we suggest zooming in our Figure 4 in the main paper for detailed comparison, since our method produces finer motions in local areas such as lips and eyes. And we have also uploaded more video results on the CelebV-HQ dataset.\n\n### Q2: This paper does not show the quantitative results in terms of some usual metrics, i.e. CSIM and FID, on the cross-identity reenactment task.\n\nThanks for the good suggestion. It is indeed our carelessness. And we have compared this metric in the **common response**. While our method performs slightly better on the CSIM metric, it produces higher-quality motion transfer, as suggested by the better ARD and AUH metrics.\n\n### Q3: The introduction should be improved, as it only concludes limited contributions.\n\nThanks for the suggestion. We will revise it in the later version. But are there any more specific suggestions? Since \"it only concludes limited contributions.\" is a little confusing to me.\n\n### Q4: Could the authors give more explorations about the meanings of flow in the figures? Does the improvement in motions mainly come from the enhanced occlusion map.\n\n* Sorry for the confusion. The flows in the figure are actually estimated from a source and driving image pair, and they come from two different iterations. For better understanding, we provide the refined motion flow across all iterations in the video (the last few seconds). \n* The improvement in motions mainly comes from the enhanced motion, but not the occlusion map. As indicated by the component ablation studies in Table A2 of the supplementary material, removing flow updation undergoes a considerable performance decrease. However, the updated occlusion map is still helpful for understanding the flow updation process, since it reflects the non-warping area which is reversed to the motion flow that controls the warping area.\n\n### Q5: Comparison with Face Vid2Vid\n\nThanks for the suggestion. We quantitatively compare our method with Face vid2vid on the Voxceleb dataset. As seen in the table, our method outperforms it by a large margin. We also provided the link to the video comparison, which can be accessed from the Area Chair, please check in for details. It can be seen that similar to other methods, Face vid2vid also fails in capturing finer face motions, and our method generally produces better animation.\n\n|  | L1     | PSNR  | LPIPS | AKD   | AED   |\n|:---------------------:|:------:|:-----:|:-----:|:-----:|:-----:|\n| FaceVid2Vid           | 0.0444 | 23.40 | 0.175 | 1.405 | 0.138 |\n| ours                  | **0.0353** | **25.51** | **0.152** | **1.176** | **0.107** |\n\n### Q6: It is better to provide computation cost evaluation at inference, i.e. Flops, memory, and FPS, compared to other SOTA methods.\n\nThanks for the good suggestion. We have analyzed this in the **common response**."}]

Author rebuttal:
## Common Response 

We thank all Reviewers for the valuable feedback and insightful comments. We appreciate the reviewer's positive comments regarding the novelty and the effectiveness of our method. We now clarify the reviewer's common questions as follows.

### Q1: Difference with optical flow and neural style transfer methods (Reviewer ELv3 and pDRj).

* We should emphasize that a key difference between our method and optical flow methods lies in building the correlation volume, which we discussed in the related work from line 90 to line 104 in the main paper. Optical flow methods compute the **appearance feature correspondence**  of two consecutive frames, while our key idea is to learn the **structural correspondence**. The methods in optical flow can't be directly applied to face animation, as computing appearance feature similarity between driving and source images will naively leak the driving appearance, which is not expected in face animation. As a result, the optical flow methods will directly find appearance from the source face that is similar to the driving face. However, the task of face animation aims to find motion between source and driving (structure correspondence), but not to find appearance correspondence. 
Existing face animation methods generally model the structural correspondence by local parametric transformations between the source and driving frame, which ignores the importance of **building the non-prior based correspondence**. Thus our efforts in constructing the structure correlation volume, though inspired by the optical flow method RAFT, are meaningful and motivatable for face animation. And based on this motivation, we made the first attempt to find a way to introduce the non-prior evidence for motion refinement. Therefore, we believe our method is novel and can fill some gaps in the field of face animation.
* Thank you for suggesting the neural style transfer methods. And we have carefully read the paper [1-2] in this area. We summarize two main differences between our method and theirs, the way to build correspondence and the purpose/motivation of building it. On one hand, they still lie in a similar paradigm with optical flow methods, which computes the visual appearance similarity between the content image and the geometric style image. While our method uses the structure features as input to build correlation volume. More essentially, on the other hand, the neural style transfer methods [1-2] utilize the correspondence matrix to estimate **a global parametric transformation**, which is totally different from our motivation that aims to use the structure correlation for estimating the **non-parametric motion refinement** across all spatial locations. This non-parametric motion modeling is specifically designed for face animation. And it is generally ignored by existing methods. Thanks for suggesting the paper again. We will include these discussions together with the optical flow methods in the later version.

[1] Geometric Style Transfer. arXiv 2020.
[2] Learning to Warp for Style Transfer. CVPR2021

### Q2: Run time and memory evaluation (Reviewer mUhK and zuqN)

Thank reviewers for the suggestion. We here present the inference-stage run time evaluation. ALL results are obtained by running reconstruction experiments on a single NVIDIA 3090 GPU. Compared to year-2022 methods, the memory cost is similar among all methods, the FPS of our method is slightly worse than that of LIA, DAM, TPSM, and FNeVR, but better than DaGAN. Considering that our method produces the best animation quality, our method achieves better trade-off between performance and inference speed.
|  | FOMM | MRAA  | LIA | DAM | DaGAN | TPSM | FNeVR | Ours |
|:--:|:------:|:-------:|:-----:|:-----:|:-------:|:------:|:-------:|:------:|
| Memory (G)$\downarrow$  | 6.48 | 6.51  | **2.63** | 6.49  | 7.07  | 6.82  | 6.53  | 7.02  |
| FLOPs (G)$\downarrow$   | **56.06** | 61.07 | 88.32 | 56.39 | 89.78 | 142.4 | 130.2 | 116.5 |
| FPS$\uparrow$        | **41.64** | 33.04 | 20.98 | 30.29 | 17.72 | 19.96 | 21.83 | 18.57 |
### Q3: Cross-identity evaluation (Reviewer ELv3, 2aeH) and identity preservation (Reviewe mUhK, SbHP and zuqN )
* Thank reviewers for the suggestion. Initially we compare the ARD and AUH methric for cross-identity animation in the Table. A1 of the supplementary, which evaluates the pose and expression quality of generatedimages . We here further compare the CSIM (cosine similarity of the face embedding using arcface[1]) and FID metric. As seen in the following table, our method also performs best in terms of CSIM and FID, indicating that our method preserves better the source identity information. It should be noted that, in self reconstruction task, the AED metric is also designed for evaluating identity preservation, and our method generally performs best in term of this metric.
| | FOMM  | MRAA  | LIA   | DAM   | DaGAN | TPSM  | FNeVR | Ours           |
|:--:|:------:|:-------:|:-----:|:-----:|:-------:|:------:|:-------:|:------:|
| ARD$\downarrow$        | 3.122 | 2.678 | 3.883 | 2.669 | 3.090 | 2.724 | 2.755 | **2.399** |
| AUH$\downarrow$                   | 0.850 | 0.729 | 0.772 | 0.717 | 0.751 | 0.668 | 0.751 | **0.625** |
| CSIM$\uparrow$        | 0.702 | 0.678 | 0.706 | 0.698 | 0.682 | 0.689 | 0.714 | **0.722** |
| FID$\downarrow$                   | 70.00 | 69.29 | 71.01 | 69.16 | 68.50 | 68.67 | 68.42 | **66.27** |

[1] ArcFace: Additive Angular Margin Loss for Deep Face Recognition, CVPR2019
* As suggested, we also conducted a user study experiment. Specifically, 20 participants are asked to evaluate 50 randomly generated videos of different methods, according to the transferred motion and the preserved identity. There are a total of 1000 votes, and we compute the ratio in favor of our method. It can be seen that our method is generally more favorable compared to TPSM and FNeVR.
| Ours $vs$ TPSM | Ours $vs$ FNeVR  | 
|:------:|:-------:|
| 61.1% | 56.9%  |

Meta-review: Solid paper with good results, it improves the state of the art. The ML contribution is not the strongest, as the technical improvements are relatively straightforward but at the same time they are necessary for accurate facial animation.
Decision: Accept (poster)