SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The overall loss is formulated by the sum of the two without weighting:
$$
\mathcal {L} = \mathcal {L} _ {\text {p e r}} + \mathcal {L} _ {\text {e q u i}} \tag {13}
$$
Table 1: Quantitative comparisons with state-of-the-art methods on the video self-reconstruction task. We present results on the Voxceleb1 and CelebV-HQ datasets. Our method generally achieves the best performance on all evaluation metrics.
Voxceleb1	CelebV-HQ
L1	PSNR	LPIPS	AKD	AED	L1	PSNR	LPIPS	AKD	AED
FOMM	0.0412	23.85	0.171	1.284	0.135	0.0531	22.95	0.204	3.491	0.219
MRAA	0.0394	24.47	0.166	1.274	0.132	0.0433	24.65	0.173	1.852	0.167
LIA	0.0425	23.64	0.212	1.457	0.138	0.0507	22.95	0.235	1.982	0.179
DAM	0.0395	24.51	0.165	1.242	0.124	0.0496	23.58	0.189	1.899	0.180
DaGAN	0.0422	24.03	0.168	1.265	0.125	0.0652	21.37	0.249	8.704	0.307
TPSM	0.0396	24.84	0.160	1.203	0.122	0.0412	25.29	0.160	1.663	0.156
MTIA	0.0370	25.09	0.159	1.190	0.120	0.0405	25.63	0.157	1.532	0.153
FNeVR	0.0404	24.36	0.165	1.238	0.126	-	-	-	-	-
Ours	0.0354	25.57	0.151	1.155	0.108	0.0376	26.11	0.147	1.424	0.137
Table 2: Cross-identity evaluation on the Voxceleb1 dataset.
FOMM	MRAA	LIA	DAM	DaGAN	TPSM	MTIA	FNeVR	Ours
ARD	3.122	2.678	3.883	2.669	3.090	2.724	2.794	2.755	2.399
AUH	0.850	0.729	0.772	0.717	0.751	0.668	0.676	0.691	0.647
FID	70.00	69.29	71.01	69.16	68.50	68.67	66.65	66.48	64.68
Implementation details: We adopt two Hourglass networks [19] for the keypoint detector and the dense motion network, and a Unet [22] for the image generator similar to previous works [24, 25]. We further employ two hourglass networks for the driving and source structure encoders. The flow updater consists of four convolution blocks that share the parameters across the iterations. We train our method for 100 epochs on four NVIDIA A100 GPU cards or eight NVIDIA 3090 GPU cards. It takes about 24 hours for training. The number of keypoints is set to 10 following previous methods [24, 29, 30, 42]. The patch radius  $r$  is set to 3 and the number of iterations is set to 6. The Adam optimizer [16] is adopted with  $\beta_{1} = 0.5$  and  $\beta_{2} = 0.999$ , the initial learning rate is set as  $2 \times 10^{-4}$  and dropped by a factor of 10 at the end of 60th and 90th epoch. We leave the architecture details in the supplementary material.
Datasets: We conduct experiments on the widely used Voxceleb1 [18] dataset and the recently collected more challenged CelebV-HQ dataset [44]. Voxceleb1 is a talking head dataset consisting of 20047 videos, among which 19522 are used for training and 525 are used for testing. The CelebV-HQ dataset consists of 35666 video clips, we randomly choose 500 of them for testing. All videos are resized to  $256 \times 256$  for a fair comparison with existing methods.
Metrics: We measure the reconstruction quality using L1, Peak Signal-to-Noise Ratio (PSNR), and LPIPS [41] following [35, 40]. To evaluate the transferred motion quality we adopt Average Keypoint Distance (AKD) following previous methods [24, 25, 30, 42]. The identity quality of the generated videos is measured by Average Euclidean Distance (AED) [24, 25]. We use FID, ARD, and AUH metrics for evaluating cross-identity face animation following [5]. All these metrics are the lower the better except the PSNR.
Baselines: A bunch of baseline methods are compared, including FOMM [24], MRAA [25], LIA [35], DAM [30], DaGAN [10], TPSM [42], MTIA [29] and FNeVR [40]. All reported baseline results are obtained by evaluating the checkpoints from their Github repo. or retrain the provided codes, except that the FNeVR does not provide the training code and thus we do not report their results on the CelebV-HQ dataset. It is worth noting that most of these baseline methods adopt a prior motion model, while DaGAN [10] further explored the face depth information and FNeVR leverage the 3DMM parameters. For comparison with them, we adopt the motion transformer [29] for our prior motion estimation, while we also employ the affine motion model [24] and thin-plate-spline motion model [42] in the section of model ablations.
Quantitative comparison: Table 1 displays the same-identity reconstruction results, which is the standard evaluation setting used in previous methods [10, 24, 25, 29, 30, 40, 42] that reconstruct a driving video based on its first-frame appearance and all-frame motion. As can be seen, our
Table 3: Model ablations on the proposed non-prior based motion refinement module. We present results on the Voxceleb1 dataset. Baseline methods are all based on our implementation, which differentiates from original methods with a multi-scale feature fusion mechanism as described in Equation (10).
L1	PSNR	LPIPS	AKD	AED
NPMR	0.0393	24.41	0.164	1.328	0.133
FOMM	0.0386	24.62	0.164	1.254	0.124
FOMM + NPMR	0.0367	25.15	0.156	1.224	0.114
TPSM	0.0370	25.09	0.159	1.231	0.120
TPSM + NPMR	0.0353	25.51	0.152	1.176	0.107
MTIA	0.0370	25.09	0.159	1.190	0.120
MTIA + NPMR	0.0354	25.57	0.151	1.155	0.108
Figure 4: Same-identity video reconstruction on the Voxceleb1 dataset. Our method generally performs the best among a variety of baselines.
method outperforms other methods across all metrics on the two datasets. Specifically, large-margin improvements on the reconstruction metrics (L1, PSNR, LPIPS) demonstrate the advantages of learning motion refinement. What's more, our method achieved the best motion quality (the best score on the AKD metric) particularly on the more challenging CelebV-HQ dataset, highlighting the powerful motion representation capabilities of our non-prior-based motion refinement approach. The superior generated motion quality of our method is further validated by the best ARD and AUH metric in cross-identity experiments in Table 2. The best AED score and FID score also indicate that our method preserves the source face identity well during motion transfer. These significant observations are further supported by the qualitative results presented in Fig. 4, Fig. 5, and Fig. 6.
Qualitative comparison: We conducted experiments on both the same-identity video reconstruction and cross-identity face reenactment similar to previous methods [10, 29, 43], showcasing our qualitative results in Fig. 4, Fig. 5, and Fig. 6. Our method consistently demonstrates greater robustness against large motions (the first row of Fig. 4), detailed face deformations (the third row of Fig. 5 and the fourth row of Fig. 6), occlusions (the second and last row of Fig. 4), and varying light conditions (the fourth row of the Fig. 4), etc. Additionally, our approach outperforms other methods in generating precise facial details such as the eyes and lips, while avoiding common artifacts. To sum up, our methods can effectively learn non-prior-based motion refinement, exhibiting a significant improvement over prior-based motion representation that is limited on learning finer facial motions.
Model ablations: We performed a model ablation study to further evaluate the proposed Non-Prior-based Motion Refinement module. As previously discussed in Equation (8), in the non-prior-based initialization process, we initialize the motion flow using the structure correlation volume itself. We
Figure 5: Cross-identity face reenactment on the Voxceleb1 dataset, our method consistently demonstrates superior performance in generating high-fidelity results compared to existing methods.
Figure 6: Cross-identity face reenactment on the CelebV-HQ dataset, our method consistently demonstrates superior performance in generating high-fidelity results compared to existing methods.
denoted this model variant as NPMR. As shown in Table 3, our NPMR-only model did not yield good results, highlighting the importance of prior-based initial motion estimation for learning finer facial motions and the crucial role it plays in our non-prior-based motion refinement module. When combined with specific prior motion estimations, such as that from the affine and thin plate spline motion models, our method showed promising improvements over the original prior motion models. These combinations with different prior motion models significantly demonstrate the flexibility of the proposed non-prior-based motion refinement module, and its strong ability to promote the enhanced motion representation over existing prior motion models.
Component ablations: We conducted a thorough examination of the inputs and outputs of our non-prior-based motion refinement module through detailed ablations. Specifically, we explored the impacts of the model variants of the source structure encoder without the input of a source image, the flow updater without the input of warped source features, and the flow updater without the output of occlusion or flow (i.e. only updating either occlusion or flow and keeping the other the same as in the initialization process). The results are presented in Table 4. It is noteworthy that both the warped source feature and the source image play an important role in the motion refinement process as demonstrated by the decreased AKD and AED metric values on the  $w / o$  warped source feature and  $w / o$  source image variants. Importantly, if we solely refine the occlusion map without updating the motion flow (and vice versa), a significant decrease in AKD and AED metrics occurs, validating the significance of our motivation to refine the coarse motion flow.
Table 4: Component ablations of the inputs and outputs of the proposed non-prior-based motion refinement module. We present results on the Voxceleb1 dataset.
L1	PSNR	LPIPS	AKD	AED
w/o flow	0.0362	25.48	0.155	1.172	0.112
w/o occlusion	0.0356	25.48	0.152	1.175	0.112
w/o warped source feature	0.0357	25.52	0.153	1.160	0.110
w/o source image	0.0354	25.45	0.152	1.165	0.113
Ours full	0.0354	25.57	0.151	1.155	0.108
Figure 7: Motion flow visualization of the refining process, we show refined motion flows of all iterations. Note that flows of different resolutions are resized to the image resolution.
Motion flow visualizations: To further understand the motion refinement process, we visualize the refined motion flows in different iterations. As can be seen in Fig. 7, the motion flow gets refined from the initial iteration to the final iterations, corresponding to our description in Equation (7).
In this paper, we investigate the task of unsupervised face animation and propose a novel motion refinement approach to address the inadequacies of existing prior-based motion models for estimating finer facial motions. Our approach utilizes a structure correlation volume constructed from keypoint features, to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow estimated by a prior motion model. We conducted extensive experiments on challenging benchmarks, and our results demonstrate that our approach enhances the capability of prior-based motion representation through learning motion refinement.
Limitations: A major limitation of our approach is that it relies on the quality of the learned keypoints, as we mainly construct our proposed structure correlation volume using these keypoints. While current prior motion models may support the learning of significant keypoints, our method may still be limited by the quality of keypoints that are learned in an unsupervised fashion. To overcome this limitation, one potential solution would be to integrate model-based techniques that leverage facial priors, such as predefined keypoints that are more structured. Another limitation is the identity shift problem in cross-identity face animation, as keypoints often leak the face shape information of a driving video. The issue can be alleviated by employing relative motion transfer as in existing unsupervised animation methods [24, 25, 29]. Moreover, we could also explore prior motion models that can disentangle the global head motion and micro-expression motion, like facevid2vid [34].
Social impact: Our method has the potential to be used in creating deepfakes, which could have negative impacts. Therefore, individuals intending to use our technique for creating deepfakes should obtain authorization to use the respective human face images. Nevertheless, our method can also be utilized in creating imaginative image animations for entertainment purposes.
Acknowledgement: This work is supported by the National Natural Science Foundation of China (No. 62176047), Shenzhen Fundamental Research Program (No. JCYJ20220530164812027), and the Fundamental Research Funds for the Central Universities Grant (No. ZYGX2021YGLH208).
In this appendix, we present detailed information on the architectural design, evaluation metrics, correlation pyramid, and parameter analysis on the number of keypoints and the number of iterations.
Architecture details: Our system mainly consists of a keypoint detector, a dense motion module, the source and driving structure encoders, a flow updater, and an image generator. The keypoint detector and dense motion modules are implemented using blocks similar to those used in previous works [23, 29]. In contrast, we provide architecture details of our source and driving structure encoders and image generator in Figure 8. All the modules are included in our provided codes for easy reference and implementation.
Evaluation metrics: We mainly introduce four metrics that utilized third-party models for evaluation.
- Average Keypoint Distance (AKD [24]). This metric computes the average Keypoint distance between generated and ground-truth images. It is designed to evaluate the pose quality of the generated images. We use existing detectors [3] to extract the facial landmarks.
- Average Euclidean Distance (AED [24]). This metric is designed to assess the identity quality of generated images based on specific feature representations, that are extracted from a pre-trained facial identification network [1]. The average Euclidean distance between generated and ground-truth video frames is computed.
- Average Rotation Distance (ARD [5]). We use the toolbox py-feat [14] to extract the Euler angles of the head poses, and then compute the average Euler angles distance between the generated and driving images. This metric evaluates the head pose quality.
- Action Units Hamming distance (AUH [5]). This metric measures the quality of facial expression, it computes the average Hamming distance between action units of generated and driving images. We use the toolbox py-feat [14] to extract facial action units.
Details of the correlation pyramid: We pool the structure correlation volume  $C \in \mathcal{R}^{h \times w \times h \times w}$  in the last two dimensions to obtain the pyramidal structure correlation volume  $\{C^i \in \mathcal{R}^{h \times w \times h / 2^i \times w / 2^i}\}_{i=0}^P$ . In each iteration, we sample patch correlation features on all  $C^i$ 's in the pyramid, as the pyramid design aims to capture more rich motion features of different scales, which is inspired by the optical flow method RAFT [31]. Both the patch radius and the pyramid level can expand the search space of the structure correlation volume, resulting in the expanded correlation feature dimensions. In the main paper's experiments, we empirically set  $P$  to 1 and  $r$  to 3.
Number of keypoints: Following previous methods [24, 29], we configured the number of keypoints to 10 in the main paper. Here we conduct experiments to study this hyperparameter, with FOMM adopted as the prior motion model. As shown in Table 5, in a relatively sparse configuration such as 10-20 keypoints, our non-prior-based motion refinement approach can consistently improve the performance of the prior motion model.
Number of iterations: We set the iteration number according to the image resolutions in the main paper. Specifically, we start the iteration at a lower feature resolution  $H / 32 \times W / 32$  that is meaningful for motion flow for warping, and end the iteration at the highest resolution  $H \times W$ , thus the total iterations are set to 6 for a 256-resolution image. By changing the highest resolution to  $H / 2 \times W / 2$ ,  $H / 4 \times W / 4$ ,  $H / 8 \times W / 8$ , we obtain iteration settings of 5, 4, 3 accordingly. We then analyze the performance and run time of different iteration settings. As seen in the table, the FPS goes down as the iteration number increases. At the same time the performance is generally better with higher iterations, especially for the motion-related metric AKD and identity-related metric AED. Overall, the 5-iteration setting can be a good trade-off.
Table 5: Parameter analysis on the number of keypoints. We present results on the Voxcelebl dataset.
L1	PSNR	LPIPS	AKD	AED
FOMM-kp10	0.0386	24.62	0.164	1.254	0.124
NPMR+FOMM-kp10	0.0367	25.15	0.156	1.224	0.114
FOMM-kp15	0.0376	24.91	0.161	1.245	0.123
NPMR+FOMM-kp15	0.0360	25.34	0.154	1.199	0.108
FOMM-kp20	0.0376	24.97	0.160	1.222	0.118
NPMR+FOMM-kp20	0.0356	25.49	0.152	1.194	0.107
Table 6: Parameter analysis on the number of iterations. We present results on the Voxceleb1 dataset.
Iteration Number	L1	PSNR	LPIPS	AKD	AED	FPS
3	0.0355	25.52	0.152	1.164	0.109	21.30
4	0.0353	25.57	0.151	1.156	0.107	20.45
5	0.0354	25.57	0.151	1.151	0.108	19.56
6	0.0354	25.57	0.151	1.155	0.108	18.57
(a) Driving Structure Encoder
(b) Source Structure Encoder
(c) Image Generator
Figure 8: Detailed architectures of the driving structure encoder, the source structure encoder, and the image generator.