Learning Implicit Scale Conditioned Memory Compensation for Talking Head Generation
Authors: Fa-Ting Hong
Dan Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Talking Head Generation

Abstract: Talking head video generation aims to animate the pose and expression of a person in a target driving video using motion information contained in the video, while maintaining a person's identity in a given still source image. Highly dynamic and complex motions in the driving video cause ambiguous generation from the source image, because the still source image cannot provide sufficient appearance information for occluded regions or delicate expressions, which severely produces artifacts and significantly degrades the generation quality. However, existing works mainly focus on learning more accurate motion estimation and representation in 2D and 3D, and they ignore the facial structural prior in addressing the facial ambiguities. Therefore, effective handling of the ambiguities in the dramatic appearance changes of the source to largely improve facial details and completeness in generation still remains barely explored. To this end, we propose a novel implicit scale conditioned memory compensation network (MCNet) for high-fidelity talking head generation. Specifically, considering human faces are symmetric and structured, we aim to automatically learn a representative global facial memory bank from all training data as a prior to compensate for the facial generation features. Each face in the source image contains a scale that can be reflected in detected facial keypoints. To better query the learned global memory, we further propose to learn implicit scale representations from the discrete keypoints, which can be used to condition on the query of the global memory, to obtain scale-aware memory for the feature compensation. Extensive experiments from quantitative and qualitative perspectives demonstrate that MCNet can learn representative and complementary facial memory, and can clearly outperform previous state-of-the-art methods on VoxCeleb1 and CelebV datasets. 

Under review as a conference paper at ICLR 2023
LEARNING IMPLICIT SCALE CONDITIONED MEMORY COMPENSATION FOR TALKING HEAD GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Talking head video generation aims to animate the pose and expression of person in target driving video using motion information contained in the video, while maintaining a person's 's identity in a given still source image. Highly dynamic and complex motions in the driving video cause ambiguous generation from the source image, because the still source image cannot provide sufficient appearance infor- mation for occluded regions delicate expressions, which severely produces ar- tifacts and significantly degrades the generation quality. However, existing works mainly focus on learning more accurate motion estimation and representation 2D and 3D, and they ignore the facial structural prior in addressing the facial ambiguities. Therefore, effective handling of the ambiguities in the dramatic ap- pearance changes of the source to largely improve facial details and completeness in generation still remains barely explored. To this end, we propose novel im- plicit scale conditioned memory compensation network (MCNet) for high-fidelity talking head generation. Specifically, considering human faces are symmetric and structured, aim to automatically learn a representative global facial memory bank from all training data as a prior to compensate the facial generation features. Each face in the source image contains scale that can be reflected in detected fa- cial keypoints. To better query the learned global memory, we further propose to learn implicit scale representations from the discrete keypoints, which can be used to condition on the query of the global memory, to obtain scale-aware memory for the feature compensation. Extensive experiments from quantitative and qualitative perspectives demonstrate that MCNet can learn representative and complementary facial memory, and can clearly outperform previous state-of-the-art methods on VoxCeleb1 and CelebV datasets.
INTRODUCTION
In this work, we aim to address the problem of generating a realistic talking head video given one still source image and one dynamic driving video, which is widely known as talking head video generation. A high-quality talking head generation model needs to imitate vivid facial expressions and complex head movements, and should be applicable for different facial identities presenting in the source image and the target video. It has been attracting rapidly increasing attention from the community, and a wide range of realistic applications remarkably benefits from this task, such as digital human broadcast, Al-based human conversation, and virtual anchors in films.
Significant progress has been achieved on this task in terms of both quality and robustness in recent years. Existing works mainly focus on learning more accurate motion estimation and representation in 2D and 3D to improve the generation. More specifically, 2D facial keypoints or landmarks are learned to model the motion flow (see Fig. [c)) between the source image and any target image in the driving video (Zhao et al. (202T): Zakharov et al.) (2019); Hong et al.] (202)) ). Some works also consider utilizing 3D facial prior model (e.g. 3DMM & Vetter (1999) with decoupled expression codes (Zhao et al.) 2021 Zakharov et al. 2019) or learning dense facial geometries in self-supervised manner (Hong et al., 2022) to model complex facial expression movements to pro- duce more fine-grained facial generation. However,
matter how accurately the motion can be estimated and represented, highly dynamic and complex motions in the driving video cause ambigu- generation from the source image (see Fig. dd), because the still source image cannot provide sufficient appearance information for occluded regions or delicate expressions, which severely pro- duces artifacts and significantly degrades the generation quality.

---

Under review as a conference paper at ICLR 2023
(a) Source
(b) Driving
(c) Motion
(d) Warped Image
(c) Memory
(i) MCNet (Ours)
Figure I: MCNet animation illustration. MCNet first learns the motion flow (c) between the source and the driving images. (d) provides an elaboration of possible occulusion or deformation caused large motion, which is produced by directly warping the source image with the motion provided by the driving images. (e) shows randomly sampled memory channels of our learned scale-aware memory bank. We also present examples of generated results of our method in (f).
Intuitively, we understand that human faces are symmetrical and highly structured, and many regions of the human faces are essentially not discriminative. For instance, only blocking a very small eye region of a face image makes a well-trained facial recognition model largely drop the recognition performance (Qiu et al.) (021), which indicates to certain extent that the structure and appearance representations of human faces crossing different face identities are generic and transferable. There- fore, learning global facial priors on spatial structure and appearance from all available training face images, and utilizing the learned facial priors for compensating the dynamic facial synthesis are crit- ically important for high-fidelity talking head generation. However, existing works did not explore these beneficial facial priors to address facial ambiguities in generation from large head motions.
In this paper, effectively deal with the ambiguities in the dramatic appearance changes from the still source image, we propose an implicit scale conditioned Memory Compensation Network, coined MCNet, to learn and transfer global facial representations compensate ambiguous fa- cial details and guarantee completeness for a high-fidelity generation. Specifically, we design and learn global spatial meta memory bank. The optimization gradients from all the training images during training contribute together to the updating of the meta memory, and thus it can capture the global and most common facial appearance and structure representations for the transferring. Since the different source face images contain distinct scales, to more effectively query the learned meta memory bank, we propose an implicit scale conditioned memory module (ISCM) (see Fig. B As the detected discrete facial keypoints inherently contain the scale information of the face, we first learn an implicit scale representation from the discrete keypoint coordinates, and further use it to condition on the query of the meta memory bank to obtain a scale-aware memory bank, which can more effectively compensate the feature of faces with different scales. The compensation is per- formed through memory compensation module (MCM) (see Fig. 4 The warped feature map generated from the estimated motion field queries the scale-aware memory bank through a dynamic cross-attention mechanism to output a refined compensated feature map for the final generation.
We conduct extensive experiments to evaluate the proposed MCNet on two competitive talking head generation datasets (i.e. VoxCeleb (Nagrani et al. 2017) and CelebV (Wu al., 2018). Experimen- tal results demonstrate the effectiveness of learning global facial memory to tackle the appearance ambiguities the talking head generation, and also show clearly improved generation results from both qualitative and quantitative perspectives, achieving state-of-the-art performances.
In summary, our main contribution is three-fold:
We propose to learn a global facial meta memory bank to transfer representative facial representations to handle the appearance and structure ambiguities caused by the highly dynamic generation from still source image. To the best of our knowledge, it is the first exploration in the literature to model global facial representations for effectively improving the ambiguities in talking head generation.
We propose a novel implicit scale conditioned memory compensation network (MCNet) for talking head video generation, in which implicit scale conditioned memory module (ISCM) and a facial memory compensation module (MCM) are designed to respectively perform the scale- memory learning and the feature compensation tasks.
2

---

Under review as a conference paper at ICLR 2023
Qualitative and quantitative experiments extensively show the effectiveness of the learned meta memory bank for addressing the ambiguities in generation, and our framework es- tablishes clear state-of-the-ar performance the talking head generation. The general- ization experiment also shows that the proposed modules can effectively boost the perfor- mance of different talking head generation models.
RELATED WORKS
Talking Head Video Generation. Talking Head video Generation can be mainly divided into two strategies: image-driven and audio-driven generation. For the image-driven strategy, researchers aim capture the expression of a given driving image and aggregate the captured expression with
the facial identity from given source image. Some approaches Yao et al. 22000; Wu et al. 2021b
Wang et al., 2021a) utilized 3DMM regressor (Tran & Liu 2018 Zhu et 2017) to extract an expression code and an identity code from a given face, and then respectively combine them
from different faces to generate a new face. Also, several other works (Tripathy et al., [202] Ha
et al. 2020 Zakharov et al. 2020 2019; a pretrained face model (Guo al. 2019) to act as anchors of the face. Then, the facial motion flow calculated from landmarks is transferred from a driving face video. However, their motion flow suffers from error accumulation caused by inaccuracy of the pretrained model. To overcome this limitation, the keypoints are learned in unsupervised fashion (Siarohin et al. 2019 Hong al., 2022 Wang et al. 2021b Liu et al.) 2021a| Zhao & ZZhngg 2022) to better represent the motion of the face with carefully designed mechanisms for modeling the motion transformations between two
Zhao et al.) (021)) utilized facial landmarks detected by
sets of keypoints. Audio-driven talking head generation (Ji et al. 2022 Lu et al. 2021 Wu et al.)
2021a Ji al., T is another popular direction on this topic, audio sequences do not contain information of the face identity, and is relatively easier to disentangle the motion information from the input audio. [Liang et al. (2022) explicitly divides the driving audio into granular parts through delicate prior-based pre-processing to control the lip shape, face pose, and the facial expression.
In this work, we focus on the image-driven talking head generation. In contrast to previous image- driven works, we aim at learning global facial structure and appearance priors through a well- designed memory-bank network to effectively compensate intermediate facial features, which can produce higher-quality generation ambiguous regions caused by large head motion.
Memory Bank Learning. Introducing an external memory prior component is popular because of its flexible capability of storing, abstracting and organising long-term knowledge into structural form. Recently, the memory bank has shown its powerful capabilities in learning and reasoning for addressing several challenging tasks, e.g. image processing Yoo et al. 2019 Huang et al.) (021), video object detection (Sun et al,, 2021), and image caption [Fei 1). As an earlier work, Weston eet al. (2014) propose memory network, which integrates inference components within memory bank that can be read and written to memorize supporting facts from the past for question answer- ing. Yoo et al. (2019) propose memory-augmented colorization network to produce high-quality colorization with limited training data. Xu et al. (2021) uses the texture memory of patch sam- ples extracted from unmasked regions to inpaint missing facial parts. |w al.) (2022) proposes a memory-disentangled refinement network for coordinated face inpainting in coarse-to-fing manner.
In contrast to these previous works, to the best of our knowledge, we are the first to propose using a global memory mechanism to deal with ambiguous generation issues in the task of talking head video generation. We also accordingly design novel saaiiittffcial-scaeee aware memory learning network and a novel memory compensation network to successfully tackle the issues.
3 METHODOLOGY
Under the same pipeline in previous work (Sarrohin et al. 2019). we introduce an implicit seale conditioned memory compensation network, termed MCNet, for talking head video generation. MCNet learns a facial-scale-aware memory bank by the designed implicit scale conditioned memory module (ISCM) compensate the warped feature in the memory compensation module (MCM).
3.1 OVERVIEW
The framework of our MCNet depicted in Fig. 2 can be divided into three parts: (i) The keypoint detector and the dense motion network. Initially, the keypoint detector receives source image S and a driving frame D to predict K pairs of keypoints, i.e. {I,,,) Vs. K ) and {Ia,, yd 1K on the source and target, respectively. With the keypoints generated from the driving frame and the
3

---

Under review as a conference paper at ICLR 2023
(F
(o) Implicit Scale Conditioned
(b) Memory Compensation Module
(F
Memory Module
Output
Compensation
Condition
oill
M.
M,
Source
MLP
F cpst
encoded feature decoded feuture implicit seale representation
S
M,: memory scale-sware
Keypoint Detector
Dense
Motion
warp
M,
memory
Driving
(a.1 (Pas Mar)
Motion Flow
Warped Feature
F
compensated
Feature
Figure 2: An illustration of the proposed MCNet, which contains two designed modules com-
pensate the facial feature map: (i) The implicit scale conditioned memory module (ISCM) learns the scale information from the input source, utilizing the warped feature map and keypoints of the source image, to produce an implicit scale representation S, which is conditioned on the meta mem- ory bank M. to learn a scale-aware memory M,. (ii) The memory compensation module (MCM) adopts dynamic cross-attention mechanism to compensate the warped feature map spatially.
source image, the dense motion network estimates the motion flow As between these two; (ii) The designed implicit scale conditioned memory module (ISCM). We first leverage the estimated motion flow As-D to warp the encoded feature F², in the i-th layer, resulting in warped feature F 'Tee warped feature F", and the source keypoints are fed into the implicit scale conditioned memory module to encode implicit scale representation, which will be conditioned on the query of the meta memory M, to produce an The memory compensation module (MCM). After obtaining the scale-aware memory bank M.. we utilize a dynamic cross-attention mechanism to compensate the warped features spatially in the MCM, and then output a compensated feature F Fpp Finally, our decoder utilizes all the N feature \ maps i.e. {Fi cpt JitJi== i= to produce the final image Ltst. In the following, we will show how to learn our memory bank in the ISCM and how it is utilized in the MCM for generation feature compensation.
scale-aware memory bank M,; (iii)
3.2 LEARNING
GLOBAL FACIAL MEMORY
We first aim learning a global meta-memory bank to model facial structure and appearance rep- resentations from the whole face dataset. It is clear that human faces have multiple scales in the real-world, thus learning meta-memory bank to directly compensate all source faces with different scales is inflexible. To handle the faces with distinct scales, we design an implicit scale conditioned memory module (ISCM) to learn a scale-aware memory, through cource-scale-condiiion the global meta-memory bank to compensate warped source face features with scale variations.
query on
Meta memory. this work, we first aim to learn a global meta memory bank to store the global and generic facial appearance and spatial structure representations from all the training data available. We initialize a meta memory bank M, as a cube tensor with a shape of C. x Hm x Wm instead of vector (Esser et al., 2021p. Moreover, the multiple channels hold enough capacity for the meta memory to learn different facial structures and appearances (see Fig. 7 As many regions of the human faces are not discriminative and transferable, we can utilize the global facial priors learned in the meta memory to compensate ambiguous regions in the generated faces. With a designed ob- jective function, the meta memory bank can be automatically updated by the optimization gradients from all the training images during the training stage. In this way, the facial prior learned in the meta memory is global rather than conditioned on any specific input sample, which provides highly beneficial global information for face compensation in generation.
Implicit scale representation learning. our framework, the detected facial keypoints are used to learn motion flow for feature warping. The facial keypoints implicitly contain the scale information of the human face because of their structural positions (Siarohin et al., 2019 Tao et al.. 2022). Therefore, we utilize both the source keypoints {x,,e, Va.: ) and the warped feature F. to learn an implicit scale representation of the source face. The reason of learning a scale representation for the source is that aim to compensate the image with the identity of the source image. And the warped feature Fi. is used because it also contains the scale information of the source image, as the warped feature is generated via warping the source feature with the keypoint- motion flow.
4

---

Under review as a conference paper at ICLR 2023
Mcta Memory M.
Keypoints (Z Nap
Ca
Flatten
MLP J000
Manipulated Convolution
Cn
Conv
GAP
Imsplcit Scale Ropresentation
Scale-aware Memory
Warp Feature
Projected Feature Fr
proj
M,
The illustration of the proposed implicit scale conditioned memory module (ISCM). The symbol C denotes the concatenation operation, and the "GAP" and "Conv" represent the global average pooling and the convolution layer, respectively. The detailed generation of the projected feature Fr proj can refer to Fig. 4 C; denotes the channel number of the i-th level warped feature F in our autoencoder framework, while Cm is the channel number of our memory bank.
Figure 3:
As shown in Fig. 3 to embed the scale information, we first utilize global average pooling function FGAP to squeeze the global spatial information of the projected feature F Prooj. which is produced from the warped feature Fü (see Fig. 4). into a channel descriptor. After that, we concatenate the flattened and normalized keypoints with the feature vector from FGAP, and feed them into MLP mapping network Fmlp to learn an implicit scale representation S of the source image. Thus:
Fmip (S'). S' Concat [FGAP(F)) (Firos) [2,,1 Ve.1. Ts, Hs,
(1)
where Concat(- :] denotes the concatenation operation. In this way, the implicit scale representation S of the source image can be learned.
Scale-aware memory learning. As discussed before, human faces present a diverse range of scales in reality. Compared to directly using the global meta memory for facial-fcature compensation, we believe that a scalc-dependent condition on the meta-memory is a more intuitive and effective way. Therefore, we propose to condition the learned implicit scale representation memory M, to obtain an identity-
on the meta scale-aware memory M, for each face image. Inspired by the style injection in StyleGANv2 (Karras et al.) (2200), we utilize the implicit scale representa- tion S to manipulate a 3 x 3 convolution layer to produce the implicit scale-aware facial memory:
wyyk = 8; * Wijk and W"JK = wyy V (w)e()))))
where w is the weight of the convolution kernel, c a small constant avoid numerical issues, 8; is the i-th element in the learned implicit scale representation S, and and & enumerate the output feature maps and spatial footprint of the convolution, respectively. Finally, we obtain the scale-aware memory as:
M, = Fc,,, (M,)
where the Fo.... is the manipulated convolution layer parameterized by w" With the aware memory bank M., each input sample can be compensated by the scale-correlated facial priors, resulting in better generation performance discussed in the experiments.
3.3 GLOBAL MEMORY COMPENSATION AND GENERATION
The warped feature map contains ambiguity for generation caused by large head motion or oc- clusion, we thus propose inpaint those ambiguous regions via compensating the warped facial features. To this end, we design memory compensation module (MCM, see Fig. 4 to refine the warped feature Fí via the learned scale-aware facial memory bank.
Warped facial feature projection. To maintain better the identity information in the source image while compensating the warped feature from the source, we employ channel-split strategy to split the warped feature F into two parts along the channel dimension: Fi.0 and Ft.. The part of the first half channels F is left to directly pass through for contributing for the identity preserving, ve while the part of the rest half channels Ft.l is modulated by the learned scale-aware memory bank M, to refine the ambiguities. After splitting, we employ a x 1 convolution layer on F to change the channel number, resulting in projected feature F Prros
Warped facial feature compensation. To compensate the feature map spatially, we adopt a dy- namic cross-attention mechanism to refine features. Specifically, we employ the scale-aware mem-
5

---

Under review as a conference paper at ICLR 2023
C1/2
C//2
Dyramic Cross-atliationn Machanism
p4o Split
P
Cony
F ymg
Cony
Werpod Festure Fo
Consistemey
Cie
fa
Implicit Seale
Conditioned
Cony
Compersated Feanure F
Memory Modulo
Meta Memory M.
Scale-
Value
M,
Figure 4: The illustration of the memory compensation module (MCM). The symbol X denotes matrix multiplication, and får. and fåe are dynamic convolution layers (Chen et al. 2020 whose kernel weights are estimated by the projected feature F proj" The ( represents the concatenation operation, and the "Conv" denote convolution layer. C: is the channel number of the i-th level feature in our autocncoder framework, while Cm is the channel number of the memory bank.
ory to produce the Key F and Value FV via two dynamic convolution layers (i.e. fdce f2.) con- ditioned on the projected feature F proj- In this way, the generated Key and Value are identity- dependent and capable of providing useful context information. We, in the meanwhile, perform non-linear projection to map F proo into query feature F22 by 1 x 1 convolution layer followed by a ReLU layer. Then, we perform cross attention to reconstruct a more robust feature F as:
F = Fcixi (Softmax (F&² T x
F
x Fi.)
(3)
where "Softmax" denotes the softmax operator, while the Feix is a 1 x 1 convolution layer to change the channel number of the cross-attention output. "x" denotes matrix multiplication. As shown in Fig. 4 to maintain the identity of the source image, we concatenate the cross-attention features F with the first half-channels Fi.0.
F ept = Concat(F" ca. FiJ),
(4)
where the Concat|:: represents a concatenation operation. As result, the final output feature F enjoys the benefits of directly incorporating learned facial prior information (Wang et al.) 202 from the memory and effectively modulating by the dynamic cross-attention mechanism.
Regularization on consistency. To learn the global and the most generic spatial facial appearance and structure representations from the input faces, we need to make the learning of the meta mem- ory constrained by each single image in the training data. Simply but effectively, we enforce the consistency between the projected feature F
proj from the current training face image, and the value
feature Fi from the global meta memory:
Leon
F de proj)
1|1,
(5)
where the de(-) indicates a gradient detach function and || 1/1 is L1 loss. By using this function, the regularization enforces the consistency on the learning of the global meta-memory while not affecting the learning of the source image features to guarantee the training stability of the overall generation framework. The above equation also makes sure that the optimization gradients from all the face images during the training state contribute together to the updating of the memory bank, and thus it can capture global facial appearance and structure representations for the transferring.
Multi-layer generation. A higher-resolution feature map contains more facial details, while smaller-resolution one contains more semantic information. We perform memory compensation for feature maps of each layer to preserve facial details TPSM (Zhao & Zhang 2022). As shown L. in Fig. We utilize the motion flow AgtD to warp the encoded feature {F Timi in each layer to produce warped features (F', J For each warped feature F we feed it into our designed ISCM
and the MCM modules sequentially to produce the compensated features {F 1- the decod- ing process, we treat the F1 cpe as F} and then the F is generated by F] through an upsampling layer. At i-th level (i > 1), the output compensated feature F cpt will be concatenated with the decoded feature F1 at the same level to produce decoded feature Fi+1 the next level via an upsampling layer. Finally, input the concatenation of F. and FN into convolution layer followed by Sigmoid unit to generate the final facial image 1,st. Each layer shares the same meta memory M..

---

Under review as a conference paper at ICLR 2023
IC
IC
OR
Driving
FOMM
DaGAN IPSM
MCNet
Driving
FOMM
DaGAN 'ISMM
MCNet
(a) Same-identity Reenactment
Cross-identity Reenactment
Qualitative comparisons of (a) same-identity reenactment and (b) cross-identity reenact- ment on the VoxCelebl (the first two rows) and CelebV dataset (the last two rows). Our method shows nigher-fidelitg generation compared to the state-of-the-an Zoom in for best view.
Figure 5:
3.4 TRAINING
Loss objectives. We train the proposed MCNet by minimizing the following losses:
L A= + Aeg + Adisthdist + Xeen
(6)
where the Ap, Aeg. Adist and Acon are the hyper- hyper-parameters to allow for balanced learning from these losses. Per FOMM (Siarohin et al. 2019), we leverage the perceptual loss Lp to minimize the gap between the model output and driving image and equivariance loss Leg to learn more stable keypoints. Additionally, we also adopt the keypoints distance loss Ldist (Hong et al.] (022) to avoid the detected keypoints crowding around a small neighbourhood. The Lcon is the consistency loss in Eq. 5) The details of these losses are described in Appendix.
4 EXPERIMENTS
In this section, we present quantitative and qualitative experiments to validate the performance of MCNet. More details (i.e. additional results and implementation) are included in the Appendix.
4.1 DATASETS AND METRICS
Dataset. In this work, we mainly evaluate our MCNet on two talking head generation datasets, i.e. VoxCelebl (Nagrani al. 2017) and CelebV (Wu al.] (018)) dataset. We follow the sampling strategy for the test set in DaGAN (Hong et al., 2022) for evaluation. Following DaGAN, to verify the generalization ability, we apply the model trained on VoxCelebl to test on Celeb
Metrics. We adopt the structured similarity (SSIM), peak signal-to-noise ratio (PSNR), and L1 dis- tance to measure the low-level similarity between the generated image and the driving image. Fol- lowing the previous works (Siarohin et al. 2019), utilize the Average Euclidean Distance (AED) to measure the identity preservation, and Average Keypoint Distance (AKD) to evaluate whether the motion of the input driving image is preserved. We also adopt the AUCON and PRMSE, similar to Hong et al. (2022), to evaluate the expression and head poses in cross-identity reenactment.
4.2 COMPARISON WITH STATE-OF-TIEE-MA METHODS
Same-identity reenactment. In Table I a). we first compare the synthesised results for the setup which the source and the driving images share the same identity. can be observed that our MCNet obtains the best results compared with other competitive methods. Specifically, compared with FOMM (Siarohin et al., 2019) and DaGAN (Hong et al. 2022). which adopt the same motion method can produce higher-quality images (72.3% of FOMM vs
estimation method as ours, 82.5% of ours, resulting in a 10.2% improvement on the SSIM metric), which verifies that introduc- ing the global memory mechanism can indeed benefit the image quality in the generation process. Regarding motion animation and identity preservation, our MCNet also achieves the best results (i.e. 1. 1.203 AKD and O. 100 AED), showing superior performance the talking head animation.
7

---

Under review as a conference paper at ICLR 2023
M, Figure 6: Qualitative ablation studies. The memory compensation module (MCM) and implicit scale conditioned memory module (ISCM) can obtain improvements. The last column verifies that our ISCM can learn different ccare memories for different scale samples.
Source
Driving Warped image
Baseline Baseline+ MCM MCNet
Table Comparisons with state-of-the-art methods on (a) same-identity reenactment on VoxCelebl and (b) cross-identity reenactment on VoxCeleb1 and CelebV dataset.
(a)
Same-identity Resnactment
Cross-ideniity Reenactment
Model
VoxColeb]
VoxColebi
CclcbV]
SSIM (2)
PSNR
LPIPS
4
AKD L
AED
AUCON PRMSE
AUCON | PRMSE
X2face Wiles et ml.
2018
71.9
22.54
0.0780
7.687
0.405
0.679
3.62
marioNETI
75.5
23.24
0.710
FOMM Siarotin cl
2019
72.3
30.39
0.199
0,0430
1.294
0.140
0.882
2.824
0.667
3.90
MeshG
Yab all
73.9
face-vid 2yid (Wang di
202
76.1
30.69
0.212
0.0430
1.620
0.153
0.839
4.398
0,805
3.15
MRAA (Sarohin et
221
80.0
31.39
0.195
0.0375
1.296
0.125
0.882
2.751
0.840
2.46
DaGAN Hong
2010
80.4
31.22
0.185
0.0360
1.279
0.117
0,888
2.822
0.873
2.33
TPSN
ZZhuo & Zhang
2022
81.6
31.43
0.179
0.0365
1.233
0.119
0.894
2.756
0.882
2.23
MCNet (Ours)
82,5
31.94
0.174
0.0331
1,203
0.106
0.895
2.641
0.885
2.10
Moreover, we show several samples Fig. 5(a), and the face samples in Fig. 5 a) contain large mo- tions (the first, the third, and the last row) and object occlusion (the second row). From Fig-5la), our model can effectively handle these complex cases and produces more completed image generations compared with the state-of- state-of-the-art the-art competitors.
Cross-identin reenactment. We also perform experiments on the VoxCelebl and CelebV datasets to conduct the task of the cross-identity face motion animation, in which the source and driving images are from different people. The results compared with other methods are reported in Table [ Our MCNet outperforms all the other comparison methods. Regarding the head pose imitation. MCNet can produce the face with a more accurate head pose (i.e. 2.641 and 2.10 for VoxCeleb1 and the PRMSE metric). We also present several samples of results with the VoxCelebl dataset in Fig. (b). is clear to observe that our MCNet can mimic the facial expression better than the other methods, such as the smiling countenance shown in the first row. For the unseen person in the CelebV dataset, e.g. the last two rows in Fig. ()), our method can still produce more natural generation, while the results of other methods contain more obvious artifacts. All of these results verify that the feature compensated by our learned memory can produce better results.
Celeb', respectively,
4.3 ABLATION STUDY
In this section, we perform ablation studies to demonstrate the effectiveness of the proposed implicit scale conditioned memory module (ISCM) and memory compensation module (MCM). We report the quantitative results in Table and the qualitative results in Fig. 6 Our baseline is the model without ISCM and MCM modules. The "Baseline + MCM" means that we drop the ISCM module and replace the scale-aware memory M, with the meta memory M, in Fig.
Meta Memory learning. We first visualize the learned meta memory in Fig. which aims to learn the global and generic facial appearance and structure representations. In Fig. channels of the meta memory bank. It can be observed that all these channels represent the faces with different appearances, structures, poses, and shapes, which are very informative and clearly beneficial for the facial compensation and generation, confirming our motivation of learning global facial representations to tackle ambiguities in the talking head generation.
visualize partial
8

---

Under review as a conference paper at ICLR 2023
Figure 7: The visualization of randomly selected channels of the meta memory M.- can observe that our meta memory learns very diverse global facial representations. Zoom in for best view.
Ablation studies: "Baseline" indicates the simplest model without the implicit scale condi- tioned memory module (ISCM) and memory compensation module (MCM). "MCM"/o Ea means that we project the entire warped feature into projected feature F, "proj- and remove the concatena- tion function Eq.5 to make the output of cross-attention as the compensated feature F
Table 2:
cpt
Model
SSIM (%) f
PSNR t
LPIPS
L1 4
AKD 4
AED 4
Baseline
81.1
31.70
0.182
0.0356
1.303
0.124
Bascline + MCM Be
82.0
31.82
0.176
0,0340
1.242
0.119
Bascline MCM
82.3
31.92
0.175
0.0334
1.237
0.114
Baseline + ISCM + MCM (MCNet)
82.5
31.94
0.174
0.0331
1.203
0.106
FOMM Siarohin al.
2019
72.3
30.39
0,199
0.0430
1.294
0.140
FOMM+ ISCM + MCM
81.8
31.73
0.179
0.0353
1.269
0.119
TPSN Zhao & Zhang,
2022
81.6
31.43
0.179
0.0365
1.233
0.119
TPSN+ ISCM + MCM
82.0
31.55
0.175
0.0356
1.216
0.115
Memory compensation In Table and Fig. 6 the proposed memory compensation module can effectively improve the generation quality of human faces. From Tab. 2 we observe that adding the memory compensation module (MCM) can consistently boost the performance via comparison between "Baseline+MCM" and "Baseline" (82.3% vs. 81.1% 1% on SSIM). In Fig. 6 we also can see that the variant "Baseline+MCM"" compensates the warped image better than the "Baseline", , e.g. the face shape in the second row and the mouth shape in the third row. Additionally, we also conduct an ablation study to verify the feature channel split strategy discussed in Sec. [3.3] The results of "Baseline + MCMw/o Eq show that the channel split can slightly improve the performance. All these results demonstrate that learning global facial memory can indeed effectively compensate the warped facial feature to produce higher-fideliyy results for the talking head generation.
Scale-aware memory learning. To verify the effectiveness of the implicit scale conditioned mem- ory module (i.e. ISCM introduced in Sec. (.2), we show the randomly sampled channels of the scale-aware memory in Fig [ As shown in the last column in Fig 6 the ISCM can produce an identity-dependen scale-aware memory bank, which have structural and scale relations with the in- put source images. By deploying the ISCM, our MCNet compared with the "BMSelineMMCM", verifying that the learned scale-aware memory condioned on the input source can provide better compensatation the source feature for more vivid generation.
produce highly realistic-lookin images
Generalization experiment. Importantly, we also insert the proposed MCM and ISCM modules into FOMM (Siarohin et al. 2019) and TPSM (Zhao & Zhang, 2022) verify designed mem- ory mechanism can be flexibly generalized to existing talking head models. As shown in Table 2 the TPSM, which has different motion estimation method compared to ours, deployed with our proposed memory modules, can achieve a stable improvement. The "FOMM-ISCM-MCNM can also gain significant improvement on SSIM compared with the pioneering work "FOMM". These results demonstrate the transferability and generalization capabilities of the proposed method.
5 CONCLUSION
In this paper, we present an implicit seale conditioned memory compensation network (MCNet) to learn a global facial prior of spatial structure and appearance to address the ambiguity problem caused by the dynamic motion the talking head video generation task. MCNet utilizes designed implicit scale conditioned memory module to learn the scale-aware memory for different samples, which will be used to compensate the feature map in the memory compensate module. Ablation studies clearly show the effectiveness of learning global facial meta memory for the talking head video generation task. Our MCNet also produces more natural-looking results compared with the state-of-the aft on all benchmarks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tadas Baltrušaitis,, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behavior analysis toolkit. In WACV, 2016.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). In ICCV, 2017.
Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In CVPR, 2020.
S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In INTER- SPEECH, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11999 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Zhengcong Fei. Memory-augmenter image captioning. In AAAI, 2021.
Xiaojie Guo, Siyuan Jinke Yu, Jiawan Zhang, Jiayi Ma, Lin Ma, Wei Liu, and Haibin Ling. Pfld: A practical facial landmark detector. arXiv preprint arXiv:1902108554 2019.
Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Sco, and Dongyoung Kim. Marionette: Few- shot face reenactment preserving identity of unseen targets. In AAAI, 2020.
Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In CVPR, 2022.
Huaibo Huang, Aijing Yu, and Ran He. Memory oriented transfer learning for semi-supervised image deraining. In CVPR, 2021.
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In CVPR, 2021.
Xinya Ji, Hang Zhou, Kaisiyuan Wang. Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. arXiv preprint 1522::2205.15278, 2022.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In CVPR, 2020.
Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong. Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Expressive talking head generation with granular audio- visual control. CVPR, 2022.
Peirong Liu, Rui Wang, Xuefei Cao, Yipin Zhou, Ashish Shah, Maxime Oquab, Camille Couprie, and Ser-Nam Lim. celf-appearance-aiiee aiddd differential evolution for motion transfer. arXiv preprint arXiv:2110.04653 046046 202
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 20.
Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: real-time photorealistic talking- head animation. TOG, 2021.
Ricardo Martin Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identifi- cation dataset. arXiv preprint arXiv: 08612, 2017.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
Haibo Qiu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. End2end occluded face recogni- tion by masking corrupted features. TPAMI, 2021.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. NeurlPS, 2019.
Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In CVPR, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint vrXiv:1409. 1556, 2014.
Guanxiong Sun, Yang Hua, Guosheng Hu, and Neil Robertson. Mamba: Multi-level aggregation via memory bank for video object detection. In AAAI, 2021.
Jiale Tao, Biao Wang, Borun Xu, Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. Structure- aware motion transfer with deformable anchor model. In CVPR, 2022.
Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In CVPR, 2018.
Soumya Tripathy, Juho Kannala, and Esa Rahtu. Facegan: Facial attribute controllable reenactment gan. WACV, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Qiulin Wang, Lu Zhang, and Bo Li. Safa: Structure aware face animation. In 3DV, 2021a.
Ting- Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural lalking-head synthesis for video conferencing. In CVPR, 2021b.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. CVPR, 2021c.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv: 1rXiv:1410.39/66 2014.
arXiv preprint
Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In ECCV,
Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. Imitating arbitrary talking style for realistic audio-driven talking face synthesis. In ACM Multimedia, 2021a.
Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy. Reenactgan: Learning to reenact faces via boundary transfer. In ECCV, 2018.
Xintian Wu, Qihang Zhang, Yiming Wu, Huanyu Wang, Songyuan Li. Lingyun Sun, and Xi Li. F'a-gan: Facial flow for face animation with generative adversarial networks. TIP, 2021b.
Zhuojie Wu, Xingqun Qi, Zijian Wang, Wanting Zhou, Kun Yuan, Muyi Sun, and Zhenan Sun. Showface: Coordinated face inpainting with memory-disentangle refinement networks. arXiv preprint SXXi:2224.008222 2022.
Rui Xu, Minghao Guo, Jiaqi Wang, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Texture memory-augmented deep patch- based image inpainting. JEEE Transactions on Image Processing, 2021.
11

---

Under review as a conference paper at ICLR 2023
Guangming Yao, Yi Yuan, Tianjia Shao, and Kun Zhou. Mesh guided one-shot face reenactment using graph convolutional networks. In ACM MM, 2020.
Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jachyuk Chang, and Jaegul Choo. Coloring with limited data: Few-shot colorization via memory augmented networks. In CVPR, 2019.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. ICCV, 2019.
Egor Zakharov, Aleksci Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one-shot realistic head avatars. In ECCV, 2020.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In CVPR, 2021.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, 2022.
Ruiqi Zhao, Tianyi Wu, and Guodong Guo. Sparse to dense motion transfer for face image anima- tion. In ICCV, 2021.
Xiangyu Zhu, Xiaoming Liu, Zhen Lei, and Stan Z Li. Face alignment in full pose range: A 3d total solution. TPAMI, 2017.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A REPRODUCIBILITY
In this work, we utilize the PyTorch framework to implement our method. We develop our code based the FOMM" All experiments in this work are conducted publicly available datasets. Hyperparameters necessary for reproducing our experiments are reported "Implementation De- tails".
B TRAINING DETAILS AND ADDITIONAL NETWORK B. IMPLEMENTATION DETAILS
The keypoint estimator and dense motion are borrowed from the FOMM (Siarohin et al. 2019). We extract each frame from the driving video as a driving image and input into the MCNet model with the source image. The source image and driving video share the same identity in the training stage, so that the ground-truth the driving frame during training. To optimize the training objectives, we set the Aree 10, Xey 10, Adist 10 and Xcon 10. The number of keypoints in this work is 15, which is the same as that of DaGAN (Hong et al. 2022). the training stage, we employ RTX the model for 100 epochs in end-to mn-tto-end manner, and it costs about 12 hours in totally. The number of layers of encoder and decoder N is set as 4 and the number of keypoints K 15 as Hong al. 2022). We set the size of meta memory as 512 32 32, where C., 512, Hm = 32 and W = 32. In the warping process, for those F that has different spatial size as the motion flow, we employ the bilinear interpolation method to adjust the spatial size of the motion flow.
3090 GPUs
B.2 LOSS DETAILS
Perceptual Loss Cy. Perceptual loss is popular objective function in image generation tasks. As introduced in DaGAN (Hong et al. 2022), generated image and its ground-truth. i.e. driving image the training stage, is downsampled 64 x 64, 32 x 32) respectively. Then we utilize a pre-trained VGG network (Simonyan & Zisserman, 2014) to extract the features from each resolution image. To simplify, we denote R1, R2, Rg, Rs as the features of generated images in different resolutions, respectively, and G1, G2, G3, G4 for different resolutions of ground-truth. Then, we measure the L₁ distance between the ground-truth and generated image the Perceptual loss:
different resolutions (i.e. 256 x 256, 128 128,
L. Σ GG, Ra)
Equivariance Loss Leg. We employ this loss to maintain the consistency of the estimated keypoints the images after different augmentations. Per FOMM (Siarohin et al.] 2019), given an image and its detected keypoints {X (X, E RIx2),
then perform known spatial transformation T on images and keypoints {X;) resulting in transformed image Iy and transformed keypoints LK
{ Then, we use detect the keypoints on the transformed image IT, denoted as (KX
We employ the equivariance Loss on the source image and driving image:
K
Leg
x
X X
(8)
Keypoint distance loss Ldist- We employ the keypoint distance loss as Hong et (2022) to pe- nalize the model if the distance between any two keypoints is smaller than user-defined threshold. Thus, the keypoint distance loss can make the keypoints much less crowded around small neigh- bourhood. In one image, for every two keypoints X, and X,, we have:
13

---

Under review as a conference paper at ICLR 2023
DownBlock2d(c 64,/2)
Dowwnlocc22d(c- 128,/2)
DownBlock2d(c = 256 (26
Down 512.
DownBlock2d(C = 1024./2)
UpBlock2d(c = 512,22)
UpBlock2d(c = 256,x2)
UpBlock2dcc = 129,x
UpBlock2d(c = 64,x2)
UpBlock2dCC = 32,x2)
3x3 Conv, BN,
Pooling /2
7x7 - Conv
(b) DownBlock2d
Interpolation x2
3x3 Conv, BN,
(a) Keypoint Detector
(c) UpBlock2d
Figure 8: The structure of keypoint detecotor. The in each layer means the output channel.
K
Ldist ΣΣO
(1 sign||
X,111 - -a. i
where the sign(.) represents sign function and the is the threshold of distance, which is 0.2 in this work.
B.3 NETWORK ARCHITECTURE DETAILS OF MCNET
The keypoint detector receives image input and outputs the K keypoints {2h, Ut Merr= The structure of keypoint detector is illustrated in Fig. g Here, we adopt the Taylor approximation as FOMM (Siarohin al. 2019) and DaGAN (Hong et al. (022) compute the motion flow. Thus, the motion estimation is not our contribution and we mainly focus on our meta memory.
C EXPERIMENTS
C.1 METRICS
We mainly introduce four important metrics in talking head talks generation field, i.e., AED, ADK, PRMSE, and AUCON. Specifically, Average euclidean distance (AED) is an important metric that measures identity preservation in reconstructed video/image. In this work, we use OpenFace (Bal- trušaitis al. 2016) extract identity embeddings from the reconstructed face and the ground truth frame pairs. The MSE loss is employed to measure their difference.
Average keypoint distance (ADK). ADK evaluates the difference between landmarks of the recon- structed faces and ground truth frames. We extract the facial landmark using the face alignment method (Bulat & Tzimiropoulos/ 2017). We compute the average distance between corresponding keypoints. Thus, the AKD mainly measures the ability of pose imitation.
of the head pose angles (PRMSE). In this work, utilize the Py- Feat toolki detect the Euler angles of head pose, and then evaluate the pose difference between different identities.
The root mean square
The ratio of identical facial action unit values (AUCON). We first utilize the Py-Feat toolkit to detect the action units of the generated face and the driving face. Then we can calculate the ratio of identical facial action unit values as the AUCON metric.
C.2 OTHER EXPERIMENTAL RESULTS
Positional Encoding for keypoints. The positional encoding method shows its strong power in the
transformer VVaswani et al. 2017 Liu et al. 2021b Dosovitskiy et al. 2020) and Nerf (Mildenhall
fuuas:/ppyffatt.org
14

---

Under review as a conference paper at ICLR 2023
Table 3: The results of applying positional encoding function keypoints. "pe(10)" means the output dimension control factor L of positional encoding function as 10. and 20 for "pe(20))
Model
SSIM (%)
PSNR
LPIPS
C1 +
AKD
AED +
Ous w/ pe(10) 82.4
31.91
0.175
0.0334
1.221
0.1073
Ous w/ pe(20)
69.4
30.03
0.269
0.0593
5,544
0.2684
MCNet 82.5
31.94
0.174
0.0331
1.203
0.1065
and "ISCM w/o keypoints" represent that ISCM does use the projected feature F proj or keypoints as input (see Fig. 3). respectively, encode implicit scale representation. "MCM w/o fee fāe indicates that we replace the fåe and få. with two normal convolution layers to produce Key and Value in MCM.
Table 4: Ablation studies. 'ISCM w/o F Fproj" and
Model
SSIM (%)
PSNR f
LPIPS
AKD
AED
MCNet (ISCM w/o Fj Fjrros)
82.3
31.89
0.175
0.0336
1.237
0,1097
MCNet (ISCM w/o keypoints)
82.4
31.93
0.175
0.0333
1.227
0.1087
MCNet (MCM w/o fdc: flc)
82.2
31.89
0.176
0.0336
1.246
0.1119
MCNet (single layer)
82.3
31.90
0.175
0.0334
1.235
0.1085
MCNet
82.5
31.94
0.174
0.0331
1.203
0.1065
et al. 2020 Martin-Brualla et al. 2021 Pumarola et al. 2021). Therefore, we consider applying the positional encoding function to our keypoints when we produce the scale-aware memory. We show the results in Tab 3 From the Tab. 3 we observe that when apply the position encoding function on keypoints cannot bring the improvements, and even degrade the model if we set the as 20. Since the keypoints will be utilized to estimate the motion flow in dense motion network, the Euclidean distance between any two keypoints physically meaningful. Therefore, we suppose that employing the positional encoding keypoints may affect the motion flow estimation, resulting in bad generation.
The input element in ISCM. We also conduct experiments investigate the usage of intermediate feature F proj ("ISCM w/o F "roj"" ") and keypoints (ISCM w/o keypoints), the results shown Tab, indicate that these two items equally crucial for the generation of the scale-aware memory bank. We can obtain the best results when we combine them together.
Single layer multi layer. In our work, we deploy the ISCM and MCM in each layer to obtain the best results. Also, we investigate the performance of using ISCM and MCM in the first layer only. The results "MCNet (single layer)" show that the single layer can also obtain similar good results, which verify the effectiveness of our designed memory mechanism.
The dy dynamic convolution in MCM. Besides, also conduct an ablation study on the dynamic convolution layer in the memory compensation module (see Fig 4). We can observe that the dynamic convolution layer can contribute to the final performance, especially for the AKD and AED.
Table 5: Comparisons results on VoxCeleb2.
Model
SSIM (%) +
PSNR
LPIPS
C 4
AKD 1
AED L
FOMM
77.19
30.71
0.257
0.0513
1.762
0.212
MRAA
78.07
30.89
0.262
0,0511
1.796
0.213
DaGAN
79.02
30.81
0.250
0.0483
1.865
0.341
TPSM
78.22
30.63
0.254
0,0527
1.703
0.210
Ours w/o ISCM
78.63
31.02
0,250
0.0481
1.726
0.199
Ours
79.86
31.18
0.244
0.0470
1.699
0.186
Here, the implementation of http://gtiiaecmmyyyyaaaeeirrrrre
15

---

Under review as a conference paper at ICLR 2023
Table 6: Comparisons results HDTF dataset.
Model
SSIM (%) f PSNR
LPIPS
Li 4
AKD 4
AED 4
FOMM
76.94
31.87
0.155
0.0363
1.116
0.092
MRAA
79.36
32.32
0,156
0,0331
1.039
0.100
DaGAN
82.29
32.29
0.136
0.0304
1.020
0.252
TPSM
86.05
32.85
0.114
0.0264
1.015
0.072
Ours w/o ISCM
85.90
33.03
0.114
0.0243
1.023
0.068
Ours
86.45
33.60
0.112
0.0238
0.998
0.064
More datasets for evaluation. To fully verify the superiority of our method, we also compare it with other methods on two other large datasets, i.e. VoxCeleb2 (Chung al., 2018) and HDTF (Zhang it al. 2021). We report the results on Tab. 5 and Tab. From these two tables, we can observe that our method can still obtain the best results compared with other SOTA methods It strongly illustrates the superiority of our designed method.
Idenetity Preservation. In this section, we reorganized the voxcelebl dataset and divided into a training set and test set. These two sets have the same identity space. That is, the identities of test videos also appear the training videos. We select 500 videos as the test set and the rest as the training set. The experimental results are shown in Tab. 7 We can observe that our method obtains higher performance under the setting of testing identity as part of the training corpus. One possible reason is that our global memory is learned from the identities in the training set. In this way, it can better compensate for the facial details of these seen identities.
Table 7: Comparisons results on HDTF dataset.
Model
SSIM (%)
PSNR
LPIPS |
L1
AKD
AED
Identities not in the training set
82.5
31.94
0.174 0.163
0.0331
1.203
0.106
Identities in the training set
83.6
32.38
0.0319
1.164
0.102
Video generation demo. We also provide several video generation demos to show a more detailed comparison qualitatively with the most competitive methods in the literature. From demo videos, we can observe that our main can compensate those regions that do not appear the source image better than other methods (e.g. the ear region in demo2). These demos are attached in Supplementary Material.
Comparison in other domains. To better verify the generalization ability of our method, also train our method on TED-talks dataset (Siarohin et al.) 2021). because the human body is also symmetrical and highly structured. We report the results Tab. From the Tab. g our method still obtain the best results among all compared method. This generalization experiments verity that our memory can learn the symmetrical and structured object to inpaint the generated image.
Table Comparisons results on TED-talks dataset.
Model
L1
(AKD 4. MKR 4)
AED +
FOMM
0.033
(7.07, 0.014)
0.163
MRAA
0,026
(4.01, 0.012)
0.116
TPSM
0.027
(3.39, 0.007)
0.124
Ours
0.023
(2.52, 0.006)
0.101
Meta memory visualization. In this section, we show all channels of our learn meta memory in Fig. 9 for better understanding. To look into details, we also show some channels in Fig. I0 in high resolution. These visualizations demonstrate the meaningful facial priors learnt the meta memory.
*These compared methods have official released code for
these datasets.
16

---

Under review as a conference paper at ICLR 2023
Figure 9: The visualization of randomly selected channels of the meta memory M..
More qualitative ablation studies To better show that our designed implicit scale conditioned memory module brings improvement to our model, we illustrate more qualitative results for ablation studies in Fig. and Fig. 12
17

---

Under review as a conference paper at ICLR 2023
Figure 10: The visualization of all channels of the meta memory M..
18

---

Under review as a conference paper at ICLR 2023
Source
Driving
Baseline
Baseline + MCM
Ours
Qualitative ablation studies in VoxCeleb1 dataset. The memory compensation module Figure 11:
(MCM) and implicit scale conditioned memory module (ISCM) can obtain improvements.
19

---

Under review as a conference paper at ICLR 2023
Source
Driving
Baseline
Baseline + MCM
Ours
Qualitative ablation studies in VoxCeleb1 dataset. The memory compensation module Figure 12:
(MCM) and implicit scale conditioned memory module (ISCM) can obtain improvements.
20


Reviews:
[{"review_id": "YX9atOIdxt", "reviewer": "Reviewer_gGqi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new video-driven talking head generation method based on memory band learning and implicit scale representation learning. The proposed model, named as MCNet, consists of:\n1) a keypoint detector and the dense motion network that predicts keypoints from source and driving frame and then estimates optical flow between them,\n2) an implicit scale conditioned module ISCM that encodes implicit scale representation from warped feature and source keypoints, which is then conditioned on the query of the meta memory to produce identity-independent scale-aware memory band,\n3) a memory compensation module MCM that utilizes a dynamic cross-attention mechanism to spatially compensate the warped features, which is finally produced into final image by decoder.\nThe MCNet learns global facial meta memory bank to obtain global facial priors on spatial structure and appearance from all available training faces and compensate for dynamic facial synthesis, addressing the ambiguities in face generation especially in situation of large head pose change.\n", "summary_of_the_review": "This paper proposes a novel method based on good observation of human face and effectively improves the performance of facial reenactment.", "strength_and_weaknesses": "Strengths:\n+ The paper proposes to learn a global facial meta memory bank to transfer representative facial representations to handle the appearance and structure ambiguities caused by the highly dynamic generation from a still source image\n+ The paper proposes an implicit scale conditioned memory module (ISCM) to learn implicit scale representation from warped feature and source keypoints and addresses the problem that different source faces contain distinct scales.\n+ The paper proposes a memory compensation module (MCM) to compensate and refine the warped feature map, which utilized the learned global prior facial information to generate better results.\n+ The qualitative and quantitative comparisons to STOA in both same-identity reenactment and cross-identity reenactment show the effectiveness of the proposed method.\n+ Ablation studies are conducted to visualize the learned meta memory and show the effectiveness of each module.\n\nWeaknesses:\n- The method is proposed as a talking head generation method, but many compared SOTA methods such as FOMM, MRAA and TPSN are general image animation methods capable of animating images from different domains, which makes the comparison with these methods in face image domain somehow less convincing.\n- The proposed implicit scale conditioned memory module (ISCM) learns implicit scale representation which is hard to visualize or compare to show its effect, and the quantitative results of the ablation study show that the ISCM module has only very slightly improvement on the evaluation metrics.\n- The evaluation datasets are not sufficient. It’s better to perform experiment on some new datasets such as VoxCeleb2 and high-resolution dataset HDTF.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WE40hFj23v", "reviewer": "Reviewer_dF7D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presented a method for learning talking head generation/reenactment. Specifically, the paper proposed a memory bank based architecture to address the inconsistency caused by varying shape/appearance of source and driving signals. A learnable external prior is introduced to learn and store the shape/appearance coupling. Some other modules and architectural designs are also proposed in the paper to improve the performance of the method, specifically handling the scale difference between source and driving signals. The paper provided qualitative and quantitative comparisons with some existing approaches, including, FOMM, DaGAN, TPSM etc. The proposed method achieves better performance on image quality and identity preservation, albeit with relatively small margin. The paper also provided quantitative ablation study to demonstrate the effectiveness of each component.", "summary_of_the_review": "I appreciate the motive, method, and results of the paper. I have some complaints about the method being complicated and might be challenge to reproduce (I hope the code and model will be published alongside the paper). I also have some minor questions/suggestions about the paper as mentioned above. Overall I'm positive on this work.", "strength_and_weaknesses": "Strength:\n\n1. The problem that the paper is aiming to address is interesting and is an essential challenge in face reenactment work. Keypoint and warping based method for transferring dynamic facial appearance often suffers from inconsistency between the driving signals and target image, especially in terms of shape and scale of the images.\n\n2. The approach is novel to me by leveraging a memory bank as a global face appearance representation and use it for a face reenactment task.\n\n3. The experimental results demonstrated the advantage of the proposed method. The qualitative results in the paper and supplemental video showed clear advantage over previous work. The quantitative performances on image quality and identity preservation evaluation also outperformed previous work.\n\n\nWeaknesses:\n\nNo major concern from me except a few minor questions:\n\n1. Can this method scale to very high resolution images? Would the facial details be challenge or too expensive to learn with this method?\n\n2. The method seems still have issue with identity preservation, which is a common issue in such tasks. Since the method is learning a memory bank from training data, would it make any difference if the testing identity is a part of the training corpus? Perhaps an ablation study on identity preservation could help explain.\n\n 2. Ablation study. The network design is quite complicated, which is my major concern with this paper. I'm not sure how much of them could be simplified. It is good that the author provided the ablation study to show the accumulating benefit of each component, but it's a little disappointing that each of which doesn't seem to contribute much upon the baseline. More importantly, there seem to be lack of qualitative comparisons for the ablation study. How much difference can we see in the results by adding the proposed architectural designs?\n\nSome inconsistency or typos:\n\n- In section 3.1, the paper refer Ms as \" identity-independent scale-aware memory bank Ms\". But in section 3.2, last paragraph, the paper stated \"person-dependent scale-aware memory Ms\" which is the direct opposite of the early statement. Is there a typo or am I missing something?\n\n- Section 3.2 last paragraph, \"scale-dependent scale-aware memory bank Ms\" -- why mention scale twice? Perhaps a typo?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9Lugb-wKM8", "reviewer": "Reviewer_StKu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new talking head generation method with memory compensation. The proposed method is novel with good experiment performances. The claimed implicitly scale condition is confusing, however, it could be addressed by editing the paper.", "summary_of_the_review": "The proposed method is novel and achieves the best performance on two datasets. There are some issues in this paper that could be addressed by re-writing part of the paper.", "strength_and_weaknesses": "Strength:\n- The proposed method is novel in terms of using the meta memory bak to transfer representative facial representations. Compared to the previous works [19], the main novelty comes from the memory module and compensation module. The two modules are reasonable to me and the author has clearly demonstrated their implementation of them. \n\n- The method is well demonstrated. The author has provided three figures Fig. 2, 3, 4 to demonstrate the architecture of the proposed method with detailed text. The proposed method is a bit complex, and I hope the author could release their source code upon acceptance. The author provides the visualization of the meta memory to show the learned memory content.\n\n- The proposed method achieves the best performance on two datasets for both same-identity reenactment and cross-identity reenactment experiments. The author also provides a detailed ablation study to show the benefit of meta-memory learning, scale-aware memory learning etc.\n\nWeakness:\n- It is not clear to me why the author claims \"scale\" conditioned. First, the author motivated the scale is important because adults and children have different face scales (beginning of section 3.2). However, there are no children in the data that they tested on. I do not understand why the author cares about the scale in this case. Second, there is no evaluation to show that the proposed method indeed implicitly learns the \"scale\" although the ablation study shows the scale module can improve the performance. I do not think the author could claim they have the \"implicit scale representation\". Last, the implementation of the scale is scaling of the convolutional weights, which is not the scale of the face. I do not think scaling the weights has the semantic meaning of scaling the face. Taken together, I am confused why the author focuses on the scale in this proposed method rather than the meta memory.\n\n- There are many parts of the method without an explanation of why the author does that. For example, the multi-layer generation in section 3.3 is upsampling operations that claim to \"preserve the details of the face\", which is not clear to me why and how. The Wrapped facial feature compensation has similar issues that I do not see why the author has this specific architectural design. Is that the author designed the module from the scratch by themselves, or did the author borrow this module from previous works (for which I did not see the citation)?\n\n\nMinor comments:\n- The images in Fig. 7 are too small to see and zooming in just ends up with blurry images. I suggest the author reduce the number of meta memory in Fig. 7 to have a better visualisation.\n- The author writes \"Specifically, compared with FOMM and DaGAN, which adopt the same motion estimation method as ours ...\". Does it mean the author takes their proposed method as a motion estimation method? I think the meta memory is the main character of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject