Free-View Expressive Talking Head Video Editing
Yuantian Huang, Satoshi Iizuka, Kazuhiro Fukui
University of Tsukuba

1Introduction

2

FREE-VIEW EXPRESSIVE TALKING HEAD VIDEO EDITINGØOur Goal

Yuantian Huang, Satoshi Iizuka, Kazuhiro Fukui• The goal of our research is to edit the talking head in a video with multiple
Universityof Tsukuba
attributes in full frames, including
headpose, facial emotion, and eye blink.

Input

Editing

Ouput

Figure1: Teaser.
Fig. 1: Our approach can edit the talking head in a videowith multiple
freely controllable attributes, including head pose, facial
3
emotion and eye blink. Edited frames can be restored to their full frames while maintaining audio-visual synchronization.

ØPrevious Research• Previous Research → Audio-driven Talking Head Generationble 1: We•compare
the featuresof our→proposed
approachwith
existing methods for talking head generation
Proposed
MethodEditing
insteadofdifferent
Generation
ting. Specifically,
edit in a fullthat
frame
but lacks
all theattributes fornaturalism,
the ourother approac
Unlike Wav2Lip
previouscan
approaches
mainlyfocus
onvitalgeneratingtalking
head whilevideos,ed to perform full frame editing. EAMM and GC-AVT can generate emotional talking head videos. Besides, PC-AVS
proposed method is able to edit the talking head and restore it back to the full frames,-AVT can apply head poses from reference videos, while our proposed method can adjust poses in free-view. Our appro
whichofsupports
a broader range of applications.
he most flexible
existing approaches.
Methods
Wav2Lip [1]
EAMM [2]PC-AVS [3]GC-AVT [4]Ours

Lip Sync

Full-Frame Editing

Head PoseXXXXX

XCopy source
Copy source
Reference required
Reference required
Free-view

XEmotion

Eye BlinkXX
XX

Table
1: Comparison
cted from the target audio, and additional
encoders
process withAn Previous
overviewWork.
of the model is shown in Fig. 2.
d pose code, emotion code and eye blink code extracted4

2

Proposed Framework5ØModel Input• Typical Input for Audio-driven Talking Head Generation§ Input frames and target frames with black-masked faces.
§ Audio Source that are corresponding to target frames.

• Attribute Input of our proposed method§ Head pose code that represents angles of the talking head (yaw, pitch, and roll).
§ Emotion code that indicates seven emotion categories.
§ Eye blink code that is related to eye openness.

EmotionEye Blink

Head Pose6ØModel Architecture
1. A set of encoders that embed inputs together and feed them into the generator.
All embed inputs are resampled to be the same length as the audio source (Mel Spectrogram).
Target Frames

Attribute Extraction

Eye Blink
Head Pose
EmotionopaBackprgation

L1 Loss and VGG loss

AudionEncoders
Input FramesroppkctioagaBa
Sync Losses

Generator
Generated Frames

Pre-trained Multi-Attribute
DiscriminatorFig. 2: Structure of our framework.
The2:encoders
embedinputs together
and feed them into the generator, while the input 7Figure
Structureof ProposedFramework.

ØModel Architecture
2. A generator that generates frames corresponding to the input.
L1 loss and VGG perceptual loss are used to improve the overall generation quality.
Target Frames

Attribute Extraction

Eye Blink
Head PoseEmotion

opaBackprgation

L1 Loss and VGG lossAudionEncoders
Input FramesroppkctioagaBa
Sync Losses

Generator
Generated Frames

Pre-trained Multi-AttributeDiscriminatorFig. 2: Structure of our framework.
The2:encoders
embedinputs together
and feed them into the generator, while the input 8Figure
Structureof ProposedFramework.

ØModel Architecture
3. A pretrained multi-attribute discriminator that calculates a set of synchronization losses
between generated frames and input attributes to enforce attribute-visual synchronization.
Target Frames

Attribute Extraction

Eye Blink
Head PoseEmotion

opaBackprgation

L1 Loss and VGG loss

AudionEncoders
Input FramesroppkctioagaBa
Sync Losses

Generator
Generated Frames

Pre-trained Multi-Attribute
DiscriminatorFig. 2: Structure of our framework.
The2:encoders
embed
inputs together
and feed them into the generator, while the input 9Figure
Structureof ProposedFramework.

ØLoss Function• Synchronization loss:
We minimize the L2 distance between embedding features that are extracted from thetarget frames and those from other inputs. So that we are able to enforce a synchronization
between generated frames and all the input attributes simultaneously.

• Full objective:

10

3Results

11ØAttributes Editing (Without Reference Input)• Our proposed framework is able to fully control three different types of facial
attributes, including head pose, facial emotion, and eye blink.
-30

~

30-15

Yaw

Yaw

Pitch

Open

Close

Angry~

15-15

Pitch

Roll~

15

Head Pose

Roll

Eye Blink

1~0HappyFull Frame[…,1,0,0]~[…,0,1,0]
Figure 3: Examples of Attributes Editing.

Fig. 4: Examples of Editing different types of attributes and restoring to the original frame.

Table 2: Quantitative comparison with existing approaches

Table 3: Ablation study using the FID metric. We evalu12

ructure of our framework. The encoders embed inputs together and feed them into the generator, while the input
spectrogram, head pose, emotion, and eye blink are extracted from target frames during the training stage. A set ofzation losses are then calculated by a pre-trained multi-attribute discriminator between generated frames and input
o enforce attribute-visual synchronization.

ØVisual Quality Comparison (With Reference Input)

on tasks. Thereforewe extend
the lip-sync
discrim• Wav2Lip
fails
to change
anym Wav2Lip into a more powerful multi-attribute disattributes
sinceonlydata
accepts, which not only
processes audio
anditvisualbut
ss additional features,
head poseany
code andaudio including
input withouttion code. Weadditional
minimize theattributes.
L2 distance between
g features that are generated from the face encoderfrom other encoders.

• PC-AVS can change head
Lsync = Laudio+ Lproperly.
pose + Lemotion
poses
However,(1)itll objective is formulated
as follows:
can not givethese changes= Ll1 + v Lperceptual+ b Lblink(2)
back to+ theoriginalframes.
s Lsync

RESULTSmodels also fail to• 3.Baselinechange emotion and eye blinksets
corresponding
the reference
hours of youtube
videos of Barack to
Obama,
initiallyvideo.
by the Synthesizing
Obama [18]. Besides that, we

Input

Attributes
Reference

Failed to chang
any attributes

Wav2Lip

PC-AVS

Failed to change
eye blinks and emotion

Ours

Figure 4: Visual Quality Comparisons .

13Yaw

Yaw

Pitch

PitchRollRollEye BlinkØQuantitativeComparison• Metrics:OpenCloseHappyAngry§ SSIM (structural similarity indexmeasure)
to evaluate
the video generation quality.
§ SyncNet scores and LMD_m (landmark distance on the mouth) to evaluate mouth movements.
Fig. 4: Examples of Editing different types of attributes and restoring to§ LMD_f (landmark distance on the face) to evaluate how well the facial expression and eye blink
fit the desired results.
Table 2: Quantitative comparison with existing approaches
Table 3: Ablation study
using metrics.
ate the effect of each elPose(P), Emotion(E), anMethods
SSIM LMDm LMDf SyncNetB) is our full model.
Wav2Lip [1]
0.71
0.57
1.39
5.67
Methods
SSIM
PC-AVS [3]
0.68
0.63
3.16
4.39
Ground Truth 1.00
0.00
0.00
5.24
A(Only Audio) 0.70
A+P
0.70
Ours
0.72
0.46
0.86
4.79
A+P+E
0.69
Figure 5: Quantitative Comparison.
A+P+E+B
0.72
pairs of short videos in each dataset. For each pair, a video
will be used as input frames and audio, the other one is used
14for attribute referencing.

ØVideo Example15ØConclusion• Free-View Expressive Talking Head Video Editing
1. A reconstruction-based generator that can generate talking heads fitting to theoriginal frame while corresponding to freely controllable attributes, including
head pose, facial emotion, and eye blink.
2. A multi-attribute discriminator that enforces attribute-visual synchronization.
3. By combining these two modules, our proposed model can edit talking head
videos in full frames on multiple attributes with or without references, which isone of the earliest attempts at this task.
o The main limitation of our proposed model is that it may not be able to generatesatisfactory results when training with multiple identities. We expect a generalmodel that includes identity learning in our future research.
16

Thank You For Listening!

17