PV3D: A 3D Generative Model for Portrait Video Generation
Authors: Zhongcong Xu
Jianfeng Zhang
Jun Hao Liew
Wenqing Zhang
Song Bai
Jiashi Feng
Mike Zheng Shou
Venue: ICLR 2023 poster (2023)

Abstract: Recent advances in generative adversarial networks (GANs) have demonstrated the capabilities of generating stunning photo-realistic portrait images. While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos. In this work, we propose PV3D, the first generative framework that can synthesize multi-view consistent portrait videos. Specifically, our method extends the recent static 3D-aware image GAN to the video domain by generalizing the 3D implicit neural representation to model the spatio-temporal space. To introduce motion dynamics into the generation process, we develop a motion generator by stacking multiple motion layers to generate motion features via modulated convolution. To alleviate motion ambiguities caused by camera/human motions, we propose a simple yet effective camera condition strategy for PV3D, enabling both temporal and multi-view consistent video generation. Moreover, PV3D introduces two discriminators for regularizing the spatial and temporal domains to ensure the plausibility of the generated portrait videos. These elaborated designs enable PV3D to generate 3D-aware motion-plausible portrait videos with high-quality appearance and geometry, significantly outperforming prior works. As a result, PV3D is able to support downstream applications such as static portrait animation and view-consistent motion editing. Code and models are available at https://showlab.github.io/pv3d.

Published as a conference paper at ICLR 2023
PV3D: A 3D GENERATIVE MODEL FOR PORTRAIT VIDEO GENERATION
Eric Zhongcong Xu' Jianfeng Zhang": Jun Hao Liew?, Wenqing Zhang"
Song Bai?, Jiashi Feng", Mike Zheng Shou
Show Lab, National University of Singapore 2 ByteDance (zhongeongxu, zhangj lnnfeng) nuuss..uuuuuuuuuuuuuuuuuuu. jjunhao. 1iew, wengingzhang, jahfeng] weweeengggeyyteceeeeeeoom com (songbai.. seabbai.site,, site, mike. zheng. shou) @eweemmall.ooommm com
nus. edu
ABSTRACT
Recent advances in generative adversarial networks (GANs) have demonstrated the capabilities of generating stunning photo-realistic portrait images. While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully ex- tending GANs for generating 3D-aware portrait videos. In this work, we propose PV3D, the first generative framework that can synthesize multi-view consistent portrait videos. Specifically, our method extends the recent static 3D-aware im- age GAN to the video domain by generalizing the 3D implicit neural represen- tation to model the spatio-temporal space. To introduce motion dynamics into the generation process, we develop motion generator by stacking multiple mo- tion layers to synthesize motion features via modulated convolution. To alleviate motion ambiguities caused by camera/human motions, we propose simple yet effective camera condition strategy for PV3D, enabling both temporal and multi- view consistent video generation. Moreover, PV3D introduces two discriminators for regularizing the spatial and temporal domains to ensure the plausibility of the generated portrait videos. These elaborated designs enable PV3D to generate 3D- aware
portrait videos with high-quality appearance and geome- try, significantly outperforming prior works. As a result, PV3D is able to support downstream applications such static portrait animation and view-consistent mo- tion editing. Code and models are available at https://showlab http:://howlabb.iiuuu.oooo//
INTRODUCTION
Recent progress in generative adversarial networks (GANs) has led human portrait generation to unprecedented success (Karras et al., 2020; 2021; Skorokhodov et al., 2022) and has spawned lot of industrial applications (Tov et al., 2021; Richardson et al., 2021). Generating portrait videos has emerged as the next challenge for deep generative models with wider applications like video manipulation (Abdal et al., 2022) and animation (Siarohin et al., 2019). A long line of work has been proposed to either learn a direct mapping from latent code to portrait video (Vondrick et al., 2016; Saito et al., 2017) or decompose portrait video generation into two stages, i.e., content synthesis and motion generation (Tian et al., 2021; Tulyakov al., 2018; Skorokhodov et al., 2022).
Despite offering plausible results, such methods only produce 2D videos without considering the un- derlying 3D geometry, which is the most desirable attribute with broad applications such portrait reenactment (Doukas et al., 2021), talking face animation (Siarohin et al., 2019). and VR/AR (Cao al., 2022). Current methods typically create 3D portrait videos through classical graphics tech- niques (Wang et al., 2021b; Ma et al., 2021; Grassal et al., 2022), which require multi-camera sys- tems, well-controlled studios, and heavy artist works. In this work, aim to alleviate the effort of creating high-quality 3D-aware portrait videos by learning from 2D monocular videos only, without the need of any 3D or multi-view annotations.
"Equal contribution. work done during an internship at ByteDance Corresponding author

---

Published as a conference paper at ICLR 2023
Figure I: Our PV3D can generate photo-realistic portrait videos with diverse motions and dynamic 3D geometry. We render surfaces extracted by marching cubes. The video frames and shape (normal map) can be rendered from arbitrary viewpoints. Please our project page for video results.
Recent 3D-awar portrait generative methods have witnessed rapid advances (Schwarz et al., 2020; Gu al., 2022; Chan et al., 2021; Niemeyer & Geiger, 2021; Or-El al., 2022: Chan et al., 2022). Through integrating implicit neural representations (INRs) (Sitzmann et al., 2020; Mildenhall et al., 2020) into GANs (Karras al., 2019; 2020), they can produce photo-realistic and multi-view con- sistent results. However, such methods are limited to static portrait generation and can hardly be extended to portrait video generation due to several challenges: 1) it remains unclear how to ef- fectively model 3D dynamic human portrait in generative framework; 2) learning dynamic 3D geometry without 3D supervision is highly under-constrained; 3) entanglement between camera movements and human motions/expressions introduces ambiguities to the training process. To this end, we propose a 3D Portrait Video generation model (PV3D), the first method that can gener- ate high-quality 3D portrait videos with diverse motions while learning purely from monocular 2D) videos. PV3D enables 3D portrait video modeling by extending 3D tri-plane representation (Chan al., 2022) to the spatio-temporal domain. In this paper, we comprehensively analyze various de- sign choices and arrive at a set of novel designs, including decomposing latent codes into appearance and motion components, temporal tri-plane based motion generator, proper camera pose sequence conditioning, and camera-condittoned video discriminators, which can significantly improve the video fidelity and geometry quality for 3D portrait video generation.
As shown in Figure 1, despite being trained from only monocular 2D videos, PV3D can generate a large variety of reoto-realistic portrait videos under arbitrary viewpoints with diverse motions and high-quality 3D geometry. Comprehensive experiments Celeb (Nagrani et al., 2017), CelebV-HQ (Zhu et al., 2022) and TalkingHead- 1KH (Wang et al., 2021a) well demonstrate the superiority of PV3D over previous state-of-the-art methods, both qual- itatively and quantitatively. Notably, it achieves 29.1 FVD on VoxCeleb, improving upon a concur- rent work 3DVidGen (Bahmani et al., 2022) by 55.6%. PV3D can also generate high-quality 3D geometry, achieving the best multi-view identity similarity and warping error across all datasets.
various datasets including Vox-
Our contributions three-fold. To our best knowledge, PV3D is the first method that is capable to generate large variety of 3D-aware portrait videos with high-quality appearance, motions, and geometry. We propose novel temporal tri-plane based video generation framework that can synthesize 3D-aware portrait videos by learning from 2D videos only. 3) We demonstrate state-of- the-art aw-aware portrait video generation on three datasets. Moreover, our PV3D supports several downstream applications, i.e., static image animation, monocular video reconstruction, and multi- view consistent motion editing.
RELATED WORK
2D video generation. Early video generation works (Vondrick et al., 2016; Saito et al., 2017) pro- pose to learn video generator transform random vectors directly to video clips. While recent video generation works adopt a similar paradigm to design the video generator, i.e., disentangle the video content and motion (trajectory), then control them by different random noises. For the video
2

---

Published as a conference paper at ICLR 2023
content, most of the works build their frameworks on top of generative adversarial networks (GAN)
designed for image domain, such as StyleGAN (Karras et al., 2019) and INR-GAN (Skorokhodov et al., 2021). Based image GANs, video GAN works further extend the generation process to temporal domain using various motion generation approaches. MoCoGAN (Tulyakov et al., 2018) and its following work MoCoGAN-HD (Tian et al., 2021) generate the motion code sequence toregressively, which is implemented as random process. StyleGAN-V (Skorokhodov et al., 2022) also generates motion code sequence for each frame, while the motion codes are sampled sepa- rately and thus its generator can synthesize video frames independently. In contrast, DiGAN (Yu al., 2022) only sample one motion code for the entire video and the motion for each frame is generated by the INR network using time instant. This compact video code design is in line with the property of temporal consistency, e.g., talking person only move the lips or twinkles, while the face shape does not change rapidly (Tewari et al., 2019).
3D-aware generation. Image GANs (Karras et al., 2020; 2021) have demonstrated impressive capability in synthesizing high-resolution photo-realistic images. By incorporating implicit neu- ral representations (Mildenhall et al., 2020; Sitzmann et al., 2020) or differentiable neural render- ing (Kato et al., 2018) into GANs, recent works (Schwarz et al., 2020; Niemeyer & Geiger, 2021; Chan et al., 2021; Shi et al., 2021) can produce multi-view consistent images. However, most of them limited image quality and resolution due the heavy computation cost of traditional 3D representations. Follow-up works are proposed to address this issue by either developing an efficient 3D representation (Chan al., 2022; Deng al., 2022: Schwarz al., 2022: Zhao al., 2022) or dividing image generation into two stages (Gu et al., 2022; Or-El et al., 2022; Zhang et al., 2022b), i.e., generating low-resolution images by volume rendering (Max, 1995) and then refine them using super-resolution approaches. However, these methods are limited to static object generation. Re- cently, CoRF (Zhuang al., 2022) proposes to condition the static 3D GAN on estimated motion features. In addition, concurrent work 3DVidGen (Bahmani al., 2022) has been proposed to extend unconditional 3D-aware generation into video domain. Although 3DVidGen can generate plausible results with changing viewpoints, its video fidelity and multi-vier consistency isfactory. Morcover, 3DVidGen does not present or evaluate 3D geometry quality, making its 3D geometry generation ability unclear, which hinders its applicability in the scenarios requiring good 3D geometry (Yuan et al., 2022). contrast, our PV3D aims to synthesize realistic videos with high-quality detailed geometry.
unsat-
PV3D: 3D PORTRAIT VIDEO GENERATION
3.1 OVERVIEW
Problem formulation. Given a monocular 2D portrait video collection D = {va (un) consisting of N video sequences, the goal of 3D-aware portrait video generation is to learn a generator that synthesizes videos given joint conditions of random noise z, camera viewpoint c and timestep t without relying 3D geometry or multi-view supervision.
t, tj
Triplano
Motion
Dsia dlPia
reaiffake
Layer
m
Ziy
T-Tr-le Synthesis
Super Resolution
cond
Triplane
reslfskke
Mapping Network
DBia
W
Motion Generatori
Cj
comd
Figure 2: PV3D creates portrait video frames from appearance code z.,, motion code 2m, timesteps (ty,ty), and camera poses {cr, cy Motion layers and mapping network encode inputs into inter- mediate motion code Wm, and style code Wa. respectively. To incorporate temporal dynamics, the temporal tri-plane synthesis network extends 3D tri-plane (Chan et al., 2022) to spatio-temporal do- main. Two camera-conditioned discriminators regularize the image quality and motion plausibility.
Framework. The overview of our proposed framework shown in Figure 2. PV3D formulates 3D-aware portrait video generation task (2((2)), (), where the generator g(.) first gen-
3

---

Published as a conference paper at ICLR 2023
erates aw-e spatio-temporal representation, followed by volume rendering and super-resolution (denoted as R(-)) to obtain the final video sequence. In this work, our generator ç builds upon tri- plane representation from EG3D (Chan et al., 2022) and extends it to peapioltemporal representation for video synthesis, which we denote as temporal tri-plane. Instead of jointly modeling appearance and motion dynamics within single latent code z, we factorize the 3D video generation into ap- pearance and motion generation components. Specifically, our PV3D takes two independent latent codes, i.e., appearance code Z, NV I) and motion code Zm (O I) as inputs. We condition the generator on Za to synthesize varying video appearance, e.g., genders, skin colors, hair styles, glasses, etc., and use Zm to model motion dynamics, e.g., a person opening his/her mouth.
During training. we randomly sample two timesteps {tz, tt} and their corresponding camera poses (ci, } for one video. Following EG3D, we project the appearance code 2n and camera pose e into intermediate appearance code Wa for content synthesis. As for the motion component, we develop motion layer to encode motion code Zm and timesteps (1,, [ into intermediate motion code Wm- Our temporal tri-plane synthesis network generates tri-plane features based on wa and Wm. With the generated tri-plane at (4.1t)) volume rendering (Max, 1995) is applied to synthesize frames with camera pose c; and Cy. respectively. The rendered frames are then upsampled and refined by super-resolution module. To ensure the fidelity and plausibility of the generated frame content and motion, we develop two discriminators Ding and Dvia supervise the training of G. Both Ding and Deia are camera-conditionel which can leverage 3D priors.
3.2 3D-AWARE VIDEO GENERATOR
Challenges. Existing 3D-aware image GANs can only model static scenes and it remains unclear how to model motion dynamics, such as topological changes of a scene (Park et al., 2022). A straightforward approach to directly combine the motion condition with latent code, camera pose, and timestep, and feed them to the encoder to generate 3D representations for video rendering. However, such naive design cannot generate temporally consistent and motion-plausible portrait videos because the generation of video content and motion is highly entangled. Besides, learning 3D-aware portrait video appearance and geometry from monocular 2D videos only is highly under- constrained, making the model training difficult and generation quality poor. Another challenge is that motions caused by camera pose changes (e.g.. camera movements) and head pose changes (e.g., look up. turn left) are highly entangled. This introduces ambiguities to the training process, largely increasing model's 's learning difficulties.
Prior works typically incorporate motion features by either manipulating the latent code for pre- trained image generator (Tian et al., 2021) or simply conditioning 3D representation on latent code and timestep (Bahmani al., 2022). Nevertheless, such designs cannot guarantee temporal consis- lency and motion diversity. To address these challenges, we decouple the latent code into appearance and motion components and propose motion generator to model temporal dynamics. Such design not only preserves the high-fidelity and multi-vien consistency of each frame but also enables the synthesis of videos with temporal coherence and motion diversity. Secondly, we propose camera conditioning strategy to alleviate the motion ambiguity issue and thus facilitates convergence.
Synthesis of motion dynamics. To generate motion at each timestep in a video, we condition the generator on latent codes 2, and Zm, timesteps (f,, tj} and camera poses {c, cz}- Without loss of generality, we only describe the generation process for timestep 6; and camera pose c₁ As shown in Figure 3, we introduce K motion layers into the synthesis layers of motion generator. Each motion layer encodes motion code Zm and timestep t; into intermediate motion code w)' In particular, the motion code is first multiplied with timestep to encode temporal information, followed by lightweight mapping head Hmm with leaky ReLU activation. A multi-layer perceptron (MIP) then encodes them into wi.k In other words, the k-th motion layer computes
(1) where k E ..., KJ. E {0, 1, N} denotes the frame index while N represents the total number of frames in one video. These intermediate motion codes wick then passed to the tem- poral tri-plane synthesis network to modulate the static appearance features via adaptive instance normalization (AdaIN) (Karras al., 2019) to incorporate temporal dynamics.
w i.k = MLP&&HH, (zm * t4)).
In practice, we employ an equivalent operator, i.e., modulated convolution (Karras et al., 2020), to compute motion features. We then fuse it with the appearance features controlled by w The
4

---

Published as a conference paper at ICLR 2023
fused features are passed to the next synthesis layer iteratively to generate tri tri-planes. This process formulated as
f. ( + ModConv(f", w('",, w²),, where f* 5% (fx-1 w().
(2)
Here, Sg and S denote the first and second synthesis block (ModConv) in k-th synthesis layer, while fü denotes the feature map synthesized by the k-th layer.
Motion diversity and temporal consistency. Recent works (Tov al., 2021; Shen et al., 2020; Richardson et al., 2021) ) investigate the semantic meaning of intermediate latent code space (W+) in pre-trained StyleGAN model, and discover that one can perform diverse manipulation of im- age content by using different style codes at different style-modulation layers. We made similar
observation for PV3D (Appendix A.2). Specifically, mixing latent codes in shallow layers only brings coarse level changes on facial attributes, such as expressions, hair, etc, which reduces the diversity. On the other hand, mixing style code in deeper layers leads to dras- tic changes in image content. As result, identities will no longer be preserved, causing severe temporal incon- sistency.
Moton Layer"
LReLU
2m
MLP
w A.A
G2
ModConv
ModComn
MonComy
To preserve the identity in generated videos, we care- fully select K synthesis layers, such that k < K, for incorporating motion features. Choosing a suitable K increases temporal consistency and improves our motion generator's 's capacity for modeling diverse motions. Be- sides, this synthesis layer selection also alleviates the overfitting of RGB video frames, which improves the quality of 3D geometry.
k
Figure 3: Architecture of the k-th syn- thesis layer in motion generator. Motion layer encodes zn and t; into intermedi- ate motion code wi.k Motion features are computed by modulating appearance features which are conditioned on w
Alleviating motion ambiguities by camera sequence condition. Learning 3D-aware portrait video generation from 2D videos faces another challenge, ie., the entangled face motions and camera movements bring motion ambiguities. The concurrent work 3DVidGen (Bahmani et al., 2022) sim- ply shares one camera pose for the entire video. However, this approach ignores motion ambiguity that harms video fidelity and geometry quality. Differently, estimated camera pose sequences (Appendix A.3). It has two advantages. 1) Suppose that we are only interested in the faces, a camera rotating around static face equivalent rotating the face in front of a static camera. Generating head rotation directly is challenging due to the large topological changes of the scene. By conditioning our generator on c; at each time instant t., we can model the head rotation by rotating camera for observation instead of deforming the 3D scene, making our model easy to optimize. 2) Given the camera pose sequence, our generator can encode view- dependent features for each frame to leverage 3D priors. Such a simple design effectively improves the multi-view consistency and facilitates the learning of dynamics in 3D geometry as verified in our experiments.
condition the generator
pre-
3.3 CONDITIONAL DISCRIMINATORS
Due to the lack of regularization in video generation procedure, the generated videos may present implausible contents or unreasonable motions. To ensure the plausibility of portrait video genera- tion, we introduce discriminator module to guide the generation process. In particular, the module consists of an image discriminator Ding for evaluating video appearance quality and a video dis- criminator Dvia for ensuring video motion plausibility.
Our image discriminator Diny follows EG3D's discriminator architecture and uses camera poses conditions to guide the generator to learn correct 3D priors and thus it can produce multi-view consistent portraits. We apply Ding on each generated frame 1; (at timestep t,) independently, which can be formulated as Ping Ding [1, cz), where Ping denotes the real/fake probability.
Our PV3D generates two images {I, jointly at two random timesteps (t,, t,} for each video during training, thus design conditonen dual-frame video discriminator D to facili- tate motion--pausiin portrait video generation. Specifically, we first concatenate the generated two frames (T,, 15) -hienel-wisell to obtain image pair. To help encode temporal information,
5

---

Published as a conference paper at ICLR 2023
further concatenate the timestep difference At =t, -t with the image pair. Our video discrim- inator learns to differentiate the real and generated image pairs based on motion features extracted from this hybrid input. To alleviate motion ambiguity and model view-dependent effects, we further condition Vii on the corresponding camera poses {ci, (y. )- Our video discriminator is formulated as Pvid Da (7t, J At). [cs; 9y)), where Peid indicates the probability of each image pair being sam- pled from real data distribution. Although Dvia only takes two frames as inputs, it can effectively learn ordinal information (Appendix A.5) and help produce motion-plausible results as verified in our experiments (Section 4.3). Moreover, such simple design largely improves training efficiency and stability compared with previous methods that take long sequences as conditions (Tulyakov et al., 2018; Tian et al., 2021).
3.4 TRAINING AND INFERENCE
Training. To synthesize an image E Rf based on tri tri-plane T, we shoot rays r(s) = o sd out from the camera origin along direction d at each pixel (Mildenhall et al., 2020). In prac- tice, we sample query points Er along each ray and get the features for each point by interpolat- ing them in T. The features are passed to the decoder to predict color and density such that (r(),,,, c(r(s))] Decoder (onooder(Innerrn (Interp(ær (2r, T)), where Decoder is an MLP with softplus activation, Interp denotes interpolation. The pixel value calculated by volume rendering as:
51 I p(s)o(r(s))c(((s)d) where p(() =exp exp (-f a(r(s))ds)
T(r)=
We compute the non-saturating GAN loss (Goodfellow et al., 2020) Ling and Lvia and RI regular- ization loss (Mescheder et al., 2018) LRI- Following EG3D, we use dual image discriminator which takes both the low-resolution density regularization T., on the generated video frames. The overall loss is formulated as:
image and high-resolution image inputs. We also compute
C C. udv
L + Lo L = Ping + L + PRI-
(4)
ing
Inference. Although trained on sparse frames only, our generator can synthesize contiguous frames during inference. For each video, we generate frames at timestep 4, where {0, N.. JV. N denotes the maximum number of frames. The mapping network in our generator takes camera pose C for each frame to generate intermediate appearance code w However, during training, each frame has its camera pose. This brings discrepancy in intermediate appearance codes within the video, which harms the temporal consistency. We propose share the same e; for the mapping network in inference, which largely improves temporal consistency as demonstrated in experiments.
4 EXPERIMENTS
We study the following questions in our experiments. 1) Can PV3D generate high-quality portrait videos with dynamic 3D geometry and multi-view consistency? 2) How does each component in our PV3D model take effect? 3) What are the performance of PV3D in downstream applications? To answer these, we conduct extensive experiments on several human portrait video datasets.
4.1 DATASETS
We experiment on three face video datasets, i.e. VoxCeleb (Nagrani et al., 2017: Chung et al., 2018), V-H (Zhu et al., 2022), and TalkingHead--IKF (Wang et al., 2021a). These datasets contain talking face clips of different identities extracted from online videos. We balance the video clips for each identity and preprocess the videos using standard pipeline (Appendix A.3).
4.2 COMPARISONS
Baselines. We compare PV3D against four sfate-of-the-ee methods for 3D-aware video generation: (1) the concurrent work 3DVidGen (Bahmani et al., 2022); (2) StyleNeRF+MCG-HHI combining the SOTA 3D image GAN model StyleNeRF (Gu et al., 2022) with the SOTA multi-stage video gen- eration work MoCoGAN-HD (Tian et al., 2021); (3) EG3D+MCG-HD- combining EG3D (Chan

---

Published as a conference paper at ICLR 2023
StyleNeRF MCG-HD
EG3D + MCG-HD
3DVidGien
3DVidGien (EG3D)
PV3D (Ours)
Figure 4:
Qualitative comparisons between PV3D and baselines, our project page for videos.
et al., 2022) with MoCoGAN-HD: (4) 3DVidGen (EG3D): replacing the 3D image GAN backbone in 3DVidGen with EG3D.
Evaluation metrics. We evaluate PV3D and baseline models by Frechet Video Distance (FVD) (Unterthiner et al., 2018), Multi-view Identity Consistency (ID) (Shi et al., 2021), Chamfer Distance (CD), and Multi-view Image Warping Errors (WE) (Zhang al., 2022a;b). The evaluation metrics for multi-view consistency are originally proposed for 3D image generation. We extend these metrics multiple frames which suitable for 3D video generation tasks. Please refer to Appendix A.4 for more details on our evaluation metrics.
Quantitative evaluations. Table I summarizes the quantitative comparisons between PV3D and baseline models. First, we observe that PV3D outperforms all of the baseline models on all datasets W.E. FVD, ID, and WE, which shows our PV3D can generate videos with diverse content and plausible motions while maintaining high multi-view consistency within videos. It is worth not- ing that our PV3D has higher CD than StyleNeRF-MGG-HDD This because CD computes the distance between two point clouds rendered from frontal and side views. As illustrated in Fig- ure 4, StyleNeRF+MCG-HD fails synthesize detailed 3D geometry, thus leading to lower CI) than PV3D. A similar observation is made in TalkingHeade H where MCG-HD+EG3D baseline fails to generate videos with diverse motion, which also leads to smaller CD. On the other hand, PV3D is able to generate high-quality dynamic 3D geometry with better multi view consistency.
Qualitative evaluations. Figure 4 shows the qualitative comparisons. We first observe that both StyleNeRF+MCG- HD and EG3D+MCG-HD produce poor results because their multi-stage frame- work design is not end-to-end trainable, resulting in implausible motions. Compared with them, 3DVidGen and its EG3D counterpart achieve relatively better video fidelity. However, the geometry quality and motion diversity of 3DVidGen are still not comparable with PV3D due to their straight- forward motion condition design. Differently, our PV3D produces temporally consistent and motion plausible videos with high-quality geometry.
Table I: Quantitative comparisons, with best results bold and second best underlined.
VoxCeleb
CelebV-HQ IDI
TalkingHead -KH ID1
FVD.
IDI
CDJ
WE.
FVD1
CDL
WEL
FVD1
CDJ
WEJ
SLyGeH
348.7
0.70
1.08
36.06
134.4
0.80
1.13
38.73
292.7
0.75
5.34
49.29
EG3D+MCG-HD
222.1
0.80
1.57
10.57
298.4
0.77
3,34
10.74
262.4
0.78
1.39
11.54
3DVidGen
65.5
0.75
3.40
44.55
63.6
0.77
3.80
37.30
83.0
0.76
4.35
46.47
3DVidGen (EG3D)
56.3
0.71
3.65
24.55
66.2
0.70
3.83
26.34
89.8
0.65
4.56
35.48
PV3D (Ours)
29.1
0.81
1.34
9.76
39.3
0.81
1.21
8.18
66.6
0.80
2.33
10.73
4.3
ABLATION STUDIES
In this section, conduct ablation studies of PV3D VoxCeleb dataset as it contains more diverse motions and appearances as well as more balanced identity distributions.
Motion layer position. PV3D computes and fuses motion features in selected synthesis layers, i.e., the first K layers in our motion generator with a default setting of K 4 (denoted as middle). To study the effect of motion layer position, we insert motion layers until early (K 2) and late (K

---

Published as a conference paper at ICLR 2023
Table 2: Ablations of PV3D on VoxCeleb. We vary the motion layers, motion generator architecture, camera conditions, and discriminator architectures study their effects.
Pos. FVD] CDL WE
Mot.
FVDJ
CDJ WE
Cam. FVDJCDJ
WEI
Vid. Dis. FVD] CD4 WEJ
carly 40.2 2.32 10.60
MLP
41.1
1.33 9.86
All 32.1 2.49 11.32
w/o Cam 34,9 4.38 10.84
middle 29.1 1.34 9.76
Naive
36.9
1.05 10.21
Non 55.3
3.28 23.71
w/o At
38.7
1.54 10.04 29.1
late
33.3 6.24 11.56
Zm >Za
37.7
1.07 9.86
Map 38.0
1.47 10.19
w/ both
1.34 9.76
learned
30.5 1.81 10.31
Ours
29.1
1.34 9.76
MapT 29.1 1.34 9.76
(d) The effect of different conditions in video dis- criminator.
(a) The effect of motion layer positions.
(b) The effect of motion
(c) The effect of camera
generator architectures.
condition in generator.
7) stages. Moreover, design simple learnable parameter for each motion layer and optimize this weight for motion feature fusion (denoted as learned). The results are summarized in Table 2a.
We observe that when the motion layers too shallow, the model's capacity for modeling dynamics is limited and both video quality and multi-view consistency degrade. However, if we insert motion features into all the synthesis layers (late), FVD is still higher and 3D geometry is largely affected because (1) appearance feature manipulation space is large but hard to optimize, which harms motion plausibility: (2) our training process is highly under-constrained. In this case, the model can easily overfit the RGB frames, which hinders the learning of geometry. Similarly, although the video quality improves when using learnable fusion weights, the multi-view consistency is not as good as reflected by CD and WE metrics because the model still tends to produce better RGB content at the cost of worse geometry due to the lack of 3D supervision.
Motion generator. Table 2b summarizes the ablations on our motion generator architectures. Our video generator takes two random codes, i.e., Zo and Zm. We first study the effect of introducing an independent motion code by replacing motion code Z.. with Za. It can be observed that although CD reduces by 0.3, all other metrics deteriorate especially FVD, suggesting that motion plausibility is affected when motion code is removed. A similar observation is made in naive implementation. Replacing modulated convolution with simple MLP significantly increases FVD, implying MLP is less effective than modulated convolution in manipulating appearance features.
Camera conditions. We investigate how different camera conditioning strategies could affect gen- eration performance. In each training iteration, we sample two video frames along with their camera poses. Thus, we have three options for camera conditioning (sharing camera pose or not) in our gen- erator: (1) All: condition the whole generator (both mapping network and rendering) on the shared camera pose; (2) Non: condition the whole generator on the camera pose of each frame; (3) Map: only share camera pose for mapping network, and use different camera pose render frames.
As shown in Table 2c, sharing one camera has poor multi-view consistency, with 2.49 CD and 11.32 WE. On the other hand, using non-shared cameras leads to even worse performance. This is because our mapping network takes camera pose when computing appearance code. Therefore, changing the camera across video frames would bring rapid changes to the appearance code, leading to tem- poral inconsistency. An alternative is to only share camera poses in the mapping network. However, this strategy works poorly because the camera pose discrepancy in generation process hinders the convergence. On the contrary, sharing the same camera pose in the mapping network only during inference stage (Map')) facilitates the training and preserves temporal consistency. Hence, we use MapT as our default setting.
Discriminator conditions. Table 2d shows the effect of camera pose and time difference condi- tions in the video discriminator. We can observe that camera conditions in video discriminator can largely improve 3D geometry, while time difference helps improve temporal coherence by encoding auxiliary temporal information.
4.4 APPLICATIONS
Static portrait animation. Our generator can independently generate video frame at a certain timestep instead of generating from the first frame auto-regressively. This flexible architecture en- ables static portrait animation. Given the input image and the estimated camera pose, we fix our gen- erator and optimize latent code at timestep t 0. The inversion is performed in W* space (Richard- son et al., 2021; Abdal et al., 2019). As shown in Figure 5, the GAN inversion based on our generator also produces high-quality 3D shape for the input frame. We then keep the latent code fixed and ran-
8

---

Published as a conference paper at ICLR 2023
domly sample motion code drive the portrait with natural motion. With the 3D priors learned by our PV3D, the synthesized videos can also be rendered under arbitrary viewpoints.
inversion
animatiio
mult view
Figure 5: Given static portraits, we fix the video generator and optimize the intermediate appearance code in W space to perform inversion. By sampling random motion code, we can animate the static portraits with natural motion and synthesize portrait videos with multi-view consistency.
Monocular video reconstruction and motion editing. Given a video and its pre-estimated camera pose sequence, we can directly reconstruct the video based on our pretrained generator. For the video content, also optimize the intermediate appearance code in W space. As for the motion component, we experimentally find that z space is more effective. Specifically, we inverse Zm for each video frame individually. Figure illustrates the results for reconstruction, our PV3D provides simple solution for 3D reconstruction monocular videos. Thanks to the disentangled design of motion and appearance components PV3D, we can fix appearance codes and sample new motion codes to manipulate the motion of input videos in the 3D domain.
peonetry
goometry
Rond at
Figure 6: PV3D provides quick solution for 3D reconstruction on monocular videos. Thanks to the disentanglement of appearance and motion in PV3D, the motion of input vidcos can be manipulated by changing motion codes. The results can still maintain multi- view consistency.
5 CONCLUSION
This work introduces the first 3D-aware generative model, PV3D, for synthesizing multi-view con- sistent portrait videos with high-quality 3D geometry. By employing independent latent codes for appearance and motion, PV3D can leverage temporal tri-plane synthesis address the challenges in 3D-aware portrait video generation. Moreover, condition PV3D camera pose sequence to al- leviate the challenging motion ambiguities. demonstrate that PV3D can generate both temporal and multi-view consistent portrait videos with diverse motions and dynamic 3D geometry. Besides, PV3D supports downstream applications such as static portrait animation, 3D video reconstruction, and multi-view consistent motion editing. We believe method will facilitate the desired practical applications in VR/AR and visual effects.
ACGNONTESGEMENTS
This project is supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF13 1D-FF13-202100000 and the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2022).

---

Published as a conference paper at ICLR 2023
7 ETHNICS STATEMENT
Static portrait animation, monocular video reconstruction, and editing could be misused for generat- ing fake videos manipulating authentic videos for improper illegal purposes. These potentially harmful applications may pose a societal threat. We strictly forbid these kinds of abuses. In addi- tion, our video generator may contain bias for face results due to the unbalanced video distribution in training datasets.
8 REPRODUCIBILITY STATEMENT
To ensure reproducibility, we describe the implementation details in Appendix. A.I. The dataset preprocessing steps introduced thoroughly in Appendix. A.3, including video sources, training data balance, face video alignment pipeline, and camera pose estimation approach. Our code and models are publicly available at https///howlabbggiubb.oooo
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019.
Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Video2stylegan: Disentangling local and global variations video. arXiv preprint arXiv:2205.13996 2022.
Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang. Gordon Wetzstein, Leonidas Guibas, Luc Van Gool, and Radu Timofte. 3d-aware video generation. arXiv preprint arXiv:2206. 14797, 2022.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). In ICCV, 2017.
Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shun-Suke Saito, Stephen Lombardi, Shih-En Wei, Danielle Belko, Shoou-I Yu, et al. Authentic volumetric avatars from phone scan. ACM Trans. Graphics, 2022.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
Eric Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. INTERSPEECH, 2018.
William HE Day and Herbert Edelsbrunner. Efficient algorithms for agglomerative hicrarchical clustering methods. Journal of classification, 1984.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019a.
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with sepelved learning: From single image to image set. In CVPRw, 2019b.
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong, Gram: Generative radiance manifolds for 3d-aware image generation. CVPR, 2022.
Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In ICCV, 2021.
10

---

Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, and Published as a conference paper at ICLR 2023
Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt. Stylevideogan: A temporal generative model using a pretrained stylegan. arXiv preprint arXiv:2107.07224, 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020.
Philip- Millia- Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. In CVPR, 2022.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d aware generator for high-resolution image synthesis. In ICLR, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In CVPR, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NeurIPS, 2021.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, 2018.
Yaser Sheikh. Pixel codec avatars. CVPR, 2021.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1995.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, 2018
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identifi- cation dataset. INTERSPEECH, 2017.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. CVPR, 2021.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. CVPR, 2022.
Keunhong Park. Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo
and Steven M Seitz. Hypereerf: A higher-dimenssonall representation for topologically varying neural radiance fields. SIGGRAPH, 2022.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pp. 296-301. Icce, 2009.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding style: stylegan encoder for mmage-to-image translation. In CVPR, 2021.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. In ICCV, 2017.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. NeurIPS, 2020.
Katja Schwarz, Axel Sauer. Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d- aware image synthesis with sparse voxel grids. NeurIPS, 2022.
11

---

Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for Published as a conference paper at ICLR 2023
semantic face editing. In CVPR, 2020.
Yichun Shi, Divyansh Aggarwal, and Anil K Jain. Lifting 2d stylegan for 3d-aware face generation. In CVPR, 2021.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. NeurlPS, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. NeurIPS, 2020.
Ivan Skorokhodov. Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In CVPR, 2021.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny, Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. CVPR, 2022.
Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Sei- del, Patrick Pérez. Michael Zollhofer, and Christian Theobalt. Fml: Face model learning from videos. In CVPR, 2019.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In ICLR, 2021.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing encoder for stylegan image manipulation. ACM Trans. on Graphics, 2021.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, 2018.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:18/2.011177 2018.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. NeurlPS, 2016
Ting-Chun -hgu Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-heas head synthesis for video conferencing. In CVPR, 2021a.
Ziyan Wang, Timur Bagautdinov, Stephen Lombardi. Tomas Simon, Jason Saragih, Jessica Hodgins, and Michael Zollhofer. Learning compositional radiance fields of dynamic human heads. In CVPR, 2021b.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In ICLR, 2022.
Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In CVPR, 2022.
Jianfeng Zhang, Zihang Jiang, Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: a 3d generative model for animatable human avatars. arXiv preprin! arXiv:221 14/589, 2022a.
Xuanmeng Zhang, Zhedong Zheng, Daiheng Gao, Bang Zhang, Pan Pan, and Yi Yang. Multi-view consistent generative adversarial networks for 3d-aware image synthesis. CVPR, 2022b.
Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G Schwing, and Alex Colburn. Generative multiplane images: Making 2d gan 3d-aware. In ECCV, pp. 18-35. Springer, 2022.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang. Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022.
Peiye Zhuang, Liqian Ma. Oluwasammi Koyejo, and Alexander G Schwing. Controllable radiance fields for dynamic face synthesis. 3DV, 2022.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.I IMPLEMENTATION DETAILS
For each video, we sample two frames within 16-frame span. Following DiGAN (Yu et al., 2022), we sample timesteps {t,,t, t,) } from beta distributions. The resolution the generated video is 512x512. We use resolution of 64 and sampling step of 48 for neural rendering during training. In inference stage, we use a rendering resolution of 128 for geometry visualization only. Each cam- pose has 25 dimensions, with 16 for extrinsics and 9 for intrinsics. Our model is implemented using PyTorch. We balance the loss terms by weighting factors: 1) =0.6-.6, x Aimg=1.0, =R, for VoxCeleb; 00.55 xv6d=0.65, Aimg=1.0, AR, =4.0 for CelebV-HQ: 3) 00c5 Avid=0.65, Aim0=1.0, AR, =2.0 for TalkingHead- KH. Our model is trained for 300k iterations with batch size 16, which takes 58 hours on 8 Nvidia A100 GPUs.
A.2 ANALYSIS OF LATENT CODE SPACE
The architecture of synthesis layer in our PV3D largely follows StyleGAN and its following works (Karras et al.. 2019; 2020; 2021). Based on the pre-trained StyleGAN models, prior works (Shen et al., 2020) also investigate the property of the latent code space. These works show that the intermediate latent code space has extensive manipulation ability for image synthesis. Al- though the original design of StyleGAN is sharing one intermediate latent code across all synthesis layers, follow-up works (Abdal al., 2019; Richardson et al., 2021; Tov et al., 2021) relax this con- straint and achieve better reconstruction results for image inversion because using different latent codes different synthesis layers can further expand the space for image generation.
=
K=4
K 6
K
Zo
21
Figure 7: Results of style mixing in first K synthesis layers in pre-trained EG3D. For each example, first sample one latent code and mix it with another latent code in the first K synthesis layers.
Our PV3D can generate temporal tri-plane features by modulating appearance features based on motion code and timestep. We encode motion code and timestep into intermediate motion codes and then compute motion features temporal tri-plane synthesis network. Because our synthesis network is built on top of EG3D, we also analyze the latent space to find out how the manipula- tion of appearance features could affect the synthesis results. shown in Figure 7, we perform style mixing in the first K synthesis layers of pre-trained EG3D model. When K increases, the image contents gradually change. Specifically, manipulating the appearance code in K = 2 layers can largely preserve the contents. However, only modulating features in the first 2 layers would potentially harm the capacity for content diversity our video generator. When K 2 6, there exists sharp change in the image content. Because important property for portrait video is the tem- poral coherence, i.e., consistent identity, we finally select K = in our motion generator to maintain good temporal consistency well as motion diversity.
13

---

Published as a conference paper at ICLR 2023
A.3 DATASET PREPROCESSING
Video sources.
VoxCeleb (Nagrani al., 2017; Chung al., 2018) is audio-visual speaker verification dataset containing interview videos for more than 7,000 speakers. It provides speaker labels for each video clip. For each speaker, we sample two video clips that have the highest video resolutions.
Ce (Zhu et al., 2022) is a large-scale face video dataset that provides high-quality video clips involving 15,653 identities. Compared with VoxCeleb, contains diverse lighting conditions.
TalkingHead-IKK (Wang et al., 2021a) consists of talking head videos extracted from 2,900 long video conferences.
For all of the datasets, we directly download videos with the highest possible resolution from YouTube using the provided uid list.
Training data balance. VeI and TalkingHead-IIK have unbalanced number of video clips for each identity. To balance the video clips, we perform face clustering predict pseudo-identities and sample video clips for each identity. Specifically, we extract facial features using pre-trained Ar- cFace (Deng et al., 2019a) and employ agglomerative hierarchical clustering (Day & Edelsbrunner, 1984) to predict pseudo-ident -ndendiieeeeiiiss and sample at most two videos for each identity.
Alignment. All datasets are preprocessed using the same alignment pipeline: We first detect land- marks for each frame using an off-the-shell face alignment package (Bulat & Tzimiropoulos, 2017). Such single frame face alignment technique would introduce temporal inconsistency. Following (Fox et al., 2021), we use low-pass Gaussian filter to smooth the estimated keypoints before warp- ing the images. Then, we follow the image warping approach of FFHQ (Karras et al., 2019) to align each frame. To facilitate the training of unconditional 3D image GAN, EG3D employs cropping step to process the image. Specifically, they realign the image in depth direction, which forces all of the keypoint of nose the same point in the world coordinate defined by the parametric face model, i.e., 3DMM (Paysan et al., 2009). We follow this step to process the video clips. Fi- nally, we apply deep face reconstruction (Deng al., 2019b) to estimate camera pose for each video frame. Again, this process brings temporal inconsistency and we also apply the low-pass Gaussian filter smooth the results.
extra
Verification. Our preprocessing pipeline is automated and purely based on off-the-sheef packages. However, there exist noise and failure cases in each preprocessing step. Therefore, we apply an extra verification step to remove the noisy video clips. In particular, we use ArcFace to extract the facial features again for every 2 frames within a video clip. If the similarity scores between one frame and others are below a threshold r = 0.5, this frame will be labeled as noisy. We discard video clips that contain more than two noisy frames.
A.4 EVALUATION METRICS
Frechet Video Distance (FVD). We compute the statistics for ground-truth and generated samples using the pre-trained I3D (Carreira & Zisserman, 2017) model (the Py PyTorch version checkpoint released by VideoGPT'). For the training dataset, randomly sample 5000 videos. Each video generator synthesizes 5000 uncurated videos randomly for test. All of the videos are encoded in H264 format. Compared with previous methods that save videos in PNG format (Skorokhodov et al., 2022), we empirically find that saving videos with H264 and decoding videos into images during testing does not cause any variation and can largely save space.
Multi-view Identity Consistency (ID). Dw-aware image GAN works compute ID by rendering both frontal view and side view images to measure the model's multi-view identity consistency. They adopt ofate-of-the-art face recognition model (Deng et al., 2019a) (denoted as F) to extract features and compute similarity scores between frontal and side view images. We simply extend this process to multiple frames. In our experiments, we generate random videos and render both frontal face and side face images Iy. at two randomly sampled timesteps (la, 4.. where y denotes yaw angles. The ID metric is formulated as:
ID ID ( = F F(1))).
(5)
Vineoi
14

---

Published as a conference paper at ICLR 2023
where to # compute ID for each frame pair within each video. The final ID metric is the average value for all of the image pairs.
Yo # Yr. In this work, report results with y {0a 330) 1000 videos. We
Chamfer Distance (CD). StyleSDF (Or-El et al., 2022) proposes to use the Chamfer Distance be- tween the frontal and side view point clouds to measure the multi-view consistency of 3D geometry- In this work, we extend this metric to multiple frames by computing CD between depth map pairs in each video. For each video, we sample two timesteps {?o. 4 and render point cloud two angles Py. where y denotes yaw angles. The CD is mathematically formulated as:
CD plo Ptt
= med min || - v1l2 + med min | - rEP yErs?
(6)
yers) zert
where med means median, to # L or Yo # Y1. Following StyleSDF we normalize the point clouds based the volume sampling bin size for each generator before computing CD. We also remove the non-terminating rays whose opacity is below 0.5 for all of the models whose backbone EG3D, To make fair comparisons, for baselines based StyleNeRF, NeRF, i.e., rendering face part only. We interpolate all of the points clouds to 64x64 64. We also use {0", 2300) } and render 1000 videos. We then average CD for all of the point cloud pairs get the final CD result.
render point clouds in foreground
Multi-view Image Warping Errors (WE). Inspired by recent 3D image GAN works (Zhang al., 2022h;a), we can further compute the warping error by reprojecting each pixel from side view to the frontal view based on the images (4%g 1 and depth maps (D%g yo² D ) at yaw angle {Yo, Y1 and determined by yaw angle We warp a pixel located at (i,j) (denoted as x) in side view image to frontal view by re-projection. It is formulated as:
timesteps 4. The camera extrinsics and intrinsics are [R]t] and K, where
K [|]| D(x))²
Based on the re- re-projected coordinates, we compute the warping error as:
WE(T To? ( L
Σ 1
Twarpl:
(8)
where (x') Ly, (x) and Yo * y,, L is the number of re-projected points that located within the field-of-view in the frontal view image, and we only compute WE based on these visible pixels. Both image and point cloud are resized to 256x256. Finally, we use y E 10p, 30") } and generate 1000 videos. We compute the warping error for each front and side view pair in one video. The WE reported in this work is the average value of all the pairs.
A.5 ADDITIONAL EXPERIMENT RESULTS
Ablations. We perform additional ablation study on PV3D by varying the design of discriminator components and volume rendering resolutions.
Table 3: Ablations on discriminators. resolution for neural rendering, and generator camera pose conditions.
Disc.
FVDJ
CDL
WEL
Rend. Res.
FVDI
CDL
WEL
Gen. Cam. FVDJ CDJ
WEJ
w/o Vid
145.9
3.89
11.82
32
42.4
3.25
12.98
w/o
44.3
2.35 13.01 1.34 9.76
w/o Img
42.2
3.44
14.40
64
29.1
1.34
9.76
w/ cam
29.1
w/ both disc.
29.1
1.34
9.76
128
34.1
0.92
9.62
(c) The effect of camera condition- ing in generator.
(a) The effect of discriminator com- (b) The effect volume rendering ponents.
resolution.
Discriminator components. PV3D employs two independent discriminators to regularize the video content and motions. We ablate the image and video discriminators to study how each one su- pervises the training of generator. Table 3a illustrates the results of removing each discriminator. Without video discriminator, the video quality largely deteriorates, suggesting that video discrimi- nator is able to guarantee motion plausibility. In addition, removing image discriminator also brings
15

---

Published as a conference paper at ICLR 2023
500
number of camera pairs
400
number of videos
150-
250
200
230
150
100-
100
0.2
cosine similarity
score difference
(a) Statistics for cosine similaritics between the cm- (b) Statistics for real/fake score differences between beddings of original and flipped camera pose pairs. original and flipped camera pose pairs.
Figure 8: The analysis for the alignment ability of our video discriminator. We sample 3000 camera pose sequences from training dataset and input both original camera pose sequences together with the reversed ones into our video discriminator to study its ability for encoding ordinal information.
significant performance drop in FVD, CD, and WE, which proves that our video discriminator can also supervise the frame content but struggle to guarantee 3D geometry.
Volume rendering resolution. Table 3b summarizes the results of using different resolutions for neural volume rendering during training. A small resolution (32) restricts model's capacity, leading to worse performance on all of the metrics. Although a higher resolution (128) outperforms our default setting (64) terms multi-view consistency and warping error, video quality still worse. The results show that there exists a tradeoff between the video quality and 3D geometry in our generator because is trained only on 2D videos. Considering the computation resources, default setting uses resolution of 64. In this work, also use 64 for testing and reporting evaluation metries. Moreover, we only use a resolution of 128 for geometry visualization.
Generator camera conditioning. PV3D takes camera poses encode 3D priors in generator. Ta- ble 3c shows that the 3D priors improve performance with a large margin. Without the camera pose condition in generator, a large decrease can be observed in all of the evaluation metrics.
Video discriminator alignment. To investigate whether our video discriminator can align the cam- poses with the input video frames, conduct two following experiments.
Effects of camera pose order on embedding. We sample 3000 camera pairs from the training dataset and pass them our video discriminator. We then exchange the order of the camera poses and compute the embeddings again. After that, we compute the cosine similarities between the embed- dings of original and the flipped camera pose sequences. The statistics for similarities are shown in Figure Sa. It can be observed that most of the camera pose sequences with reversed orders will be mapped into embeddings that are far apart. Thus, our video discriminator is sensitive the camera pose orders.
Effects of camera pose order on discrimination ability. We further input the videos generated by the sampled camera poses together with both original and flipped camera pose sequences into the video discriminator. We compute the LI distances between the real/fake scores and visualize statisties in Figure 8b. It illustrates that our video discriminator predicts different real/fake scores for most of the video frame pairs the camera pose sequences are in reverse order. The results prove that our video discriminator arder-aware and can align the camera pose sequence with the video frame pair.
Longer-terus video synthesis. PV3D trained on two video frames per clip, this architecture makes it suitable for long-term video synthesis. To study its ability for long-term video generation, we further train and test PV3D on video clips of 48 frames. The results are shown on our project see, training data with longer duration contains more diverse motions and PV3D
page. As can learn to generate such motions accordingly.
Qualitative results. We demonstrate results of PV3D in Figure 9.
16

---

Published as a conference paper at ICLR 2023
Figure 9: Samples generated by PV3D, see our project page for video results.
A.6 LIMITATIONS AND FUTURE WORK
Our PV3D has several limitations: 1) PV3D is trained and tested on vidco clips that contain at most 48 frames. The model's ability for modeling long-term (minutes order) dynamics is unknown. 2) The 2D video dataset quality is not comparable to image datasets such as FFHQ and CelebA. Our model has flexible architecture that may support pre- pre-training joint training image datasets, yet this augmentation strategy has not been explored although it is promising and meaningful. For future work, we will explore modeling long-term dynamics with novel 3D representations that more suitable for 3D video generation, and leverage high-quality image datasets for data augmentation.
17


Reviews:
[{"review_id": "xi6PpZ25C7n", "reviewer": "Reviewer_7Zbn", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a 3D portrait video generative model learned from a dataset of monocular videos. A motion latent code is added to a triplane-based 3D GAN, in addition to an appearance code. A single motion latent controls the motion for the entire temporal sequence. The network also receives the timestep and camera poses as conditioning inputs. Temporal and Static discriminators are used for training.  ", "summary_of_the_review": "The paper presents an interesting solution to a novel problem with high-quality results. It would be good to better demonstrate the limits of the method in terms of duration of synthesized videos. ", "strength_and_weaknesses": "Strengths: These are the first 3D portrait video results. The quality of results are impressive, in terms of geometry, appearance and motion. The paper is well-written, and the evaluations are thorough. \n\nWeaknesses: \n- Since the motion component only controls the first few generator layers, fine-grained motion would be difficult to model. In addition, any lighting effects would not be modeled with the first four latents. I suspect the method does not generate any result where the lighting on the face changes between different timesteps. \n- As mentioned in the limitations, the method only deals with very small motion (upto 16 frames). It would be good to show some results with large motion, just to understand how it would fail. I suspect that the design choice of using a single motion latent vector for all timesteps is the limiting factor here. \n- For inversion, in Fig. 6, a different motion code is optimized for each frame. How different are the optimized codes for different frames? Would it be possible to use these optimized motion codes to retarget a different appearance code, or are they out-of-distribution? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "WEbq2gvRMCI", "reviewer": "Reviewer_YSuv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a generative framework for synthesizing multi-view consistent portrait videos. Towards this, the authors employed EG3D architecture in NeRF and made the network conditioned on motion features. 2 discriminators are used to maintain spatial and temporal consistency. While it’s claimed in the paper that this is the first work for the task, I have noticed the existing method [1] doing the same task and they are able to generate 3D-aware portrait videos with motion controllability.\n\n[1] P. Zhuang et al., Controllable Radiance Fields for Dynamic Face Synthesis, 3DV 2022.\n", "summary_of_the_review": "The paper is well-structured and the topic is interesting. The video quality is good. I’ll wait for the authors to reply to the questions.\n\n", "strength_and_weaknesses": "Strength:\n- The motivation of the proposed paper is interesting. The paper is well-written. The video quality looks reasonable.\n\nWeaknesses:\n- It’s unclear in the paper how is this work different from [1]. Please clarify.\n- I doubt if the video discriminator that only takes paired frames w/ the corresponding time delta is sufficient to maintain spatio-temporal consistency, especially for texture and albedo. One example is the man on the left in the first video. His mustache flickered over time, i.e., it looks faded sometimes. Also, from other generated portrait videos, I can see that the head shape deformation over time as it dynamically became bigger and smaller. Similar inconsistency happens to hair.\n- Following the last question w.r.t. spatio-temporal consistency, could the authors generate some videos from multiple views at one time for the same person, e.g., rendering 3 videos with the fixed left, middle, and right views, simultaneously?\n- It's unclear how the same motion features work on various face identities. To verify the constant effectiveness of motion conditioning, could the authors show multiple image/video results (i.e., different face identities) with the same motion features?\n- Can this work generate face videos with complex expressions, which the earlier work [1] can handle?\n- I’m worried about data processing. The authors used alignment which is commonly used in image preprocessing but it might not be proper for videos. Usually, people crop videos with a fixed cropping window. If doing alignment, will the processed videos become jittering? Did the authors observe this issue and fix it? Please explain.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VtaOI_nJt22", "reviewer": "Reviewer_zYuC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of training 3D generative models for portrait video generation with only 2D videos. In order to enforce temporal consistency and diverse motion, PV3D utilizes disentangled appearance and motion latent space. Extensive experiments demonstrate the effectiveness of the proposed approach.", "summary_of_the_review": "This paper tackles the important problem of 3D portrait video generator. The proposed approach is interesting and has been thoroughly verified. Therefore, I vote for acceptance.", "strength_and_weaknesses": "## Strengths\n\nThe proposed idea is interesting. The experiments are thorough and solid. The paper is clearly written.\n\n## Weakness\n\nI do not observe major issues. However, there are some question I hope authors can clarify. Please see below.\n\n## Questions\n\n**Network structure**\n\nIt is a little bit unclear to me how the exact structure of the \"T-Tri-plane Synthesis\" module in Fig. 2 looks like. From Appendix A.2, I feel like it is essentially some top-layers of vanilla StyleGAN2's generator since the first K layers of the original StyleGAN2's generator have been replaced by the \"motion layer\". Can authors clarify?\n\n**Camera pose conditioning**\n\n1. From the quantitative perspective, it is unclear to me whether conditioning the generator on the camera pose in necessary since there is no ablations about removing the generator's camera conditioning.\n\n2. From Fig. 2: it seems like during training, for a specific time $t_i$, the motion generator, the rendering from tri-plane, and the discriminator take the same camera pose $c_i$. Is this correct?\n\n3. For Sec.4.3's \"Camera Conditioning\":\n    1) For Tab. 2(c): it is unclear what the final strategy PV3D chooses is.\n    2) For \"All\" strategy, I think the description of \"condition the whole generator on the shared camera pose\" is misleading. At least from Fig. 2, \"the whole generator\" only has \"mapping network\" that can take camera pose as conditioning. My guess is that authors also refer to the rendering process. However, it is not in the generator. \n    3) For \"Map\" strategy: can authors clarify how to choose the \"shared pose\" given two poses?\n\n**Typos**\n- Eq. (2): there are unmatched parenthesis\n\n**Lacked references**\n\n- Zhang et al., Controllable Radiance Fields for Dynamic Face Synthesis. 3DV 2022. \n- Schwarz et al., VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids. NeurIPS 2022.\n- Zhao et al., Generative multiplane images: making a 2D GAN 3D-aware. ECCV 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vh5zML5Evx", "reviewer": "Reviewer_Y7rC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to synthesize multi-view consistent portrait videos from 2D observations.\nThe framework is built upon recent state-of-the-art 3D-aware image GAN and aids extra-temporal information to it.\nSeveral designs are introduced for both temporal smoothness and multi-view consistency.\nThe demo videos show high-quality results.", "summary_of_the_review": "Shown in the above.", "strength_and_weaknesses": "Strengths:\n1. This paper is well-written.\n2. The quality of multi-view videos is good.\n3. The motion synthesis layer is well-designed. The feature modulated by the motion code are fused with apperance feature in a residual manner, which is well-motivated and resonable.\n\nWeakness:\n1. The video discriminator only receives two frames, which harm the ability to synthesis long videos. I am wondering whether this method can enable long-term video synthesis.\n2. I am confused about the details of the generation pose condition. Do you simply concatenate the camera pose with latent code like EG3D? If so, are there any 2D billboard artifacts?\n3. For the discriminator pose condition, how do you inject the discriminator with two poses? If the [vi, vj] are embedded into a single vector, maybe the discriminator cannot differentiate the alignment between pose pairs and image pairs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster