A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization
Authors: Kim Youwang
Lee Hyun
Kim Sung-Bin
Suekyeong Nam
Janghoon Ju
Tae-Hyun Oh
Venue: Accepted by TMLR ()

Abstract: We propose NeuFace, a 3D face mesh pseudo annotation method on videos via neural re-parameterized optimization. Despite the huge progress in 3D face reconstruction methods, generating reliable 3D face labels for in-the-wild dynamic videos remains challenging. Using NeuFace optimization, we annotate the per-view/-frame accurate and consistent face meshes on large-scale face videos, called the NeuFace-dataset. We investigate how neural re-parameterization helps to reconstruct image-aligned facial details on 3D meshes via gradient analysis. By exploiting the naturalness and diversity of 3D faces in our dataset, we demonstrate the usefulness of our dataset for 3D face-related tasks: improving the reconstruction accuracy of an existing 3D face reconstruction model and learning 3D facial motion prior.

Published in Transactions on Machine Learning Research (08/2024)
A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization
Kim Youwang
nowwang.kim@@ppseec....
Deportment of Electricnl Engineering, POSTECH
Lee Hyun
hgunlees@postechh.c..
Department of Electrical Engineering, POSTECH
Kim Blung-Bini Deportment of Electrionl Engineering, POSTECH
sunghin
Suekyeong Nam KRAFTON
sk.nomm@krafton.com
Janghoon Ju KRAFTON
janghoon.ju@kkmetonnncomm
Tae-Hyun Oh
taekyun @postech.. oc.
Department Electricol Engineering & Grad. School of AI, POSTECH
I-CREATE, Yonsei University
Reviewed on OpenReview: https: // openreview. net/ forun? id= VVDNNN66JJW
Abstract
We propose NeuFace, a 3D face mesh pseudo annotation method on videos via neural re-paramcterized optimization. Despite the huge progress in 3D face reconstruction methods, generating reliable 3D face labels for in-the-wild dynamic videos remains challenging. Using NeuFace optimization, annotate the per-view/-frame accurate and consistent face meshes on large-scale face videos, called the NeuFace-dataset. We investigate how neural re- parameterization helps to reconstruct 3D facial geometries, well complying with input facial gestures and motions. By exploiting the naturalness and diversity of 3D faces in our dataset, we demonstrate the usefulness of our dataset for 3D face-related tasks: improving the reconstruction accuracy of an existing 3D face reconstruction model and learning 3D facial motion prior. Project page: https: youwaan youwang github. io/neuface
1 Introduction
A comprehensive understanding of dynamic 3D human faces has been a long-standing problem in computer vision and graphics. Reconstructing and generating dynamic 3D human faces key components for diverse tasks such as face recognition (Weyrauch et al., 2004; Blanz & Vetter, 2003), face forgery detection (Cozzolino et al., 2021; Rössler et al., 2018; 2019), video face editing (B R et al., 2021; Kim et al., 2018; Tewari et al., 2020), facial motion or expression transfer (Thies et al., 2015; 2016a; 2018), XR applications (Elgharib et al., 2020; Wang et al., 2021; Richard et al., 2021; Saito et al., 2024; Sung- Bin ct al., 2024a;b; EunGi et al., 2024), and human avalar generation (Raj et al., 2020; Ma et al., 2021; Youwang el al., 2022; 2024).
Recent studies (Wood al., 2021; 2022; Bae al., 2023; Yeh al., 2022) have shown that reliable datasets of facial geometry, even synthetic pseudo oncs, help achieve comprehensive understanding of "static"
Now Samsung Advanced Institute of Technology (SAIT).
i Eqqally contributed 2nd authors.

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: NeuFace-dataset provides reliable 3D face mesh annotations for MEAD, VoxCeleb2 and CelebV- HQ videos, which is significantly richer than the existing datasets in terms of the scale, diversity and naturalness. Abbr. (seq.: sequences, id.: identities, Dur.: duration, Env.: environment, emo.: emotion)
Dataset
No. seq. [K] No. id Dur. [Lrs]
Env.
No. emo.
Existing 3D face video datasets
BIWI
1.1
14
1.4
Lab.
12
COMA
0.15
12
0.1
Lab.
12
VOCASET
0.5
12
0.5
Lab.
NeuFace-dataset (ours)
1,245
21,048
2,090
Wild + Lab
NeuFaceMEAD
210
48
25
Lab.
+ NeuFacrvocCelld22
1,000
6,000
2,000
Wild
+ NeuFacrcoleby IQQ
35
15,000
65
Wild
3D faces. However, there is currently a lack of reliable and large-scale datasets containing "dymamic" " and "natural" 3D facial motion annotations. The lack of such datasets becomes a bottleneck for studying inherent. facial motion dynamics or 3D face roconstruction tasks by restricting them to rely on weak supervision, e.g., 2D landmarks or segmentation maps. Accurately acquired 3D face video data may miligate such issues but typically requires intensive and time- consuminn efforts with carefully calibrated multi-view cameras and controlled lighting conditions (Yoon et al., 2021; Joo et al., 2015; 2018; Cudciro et al., 2019; Ranjan et al., 2018). Few seminal works (Fanelli et al., 2010; Ranjan et al., 2018; Cudeiro et al., 2019; Zielonka et al., 2022) take such effort to build 3D face video datase Despite significant. efforts, the existing datasets obtained from such restricted settings are limited in scale, scenarios, diversity of actor identity and expression, and naturalness of facial motion (sce Table 1).
In contrast to 3D, there are an incomparably large amount of 2D face video datasets available online (Wang et al., 2020; Nagrani et al., 2017; Chung et al., 2018; Zhu et al., 2022; Parkhi et al., 2015; Cao et al., 2018; Karras et al., 2019; Wang et al., 2021; 2019; Liu et al., 2015), which are captured in diverse in-the-win wild environments but without 3D annotations. As successfully demonstrated in some 3D tasks (Fang et al., 2021; Bouazizi et al., 2021; Huang et al., 2022; Müller et al., 2021; Hassan et al., 2019; Bayer et al., 2016; Ng et al., 2022) as well as other analysis tasks (Miech et al., 2019; Nagrani et al., 2022; Lee et al., 2021), leveraging off thetlf reconstruction models is a common practice to obtain pseudo ground-trute truth of such in-the-wild videos that were already captured. They showed that high-quality and large-scale pseudo ground-truth is sufficient to achieve the statc-of-tho-art at the time of their works. Similarly, naïve approach is to construct large-scale 3D face video dalaset by curating existing 2D video datasets and obtain 3D face annotations with off- the-shelf face reconstruction models (Feng et al., 2021; Danecek et al., 2022). However, existing 3D face reconstruction models have limitations for reconstructing temporally smooth or multi-view consistent 3D face meshes from videos. This is because state-of-the-art face reconstruction models are typically trained on single-view static images only with 2D supervision; thus fail to extrapolate to faces having rare poses and yield jittered motion due to the per-frame independent inference.
To address these difficultiss, we propose NeuFace optimization, which reconstructs accurate and spatio- temporally consistent parametric 3D face meshes on videos. By re-parameterizing 3D face meshes with neural network parameters, NeuFace infuses spatio-tempont cues of dynamic face videos on 3D face reconstruction. NeuFace optimizes spatio- temporal consistency losses and the 2D landmark loss to acquire reliable face mesh pseudo-labels for videos.
Using this method, we create the NeuFace-datased the first large-scale, accurate and spatio-temporally consistent 3D face meshes for videos. Our dataset contains 3D face mesh pseudo-labels for large-scale, multi-view in-the-wild 2D face videos, MEAD (Wang et al., 2020), VoxCeleb2 (Chung et al., 2018), and CelebV-HQ (Zhu et al., 2022), achicving about 1,000 times larger number of sequences than existing facial motion capture datasets (see Table 1). Our dataset inherits the benefits of the rich visual attributes in large-scale face videos, e.g., various races, appearances, backgrounds, natural facial motions, and expressions. We assoss the fidelity of our dataset by investigating the cross-view vertex distance and the 3D motion stability index. We demonstrate that our dataset contains more spatio-temporally consistent and accurate 3D meshes than the competing datasets built with strong baseline methods. To demonstrate the potential of
2

---

Published in Transactions on Machine Learning Research (08/2024)
our dataset, we present two applications: (1) improving the accuracy of face reconstruction model and (2) learning a generative 3D facial motion prior. These applications highlight that NeuFace-dataset can be further used in diverse applications demanding high-quality and large-scale 3D face meshes. We summarize our main contributions as follows:
NeuFace, an optimization method for reconstructing accurate and spatio-temporally consistent 3D face meshes videos via neural re-parameterivation.
NeuFace-dataset, the first large-scale 3D face mesh pseudo-labels constructed by curating existing large-scale 2D face video datasets with our method.
Demonstrating the benefits of NeuFace- regressors, (2) learn 3D facial motion prior for long-term face motion generation.
(1) improve the accuracy of off-the-shelf face mesh
2 Related work
3D face datasets. To achieve a comprehensive understanding of dynamic 3D faces, large-scale in-lhe-wild 3D face video datasets are essential. There exist large-scale 2D face datasets that provide expressive face images or vidcos (Wang et al., 2020; Nagrani et al., 2017; Chung et al,, 2018; Zhu et al., 2022; Parkhi et al., 2015; Cao el al., 2018; Karras el al., 2019; Liu el al., 2015) with diverse attributes covering wide variety of appearances, races, environments, scenarios, and emotions. However, most 2D face datasets do not have corresponding 3D annotations, due to the difficulty of 3D face acquisition, especially for in-the-wild environments. Although some recent datasets (Yoon et al., 2021; Ranjan et al., 2018; Cudciro et al., 2019; Zielonka et al., 2022; Wood et al., 2021) provide 3D face annotations with paired images or videos, 1 they are acquired in the restricted and carefully controlled indoor capturing environment, e.g., laboratory, yielding small scale, unnatural facial expressions and a limited variety of facial identities or features. Achieving in-the-wild naturalness and acquiring true 3D labels would be mutually exclusive in the real-world. Due to the challenge of constructing a real-world 3D face dataset, FaceSynthetics (Wood et al., 2021) synthesizes large-scale synthetic face images and annotations derived from synthetic 3D faces, but limited in that they only publish images and 2D annotations without 3D annotations, which restrict 3D face video applications. In this work, we present the NeuFace-dataset the first large-scale 3D face mesh pseudo-labelt paired with the cxisting in-tho-wild 2D face video datasets, resolving the lack of the 3D face vidoo datascts.
3D face reconstruction. To obtain reliable face meshes for large-scale face videos, we need accurate 3D face reconstruction methods for videos. Reconstructing accurate 3D faces from limited visual cues, e.g., a monocular image, is an ill-posed problem. Model-based approaches have been the mainstream to mitigate the ill-posedness and have advanced with the 3D Morphable Models (3DMMs) (Blanz & Vetter, 1999; Paysan et al., 2009; Li et al., 2017) and 3DMM-based reconstruction methods (Zollhöfer et al., 2018; Egger et al., 2020; Feng et al., 2021; Danecek et al., 2022; Zielonka et al., 2022).
3D face reconstruction methods can be categorized into learning- based and optimization- approaches. The learning- based approaches, e.g., (Feng et al., 2021; Danecek et al., 2022; Zielonka et al., 2022; Sanyal et al., 2019a; an Tran et al., 2016), use neural networks trained on large-scale face image datasets to regress the 3DMM parameters from a single image. The optimization- based approaches (Blanz & Vetter, 2003; Huber et al., 2015; Chen et al., 2013; Wood et al., 2022; Thies et al., 2015; Gecer et al., 2019) optimize the 2D landmark or photometric losses with extra regularization terms directly over the 3DMM parameters. Given a specific image, these methods overfit to 2D landmarks obscrvations, thus showing better 2D landmark fit than the learning-based hods. These approaches are suitable for our purpose in that we need accurate reconstruction that best fits each video. However, the regularization terms are typically hand-designed with prior assumptions that disregard the input image. These regularization terms often introduce mean shape biases (Feng et al., 2021; Pavlakos et al., 2019; Bogo et al., 2016; Joo et al., 2020), due to their independence to input data, which we call the dala-independent prior. Also, balancing the losses and regularization is inherently cumbersome and may introduce initialization sensitivity and local minima issues (Joo et al., 2020; Pavlakos al., 2019; Bogo et al., 2016; Choutas al., 2020).
MICA released the medium-scale 3D annotated face datasets, but only a single identity parameter per video is provided, not the facial poses expression parameters, i.c., static 3D) faces.
3

---

Published in Transactions on Machine Learning Research (08/2024)
Detected 2D landmarks
22
multiview
view (aligned)
Multi-view consistency
@W
Cross-view inconsistent faces
Cross-view consistent
view
Face mesh regressor
Temporal smoothing
(Leamnble))
time Temporally noisy faces
time
Temporally smoothed faces
frames
Initial recon.
temporal
7C
= 2D + temporal multiview
Figure 1: NeuFace optimization. Given 2D face videos, NeulFace optimizes spatio-temporally consistent 3D face meshes. NeuFace updates the neural network parameters that re-parameterize the 3D face meshes with 2D landmark loss and spatio-temporal consistency losses.
Instead of hand-designed regularization terms, we induce such effects by optimizing re-parameterized 3DMM parameters with a 3DMM regression neural network, called NeuFace optimization. Such network parameters are trained from large-scale real face images, which implicitly embed strong prior from the trained data. Thereby, we can leverage the favorable properties of the neural re-parameteriaation 1) an input dala dependeepnddnt initialization and prior in 3DMM parameter optimization, 2) less bias toward mean shape, and 3) stable optimization robust to local minima by pee-pearaeterriee model (Cooper, 2021; Du et al., 2019a; Neyshabur et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019b; Youwang et al., 2024). Similar re-parameterizations were proposed in (Joo et al., 2020; Grassal et al., 2022), but they focus on the human body in a single image input with fixed 2D landmark supervision, or use MLP to re-parameterize the per-vertex displacement of the 3D face. We extend it dynamic faces in the multi-view and vidco scttings by sharing the neural parameters across views and frames, and devise an alternating optimization to self-supervise spatio-temporal consistency,
3 NeuFace: a 3D face mesh optimization for videos via neural re- pataammtteriztioon
In this soction, we introduce the neural ro-parameterization 3DMM (Sec. 3.1) and NeuFace, an optimization to obtain accurate and spatio-temporally consistent face meshes from videos (Sec. 3.2). We discuss the benefit. of neural re- parameterization (Sec. 3.3), and show NeuFace's reliability face mesh annotator (Sec. 3.4).
3.1 Neural re- parameterization of 3D face meshes
We use FLAME (Li et al., 2017), a renowned 3DMM, as a 3D face representation. 3D face mesh vertices M and facial landmarks J for frame vidcos can be acquired with the differentiable skinning: M,J=FLAME(r, 0, B, /b), where F 33 E 12 E 100 and sb 5 denote the head orientation, face poses, face shape and expression coefficients, respectively. For simplicity, FLAME. parameters O be represented as, θ 0, B, vb). We further re-parameterize the FLAME parameters O and weak perspective camera parameters p E IR Fx3 for vidco frames {I / into neural network, $ with parameters w, f.e., [e, (w((1)) We use the pre-trained DECA (Feng et al., 2021) or EMOCA (Danecek et al., 2022) encoder for Φw-
3.2 NeuFace optimization
Given the Nr frames and Ny views of face video { Ny. Ny,Ny f=1,==1 NeuFace aims to find the optimal neural network parameter w that re- re-parameterizes accurate, multi-view and temporally consistent face meshes (see Fig. 1). The optimization objective is defined as:
w = arg min LeD + Cremprrall + Aview Lumltiview,
(1)
w
4

---

Published in Transactions on Machine Learning Research (08/2024)
where } denotes the weights for each loss term. Complex temporal and multi-view dependencies among variables in the losses would make direct optimization difficult (Afonso et al., 2010; Salzmann, 2013; Zhang, 1993). We ease the optimization of Eq. (1) by introducing latent target variables for self-supervision in an Expectation-Maximiaann
(EM) style optimization (sce Sec. B for the details of EM style optimization).
2D landmark loss. For each iteration t, we compute LND as a unary term, following the conventional 2D facial landmark re-projection loss (Feng et al., 2021; Danecek et al., 2022) for the landmarks in all different. frames and views:
1 EN..Ny C2D = Ny M [T
J. (. pj.)) - J5.ell2-
(2)
where (): denotes the weak perspective projection, and J(w) is the 3D landmark from (w(.). Eq. (2)
computes the pixel distance between the pre-detected 2D facial landmarks j and the regressed and projected 3D facial landmarks T ([[(w,,)) p)) j stays the same for the whole optinization. We use FAN (Bulat & Trimiropoulos, 2017) to obtain j with human verification to reject the failure cases.
Temporal consistency loss. Our temporal consistency loss reduces facial motion jitter caused by per- frame independent mesh regression on vidcos. Instead of a complicated Markov chain style loss, for cach iteration 6, we first estimate latent larget meshes that represent temporally smooth heads in Expectation step (E-step). Then, we simply maximize the likelihood of regressed meshes to its corresponding latent target in Maximization step (M-step). In E-step, we feed {I,,0]/=1,0 Ny.NN into the network w and obtain FLAME and camera parameters, ,O', For multiple frames in view v, we extract the head orientations ri from O and convert it to the unit quaternion q. To generate the latent target, i.e., temporally smooth head orientations L..,, we take the temporal moving average over L. In M-step, we compute the temporal consistency loss as:
Cxemporal
No.NN Np 2f=1e= I1gh, 4/.vll22
(3)
where q is the unit-quaternion representation r. We empirically found that such simple consistency loss is sufficient enough to obtain temporal smoothness while allowing more flexible expressions.
Multi-view consistency loss. Although the aforementioned L2D roughly guides the vuew consistency of landmarks, it cannot guarantee the consistency for off-landmark or invisible facial re- gions across views. Therefore, for multi-view cap- tured face videos (Wang et al., 2020), we leverage a simple principle to obtain consistent meshes over different views: face geometry should be consis- tent across views at the same time. The goal is to bootstrap the per-view estimated noisy meshes by referencing the visible, or highly confident facial regions across different views. Analogous to the temporal consistency loss, in M-step, we compute the multi-view consistency loss as follows:
peram. Visible vertex Invisible vetex
dw
Align
view
M5.
C Menge
Ny
4w
Align
view
Mf.2
Sif
Align
view
Mf
Figure 2: Multi-view bootstrapping. Given initial mesh predictions for each view in frame f, we align and merge the meshes depending on the confidence. The boot- strapped mesh serves as a target for computing Luulttview-
Amultiview
Ny NpNv E IM M/,
E
(4)
where M, denotes the latent target mesh vertices estimated in E-step of each iteration. In E-step, given vertices M f.: of multiple views in frame f, we interpret the vertex visibility as the per-vertex confidence. We assign the confidence score per each vertex by measuring the angle between the vertex normal and the camera ray. We set the vertices as invisible if the angle is larger than the threshold Ta, and the vertex has a deeper depth than Tz, i.c., 2 < T.. We empirically choose Ta = 72p, T₂ -0.08. To obtain the latent target mesh M, we align per-view estimated meshes to the canonical view, and bootstrap the meshes by taking the weighted average of M depending on the confidence (see Fig. 2). With this, Eq. (4) constrains the vertices of each view to be consistent with My.
5

---

Published in Transactions on Machine Learning Research (08/2024)
Normalized absolute megnitude gredient map
Input
Baseline
NeuFace (ours)
Figure 3: Data-dependent gradient. NeuFace optimization obtains a richer gradient map regarding the pixel-level facial details (1st row). Thus, our method achieves more expressive and accurately meshes than the baseline (2nd row).
Overall process. We first estimate all the latent variables, å and M E-step. With the estimated latent variables as the self-supervision target, optimize Eq. (1) over the network parameter w as M-step. This single alternating iteration updates the optimization parameter w at iteration t. We iterate alternating F-step and M-step until convergence. After convergence, we obtain the final solution [O", pd] by querying video frames to the optimized network, i.e., [0", "p""""">"ww- (fT N Nv.
3.3 Why is NeuFace optimization effective?
Note that one simply update FLAME parameters directly with the same loss in Eq. (1). Then, why do need neural re-parameterizzatoon of 3D face meshes? We claim such neural re-parameterization allows data- dependeetnd mesh update, which the FLAME fitting cannot achieve. To support our claim, we analyze the benefit of our optimization by comparing it with the solid baseline.
Baseline: FLAME fitting. Given the same video frames {Lfu]f= and the same initial FLAME and camera parameters [Ob: Pb] as NeuFace?, we implement the baseline optimization as:
O Pt.
argggmm min Lan Sttmppeemporal Aview + XrCr AoCo + >aLa + Aw @p.Pb
(5)
where the losses L2D, fremporal and mulltiiew are identical to the Eqs. (2), (3), and (4). for, Co. La and Lass are the common regularization terms used in (Li al., 2017; Wood al., 2022).
Data-dependent gradients for mesh update. We analyze the data-dependency of the baseline and NeuFace optimization by investigating back-propagater gradients. For the FLAME fitting (Fq. (5)), the update rule for FLAME parameters Ob at optimization step t is as follows:
2L
o O a0,
(6)
Ny.NY where C denotes the sum of all the losses used in the optimization. In contrast, given video frames { or simply I, the update for our NeuFace optimization is as follows:
f=l,
OC w - aa 0 Ow w 1+1 v a Ow!
JL 80 w = w' - a( OL
Ow ow (I)),
(7)
00
where Oy, is tee/over-parameteeriee by the neural network Dwt, i.e., O' 2
By comparing the back-propagated gradient terms in Eqs. (6) and (7), we can intuitively notice that the update for NeuFace optimization (Eq. (7)) is conditioned by input I, yielding data-dependent mesh update. With data-dependent gradient Jw 2 wr (I), NeuFace optimization may inherit the implicit prior embedded in the pre-trained neural model, e.g., DECA (Feng et al., 2021), learned from large-scale real face images.
8
2To conduct a fair comparison with strong baseline, initialize 1O. Pt. as the prediction of the pre-trained DECA. This identical Face optimisation (Fq. (1)); only the optimization variable different.

---

Published in Transactions on Machine Learning Research (08/2024)
This allows NeuFace optimization to obtain expressive 3D facial geometries, well complying with input facial gestures and motions.
It is also worthwhile to note that, thanks to weerpprrameterization of Pw(-) w.r.t. O we benefit from the following favorable property. For simplicity, we consider a simple 1y-loss and a fully connected ReLU network, but it is sufficient to understand the mechanism of NeuFace optimization.
Proposition (Informal). Global convergence. For the imput data x E [0, 1juxdn, paired labels y" E Ruxdous, and an over-pon parameterrize L-layer fully connected network (.()) with ReLU activation and uniform weight widths, consider oplimizing the non-conver problem: arg minw [(w) = = = y-'18- Under some assumptions, gradient descent finds global optimum in polynomial time with high probability.
Proposition can be derived by simply remposisgg the results by Allen-Zhu et al. (2019). Its proof sketch can be found in the appendix. This hints that our moe-parameteriiation helps NeuFace optimization achieve robustness to local minima and avoid mean shape biases.
To see how data-dependent gradient of NeuFace affects the mesh optimization, we visualize the absolute magnitude of the back- procc--rppggtedd gradients of each method in Fig. 3. The bascline optimization produces a sparse gradient map along the face landmarks, which disregards the pixel-level facial details, e.g., wrinkles or facial boundaries. In contrast, NeuFace additionally induces the dense gradients over face surfaces, not just sparse landmarks, which are helpful for representing image-aligned and detailed facial expressions meshes. Thanks to the rich gradient map, our method yields more expressive and accurately image-aligned meshes than the baseline.
3.4 How reliable is NeuFace optimization?
Many recent face-related applications (Ng et al., 2022; Khakhulin et al., 2022; Feng et al., 2022) utilize a pre-trained, off-the-shed 3D face reconstruction model or the FLAME fitting (Eq. (5)) as a pseudo ground- truth annotator. Compared to such conventional face mesh annotation methods, we discuss how reliable Neuface optimization is. Specifically, we measure the vertex-level accuracy of the reconstructed face meshes by NeuFace optimization on the motion capture videos, VOCASET (Cudeiro et al., 2019).
VOCASET is small-scale facial motion capture dataset that provides registered ground-truth mesh se- quences. Given the ground-truth mesh sequences from the VOCASET, we evaluate the Mean-Pee--Vrree--Erree (MPVE) (Cho et al., 2022; Lin et al., 2021b;a) of face meshes obtained by pro-trained DECA, FLAME fitting and our method. In Fig. 4, NeuFace optimization achieves more vertex-level accurate meshes than other methods, i..,, lower MPVE. Note that FLAME fitting still achieves competitive MPVE with ours, which shows that it is a valid, strong baseline. Such favorable mesh accuracy of NeuFace optimization motivates us to leverage it as a reliable face mesh annotator for large-scale face videos, and build the NeuFace-dataset.
VOCASET GT
0 DECA prediction FLAME fitting
NeuFace optimization
Method
DECA FLAME fitting (Eq. (5))
NeuFace (ouns)
MPVE [mm] 4
38.4
31.5
30.6
Figure 4: Given the ground-truth meshes,
optimization reconstructs more vertex-level accu- rate meshes than the competing methods.
4 The NeuFace-dataset
The NeuFace-dataset provides accurate and spatio-temporally consistent face meshes of existing large-scale 2D face video datasets; MEAD (Wang et al., 2020), VoxCeleb2 (Chung et al., 2018), and CelebV-HQ (Zhu et al., 2022). Our datasets are denoted with NeuFace(-)) and summarized in Table 1. The NeuPace-datasel is, namely, the largest 3D face mesh pseudo-labeled dataset in terms of the scale, naturalness, and diversity of facial attributes, emotions, and backgrounds. Please refer to the appendix for the visualization of the dataset, acquisition method, and filtering details.
By sacrificing the complexity of proof, the same conclusion holds ResNet (Allen-Zhu al., 2019).
7

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: Quantitative evaluation. NeuPace-D) D/e--daa--sssss (ours) significantly outperform the other datasets in multi-view consistency (CVD), temporal consistency (MSlaD), and the 2D landmark accuracy (NME). Abbr. (L: landmark, V: vertex.)
MEAD
VoxCeleb2
CelebV-HQ
Dataset
MSLjn t
MSI +
CVD
NME
MSI
MSL f
NME
MSI 1
MSI t
NME $
Bese-datasct (Eq. (5))
0.034
0.053
0.192
4.34
0.034
0.056
3.32
0.030
0.047
3.65
DECA-dutaset
0.011
0.016
0.209
4.65
0.028
0.044
4.78
0.012
0.018
5.34
CeuFFae-D-datased (ours)
0.206
0.305
0.103
2.58
0.095
0.137
2.19
0.054
0.074
2.55
EMOCA-dataset
0,010
0.016
0.199
5.42
0.003
0.004
4.77
0.005
0.007
5.57
NeuFace-
NeuFace-E-dataset E-dataset (ours)
0.209
0.312
0.104
2.28
0.028
0.048
2.38
0.053
0.077
2.86
Input
S
S
S
S
8
S5
8
S
8
9
US
A
DECA-dataset
a
a
a
0
2
NeuFace D-dataset
Down
Front
L-30
L-60
R-30
R-60
Top
Figure 5: Multi-view consistent face meshes. NeuFace-dataset contains multi-view consistent meshes compared to the DECA-dataset. L- and R- denote Left and Right, and 30 and 60 denote the camera view angles from the center.
4.1 Fidelity evaluation of the NeuFace-datasse
We assess the fidelity of our dataset in terms of spatio-temporal consistency and landmark accuracy. We make competing datasets and compare the quality of the generated mesh annotations. First, we compose the strong baseline, Base-dataset, by fitting FLAME with Eq. (5). We also utilize pre-trained DECA and EMOCA as mesh annotators and built DECA-dataset and EMOCA-dataset, respectively. Finally, we build two versions of our dataset, i.e., NeuFace-D, D, and NeuFace--,, where each dataset generated via Eq. (1) with DECA and EMOCA for the neural re-parameterization @w, respectively.
Temporal consistency. We extend the Motion Stability Index (MSI) (Ling al., 2022) to MSID and evaluate the temporal consistency of each dataset. MSI computes reciprocal of the motion acceleration variance of either 3D landmarks or vertices and quantifics facial motion stability for given NE frame vidco, E. न|क. where a' denotes the 3D motion acceleration of i-th 3D landmarks
(T as MSlsn((1,)) vertices, a()) the temporal variance, and K the number of landmarks vertices. If the mesh sequence has small temporal jittering, i.e., low motion variance, it has high MSI3D value. We compute MSIn for landmarks and vertices, i.e., MSI and MSIY, respectively. Table 2 shows the MSI and MSI averaged over the validation sets. For the VoxCeleb2 and CelebV-HQ splits, the NeuFace-D/ D/Facct-//F-dataeet outperform the other datasots in both MSIDDS. Remarkably, we have improvements on MSIDD more than 20 times MEAD. We postulate that the multi-view consistency loss also strengthens the temporal consistency for MEAD. other words, our losses would be mutually helpful when jointly optimized. We discuss it through loss ablation studics in the later section (Sec. 4.2).
Multi-view consistency. We visualize the predicted meshes over different views in Fig. 5, where per-view independent estimations are presented, not single merged one. We verify that the NeuFace-D-dalasel contains multi-view consistent meshes compared to the DECA-dataset, especially near the mouth region. See appendix for the comparison of the EMOCA-dataset and NeuFace-Fddataset. As quantitative mcasure, we compute the cross-view vertex distance (CVD), i.e., the vertex distance between two different views, and j, in the same frame f: || MM, Myill We compare the averaged CVD of all views in Table 2. CVD is
8

---

Published in Transactions on Machine Learning Research (08/2024)
Table 3: Ablation results on the different loss functions. We evaluate the effect of our proposed spatio-temporally consistent losses by changing the configurations of the loss combinations. Optimizing full loss functions shows favorable results on CVD and NME while outperforming MSI compared to other configurations. We cannot analyze the effect of Lmultiview for VoxCeleb2 and CelcbV-HQ since they are taken in the single-came camera setup, Abbr. L: landmark, V: vertex.
NeuFace
MEAD (Wang et al, 2020)
VorCeleb2 (Chung sl,, 2018)
CelebV-IIQ (Zhm al,, 2022)
Cap
Cmuttiveww
CVD 4
MS 1
MSI 1
NME 1
MSSI t
MSI 1
NME 4
MSE 1
MSE 1
NME.
0.333
0.015
0.022
2.44
0.001
0.001
2.71
0.001
0.003
3.88
0.110
0.015
0.023
2.49
0.294
0.176
0.250
2.56
0.095
0.137
2.19
0.54
0.074
2.55
0.103
0.206
0.305
2.58
DECA (Feng et al,, 2021)
0,209
0.011
0.016
4.65
0.028
0.044
4.78
0.012
0.018
5.34
Down 0.000
0.000
Front 0.333
0.003
frant 0.79
0.000
DECA
NeuFace
Down
L-60
0224
8:251
0.327
0.003
L-60
6.12
6.12
6062
0.000
R-30
0:151
0.390
0.000
0.106
0.318
L-60
R-60
0.266
9.22
0.325
0.251
0.138
R-40
0.323
0.330
0.133
0.111 0.005 0.000
Top
0.22
0.122
0.219
0.01
0:200
0.27
2.000
0.134
0.081
0.108
0.126
0.100 0.325 0.000
Input
DECA
Ours lonly kza)
Ours (Fall)
Down
Front
L-30
L:00
R:30
8:60
Top
Down
Pront L-30 L.00 R-30 R:60 Top
Figure 6: Effect of the multtvveww By optimizing full objectives (Eq. (1)), the generated meshes are fitted to each of its 2D landmarks and are also multi-view consis- tent. Views are aligned for the visualization.
Figure 7: Comparisons of cross-view vertex dis- tance. We quantitatively show the multi-view consis- tency of our method by averaging the cross-view vertex distance on the validation set. L- and R- denote Left and Right, respectively, and 30 and 60 denote the view angles which the video is captured from.
only evalnated on the MEAD dataset, which is in a multi-camera setup. While the DECA-/EMOCA-atttee results in high CVD, the NeuFace-dataset shows siguificantly lower CVD on overall views.
2D landmark accuracy. A trivial solution obtain low CVD and high MSIan is to regress the same mean face meshes across views and frames regardless of the input image. To verify such occureecce, measure the landmark accuracy of the regressed 2D facial landmarks using the normalized mean error (NME) (Sagonas et al., 2016). The NcuFacc-D/ D/E-datastt outperform the other datasets in NME, i.e., contain spatio-temporally consistent and accurately landmark-aligned meshes.
4.2 Ablation on loss functions
We analyze the effect of our proposed spatio-temporal consistency losses in the NeuFace optimization, Cuultiview, and Ltemporal- We evaluate the quality of the meshes obtained by optimizing cach of the loss configurations. All the experiments are conducted on the same validation set as the Table 2.
Only C2D. We start by optimizing the loss function with only 2D facial landmark re- re-projection error, CTD- As we optimize C2D, the obtained meshes achieve lower NME than DECA over the whole validation set (see Table 3). However, we observe that this may brcak both multi-view and temporal consistencics, degrading the CVD and MSI compared to that of DECA. The qualitative results in Fig, 6 also show that optimizing only L2D reconstructs more expressive but inconsistent meshes over different views. Therefore, we propose loss functions that can induce multi-view and temporal consistencies to the meshes during the optimization.
By optimizing Lmultiview along with LTD, we obtain significantly lower CVD than DECA and the meshes optimized with only L2D. We have not optimized any regularization term which induces temporal consistency; thus, MSTs remain low. As discussed in Sec. 4.1, a trivial solution for achieving low
Cmultiviee.
9

---

Published in Transactions on Machine Learning Research (08/2024)
Multi-view videos
(views merged)
merged) In-the-wild
(x-t)
Spatio-temporal consistency comparison. NeuFace reconstructs multi-view consistent 3D faces (left); roduced misalignment for faces (each view is aligned to the canonical posc, color-coded and alpha-blendedt and lemporally stabilized motion (right); reduced jitter than the competing method (we concatenate vertical cyan line in each frame along time), while being accurate.
Figure 8:
CVD is to regress mean faces over different views. However, the optimization with Cmultviiw achieves low CVD while comparable NME to the Lan optinization, which proves not to be falling into trivial solution. Note that Lmlttiviw can only be measured in the multi-camera setup, e.g. MEAD.
+ As we optimize Lremporal with LND, we observe substantial improvements in MSIs over the whole validation set. Interestingly, jointly optimizing these two losses can further achieve better NME in VoxCeleb2 and CelebV-HQ datasets. We postulate that, for in-the-wild challenging cases, e.g., images containing extreme head poses diverse background scenes, only optimizing L2D could fail to regress proper meshes, as it may generate meshes that break out of the facial regions. On the other hand, Liemporal could prevent the regressed mesh from breaking out from the facial regions to certain extent.
Full loss function. With the observations of the effect on each proposed loss, we optimize the full loss on MEAD. The quantitative results in Table 3 show that NeuFace optimization achieves comparable NME while outperforming CVD and MSIs compared other settings. As analyzed in Sec. 4.1, we postulate that our proposed spatio-temporal losses mutually helpful for generating multi-view and temporally consistent meshes.
functions (NeuFace optimization),
Lupatial
The advantage of jointly optimizing all the losses can also be found in the qualitative results; the reconstructed face meshes are well-fitted to its 2D face features, e.g., landmarks and wrinkles, and multi-view consistent (See Fig. 6). In addition, we compare the view-wise averaged CVD between NeuFace optimization and DECA in Fig. 7. While DECA results in high CVD, especially for the views with self-occluded regions, such as Left-60 and Right-60, NeuFace shows significantly lower CVD on overall views. We also evaluate the meshes reconstructed by pre-trained DECA (Feng et. al., 2021) for comparison (see Fig. 8).
5 Applications of the NeuFace-datasets
In this section, we demonstrate the usefulness of the NeuFFace-dataset. We learn generative facial motion prior from the large-scale, in-the-wild 3D faces in our dataset (Sec. 5.1). Also, we boost the accuracy of an off-the-shelf face mesh regressor by exploiting our dataset's 's 3D supervision (Sec. 5.2).
5.1 Learning 3D human facial motion prior
A facial motion prior is versatile tool to understand how human faces move over time. It generate realistic motions or regularize temporal 3D reconstruction (Rempe et al., 2021). Unfortunately, the lack of large-scale 3D face video datasets makes learning facial motion prior infeasible. We tackle this by exploiting the scale, diversity, and naturalness of the 3D) facial motions in our dataset.
We learn a 3D facial motion prior using HuMoR (Rempe al., 2021) with simple modifications. HuMoR is conditional VAE (Sohn et al., 2015) that learns the transition distribution of human body motion. We represent the state of facial motion sequence the combination
Learning facial motion prior.
10

---

Published in Transactions on Machine Learning Research (08/2024)
VOCASET
NeuFacewика
NeuFaCPyaccntt
time
0.0s 1.0s 2.0s
5.0s
8.0s
10.0s
Figure 9: Long-term facial motion generation using learned motion prior. The motion prior trained with small-scale, diversity-lmitte VOCASET fails to generate natural motion, while the motion prior trained with NeuFacevosChleh2 generales diverse and natural long-term facial motion.
Table 4: Quantitative evaluation of learned facial motion prior. We evaluate the naturalness and diversity generated long-term motions from different motion prior models. HuMoR-Face trained with existing facial motion capture dataset, e.g., VOCASET, fails to generate natural and diverse facial motions.
HuMoR-Face trained w/
Scale
Environment
FD
APD [em]
Existing motion capture dataset VOCASET [Cudeiro et al., 2019)
Small
In-the-lab
420.92
Our dalaset
NenFactyEAD
Large
In thethe-lal
78.99
3.56
NeuFacevoeesllh22
Large
wi-dhe-wlld
31.32
52.69
of FLAME parameters and landmarks in the NeuFace-dataset and train the dedicated face motion prior, called HuMoR-Face. We train three motion prior models (HuMoR-Face) with different training datasets, d.e., VOCASET (Cudeiro et al., 2019), NeuFaCeMEAD, and NeuFacevoxCelebz- Please refer to appendix and HuMoR (Rempe et al., 2021) for the details.
Long-term face motion generation. We evaluate the validity and generative power of the learned motion prior by generating long-term 3D face motion sequences (10.0s). Long-tern motions are generated by auto-regressive sampling from the learned prior, given only a starting frame as the condition (see Fig. 9). VOCASET provides small-scale, in-the-lab captured meshes, thus limited in motion naturalness and facial diversity. Accordingly, the HuMoR-Face trained with VOCASET fails to learn valid human facial motion prior and generates unnatural motion. Using only the subset, NeuFACeMEAD, the long-term stability of head motion has significantly enhanced. Please refer to the supplementary video for motion comparisons.
We quantitatively evaluate HuMoR- models using two metrics: motion Fréchet distance (FD) (Ng et al., 2022) and average pairwise distance (APD) (Aliakbarian al., 2020; Rempe al., 2021). FD measures the naturalness like the FID score (Heusel al., 2017) and APD measures the diversity of generated motions. For APD, generate 50 long-term motions from the same initial state and compute the mean landmark distance between all pairs of samples. HuMoR--ace models trained with large-scale and diverse motions, i.e., the NeuFace-datasct, show superior performance in naturalness and diversity (sce Table 4). Specifically, the HuMoR-Face trained with NeuPacevsocCeett shows substantia ial enhancement on APD. APD is not reported for the HlMMRR-Face trained with VOCASET, since the model fails to generate realistic motion. We attribute such high-quality motion prior the benefit of the NeuFaco-dataset: large-scale facial motion annotations. Further, exploiting diverse in-the-wilh wild dynamic, and natural motion annotation from NeuFacevoxCeleezz helps HuMoR- Face learn weal-world motion prior and surprisingly generate much diverse and dynamic motions.
11

---

Published in Transactions on Machine Learning Research (08/2024)
Table 5: Improving the face reconstruction accuracy. (a) NeuPace-datasel helps the model reconstruct more occlusion robust and expressive 3D faces than the original model. Green and red dots denote visible and invisible 3D landmarks, respectively. (b) As a result, DeCANeuFace, 2D, DeCANeuFace,3D achieve better 3D reconstruction accuracy than DECAoriginal.
Frror
(moe)
Model
Test-
Medton
Menn
Std
SDMM CNN (Taa Tran 2017)
2:33
2.05
PRN:: (Faage al, woo wus
1.95
1.58
2019k)evrn 2019 [Stanng al, 2020) sccy acar SDDFA- V2 (Gro al, 2020) socv soaa Demonlandmarre. (Wood et al, wol)) voov MICA (Ziclonio c5 2022)
BlngNet (Sugga
1.31
1.87
2.63
1.23
1.57
1.300
1.02
1.26
1.08
0.50
1.11
0.92
DECAsrates (Fiug ct al, 2023) soonaru DECAA (Dins
1.15
1.46
1.25
1:35
1:34
1:20
DECAN (Ouzs)
1.38
1.19
DECAergial
DECAN
DECA Nec
(a)
(b)
5.2 Improving the 3D reconstruction accuracy
Due to the absence of large-scale 3D face video datasets, existing face mesh regressor models utilize limited visual cues, such 2D landmarks or segmentations. Thus, utilize the NeuFace-dataset to add direct 3D supervision to enhance the performance of such model.
3D supervision with the NeuFace-dataset. We implement the auxiliary 3D supervision as conventional 3D vertex and landmark losses (Kolotouros al., 2019; Cho et al., 2022; Lin et al., 2021b;a). Given regressed and our annotated mesh vertices, M, M E M and regressed and our annotated 3D landmarks, J,J E R the auxiliary 3D losses computed as: L-/ 1ob--N 1 | - |J-- where NM, Ny is the number of mesh vertices and landmarks, respectively.
Enhancement on 3D reconstruction accuracy. By fine-tuning DECA (Feng et al., 2021) using the images of MEAD (Wang et al., 2020), VoxColeb2 (Chung et al., 2018) and CelebV-HQ (Zhu et al., 2022), with and without our 3D supervision, we obtain DECANouFace.3D and DECANeuFaco.2DD Following the evaluation protocol of the NoW benchmark (Sanyal et al., 2019a), we reconstruct 3D faces for the provided images via each model and report the 3D reconstruction errors. In Table 5, our DECA shows lower 3D) roconstruction error than DECAorigiual and DECANeuFace.22.
6 Discussion, Limitation and Conclusion
We develop NeuFace, an optimization for generating accurate and spatio-temporally consistent 3D face mesh pseudo-labels on videos with provable optimal guarantee. Moreover, with the technique, we build the Neuaace-dataset, large-scale 3D face meshes paired with in-the-wild 2D videos.
We demonstrate the potential of the diversity and naturalness of our NeuFace-dataset learn generative 3D facial motion prior. Also, we improve the reconstruction accuracy of a de-faclo standard 3D face reconstruction model using our dataset. Note that the focus of our NeuFacc optimization is to build the 3D face dataset for videos. The experiment for the static face reconstruction in Table 5 was conducted to demonstrate the potential application of our method and dataset. Given our emphasis on video face data, our model may not obtain the best accuracy for static face reconstruction. However, it is important to note that our method still shows marked improvement over existing data. This underscores the effectiveness of the NenFacc-dataset in enhancing 3D face reconstruction. We expect NeuFace to open up opportunities by providing large-scale, real-world 3D face video data, the NeuFacc-dataset, The failure cases of NeuFace optimization could occur when the 2D vidco contains extreme degradations, e.g., motion blur, low resolution, extremely (>50%) occluded, so that the 2D keypoint delection fails. Please note that when we construct the Neuaace-dataset, we tackle these cases with automatic filtering followed by human verification (will be discussed in appendix Sec. A), which guarantocs the reliability of the datasct. Since 2D landmark human annotations are relatively cheaper than any other signals, we think using better 2D landmarks can mitigate this limitation.
training dataset to
reliable data curation method.
12

---

Published in Transactions on Machine Learning Research (08/2024)
Acknowledgment
This research was supported by a grant from KRAFTON AI, and also partially supported by Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea (MST) (No.RS-2023-0022556) Development of Artificial Intelligence for Text-based 3D Movic Generation; RwR.SS222--I22222222 Visual Intelligence for Space- Time Understanding and Generation based on Multi-layered Visual Common Sense; and RRS-2022-TT22112 Development. of Artificial Intelligence Technology for Self- fel--inpoving Competency-Awen Aware Learning Capabilities).
Ethics Statement
For face reconstruction tasks and datascts, the diversity of race or ethnicity, gender, appearance, and actions is an important topic discuss ( (Wang el al., 2019; Zhu et al., 2022). Existing 3D face video datasets (Zielonka et al., 2022; Ranjan al., 2018; Cudeiro et al., 2019) typically have limited diversity regarding ethnicity, gender, appearance, and actions. Such 3D face datasets rarely provide video pairs, but with artificial facial markers attached to human faces and small set of identities. On the other hand, our NenFace-dataset mitigates such issues since our dataset is acquired on top of large-scale in-the-wild face video datasets, which typically rely on internet videos. Such video datasets are diverse in terms of ethnicity, gender, facial appearances, and actions when compared to the small/ medium--miuueeeeeeee 3D facial motion capture datasets. Since our dataset is acquired based on the existing public video datasets (Wang et al., 2020; Chung et al., 2018; Zhu et al., 2022), all the rights, licenses, and permissions follow the original datasets. Moreover, we will release the Neulace-dataset by providing the reconstructed 3DMM parameters without the actual facial video frames. NeuFace- dataset does not contain identity-specific metadata and facial texture maps. Nonetheless, per-identity shape coefficients can give a rough guide about human facial shape. Thus, we release our dataset for research purposes only.
Reproducibility Statement
We will make our code and data accessible to the public.
References
Manya V Afonso, José M Bioucas-Dias and Mário AT Figuciredo. An augmented lagrangian approach to the constr oplimization formulation of imaging inverse problems. IEEE Transactions on Image Processing (TIP), 20(3))681-693 695, 2010. 5
Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. A stochastic conditioning scheme for diverse human motion prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 11
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning (ICML), 2019. 4, 7, 24, 25, 26
Anh Tu Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. Regressing robust and discriminative 3D morphable models with very deep neural network. In IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), 2016. 3
Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojeiech Matusik, Mohamed Elgharib, and Christian Theobalt. Monocular reconstruction of neural face reflectance ficlds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Gwangbin Bae, Martin de La Gorce, Tadas Baltrušaitis, Charlie Hewitt, Dong Chen, Julien Valentin, Roberto Cipolla, and Jingjing Shen. Digifacc-Im: 1 million digital face images for face recognition. In IEEE Winter Conf. Applications of Compuler Vision (WACV), 2023.
Jan Bayer, Petr Čižck, and Jan Faigl. On construction of a reliable ground truth for evaluation of visual slam algorithms. In Conference on Planning in Artificial Intelligence and Roboties, 2016. 2
13

---

Published in Transactions on Machine Learning Research (08/2024)
V. Blanz and T. Vetter. Face recognition based fitting 3d morphable model. IEEE Transactions Pollern Analysis and Machine Intelligence (TPAMT), 25(9), 2003. 1
Volker Blanz and Thomas Vetter. A morphable model for the syuthesis of 3d faces. ACM Transactions on Graphics (SIGGRAPH), 1999.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. Keep it SMPL: Automatic estimation of 3D human posc and shape from a single image. In European Conference on Compuler Vision (ECCV), 2016. 3, 23
Arij Bouazizi, Ulrich Kressel, and Vasileios Belagiannis. Learning temporal 3d human pose estimation with pseudo-labels. In IEFE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2021. 2
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In IEEE International Conference on Computer Vision (ICCV), 2017. 5, 20
Chen Cao, Yanlin Weng, Stephen Lin, and Kun Zhou. 3d shape regression for real-time facial animation. ACM Transactions on Graphics (SIGGRAPH), 32(4). 2013. 23
Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. Real-time high-fidelity facial performance capture.
ACM Transactions on Graphics (SIGGRAPHI), 34(4). 2015. 23
Qiong Cao, Li Shen, Weidi Xic, Omkar M. Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face and Gesture Recognition, 2018. 2, 3
Yen-Lin Chen, Hsiang- Tao Wu, Fuhao Shi, Xin Tong, and Jinxiang Chai Accurate and robust 3d facial capture using single rgbd camera. In IEFE International Conference Computer Vision (ICCV) 2013.
3
Junhyeong Cho, Kim Youwang, and Tac-Hyun Oh. Cross-attention of disentangled modalitics for 3d human mesh recovery with transformers. In European Conference on Computer Vision (ECCV), 2022. 7, 12
Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body-driven attention. In Europeon Conference on Computer Vision (ECCV), 2020. 3
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. INTER- SPEECH, 2018. 2, 3, 7, 9, 12, 13, 20
Yaim Cooper. Global minima of overparameterized neural networks. SIAM Journal on Mathematics of Data Science, 2021.
Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, and Luisa Verdoliva. Id-reveal: Identity- aware deepfake video detection. In IEEE International Conference Computer Vision (ICCV), 2021.
Daniel Cudciro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. Capture, learning, and synthesis of 3D speaking styles. In IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), 2019. 2, 3, 7, 11, 13
Radek Danecek, Michael J. Black, and Timo Bolkart. EMOCA: Emotion driven monocular face capture and animation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 4, 5, 20, 22
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient descent finds global minima of deep neural networks. In International Conference Machine Learning (TCML), 2019a. 4, 26
14

---

Published in Transactions on Machine Learning Research (08/2024)
Simon Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. In International Conference on Machine Learning (ICML., 2019b. 4
Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 3d morphable face models models-past, present, and future. ACM Transactions on Graphics (SIGGRAPH), 39(5), 2020. 3
Mohamed Elgharib, Mohit Mendiratta, Justus Thies, Matthias Nießner, Hans- Peter Seidel, Ayush Tewari, Vladislav Golyanik, and Christian Theobalt. Egocentric videoconferencinng ACM Transactions on Gruphics (SIGGRAPH), 39(6), 2020.
Han EunGi, Oh Hyun-Bin, Kim Sung-Bine Bin, Corentin Nivelet Etcheberry, Suekyeong Nam, Janghoon Ju, and Tae-Hyun Oh. Enhancing speech-driven 3d facial animation with audio-visual guidance from lip reading expert. In INTERSPEECH, 2024.
Gabriele Fanelli, Juergen Gall, Harald Romsdorfer, Thibaut Weise, and Lue Van Gool. A 3-d andio-visual corpus of affective communication. IEEE Transactions on Multimedia, 12(6), 2010. 2
Qi Fang, Qing Shuai, Junting Dong, Hujun Bao, and Xiaowei Zhou. Reconstructing 3d human pose by watching humans in the mirror. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2
Haiwen Feng, Timo Bolkart, Joachim Tesch, Michael J. Black, and Victoria Abrevaya. Towards racially unbiased skin tone estimation via scene disambiguation. In European Conference Computer Vision (ECCV), 2022. 7
Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In European Conference Computer Vision (ECCV), 2018. 12
Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning animatable detailed 3D face model from in-the-win images. ACM Transactions on Graphics (SIGGRAPH), 40(8). 2021. 2, 4, 5, 6, 9, 10, 12, 20, 23, 24
Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stelanos Zafeiriou. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019. 3
Thilip-Willaan Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022. 4
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li. Towards fast, accurate and stable 3d dense face alignment. In European Conference on Computer Vision (ECCV), 2020. 12
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black. Resolving 3D human pose ambiguities with 3D scene constraints. In IEEE International Conference on Computer Vision (ICCV), 2019. 2
Martin Heusel, Hubert Ramsaner, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two timo-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 11
Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Black. Capturing and inferring dense full-body human-scene contact. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022. 2
Patrik Huber, Zhen-Hua Feng, William Christmas, Josef Kittler, and Matthias Rätsch. Fitting 3d morphable face models using local features. Tn IEFE International Conference on Image Processing (ICIP), 2015. 3
15

---

Published in Transactions on Machine Learning Research (08/2024)
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In IEFF International Conference on Computer Vision (ICCV), 2015. 2
Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodics. In IEEE Conference on Computer Vision and Pattern Recognátion (CVPR), 2018. 2
Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar fine-tuning for 3d human pose fitting towards wi-the-wild 3d human pose estimation. In International Conference on 3D Vision (3DV), 2020. 3, 4
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEFE Conference on Computer Vision and Pattern Recognátion (CVPR), 2019. 2,
Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. In European Conference on Computer Vision (ECCV), 2022. 7
Hycongwoo Kim, Pablo Garrido, Ayush Tewari, Weipong Xu, Justus Thics, Matthias Niessner, Patrick Pércz, Christian Richardi, Michael Zollhöfer, and Christian Theobalt. Deep video portraits. ACM Transactions on Graphics (SIGGRAPH), 37(4), 2018.
Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Daniilidis. Learning to reconstruct 3d human posc and shape via model-fitting in the loop. In IEEE International Conference on Computer Vision (ICCV), 2019. 12, 23
Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning. Tn IEEE International Conference on Computer Vision (ICCV), 2021. 2
Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics (SIGGRAPH Asia), 36(6), 2017. 3, 4, 6, 20, 23
Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In IEEE International Conference on Computer Vision (ICCV), 2021a. 7, 12
Kevin Lin, Lijuan Wang, and Zicheng Liu, End-to-end human pose and mesh reconstruction with transformers.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b. 7, 12
Jun Ling, Xu Tan, Liyang Chen, Rumnan Li, Yuchao Zhang, Sheng Zhao, and Li Song. Stableface: Analyzing and improving motion stability for talking face generation. arXiv preprint arXiv:2208. 137177 2022. 8
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou. Tang. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision (ICCV), 2015. 2,3
Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De Torre, and Yaser Sheikh. Pixel codec avatars. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2021.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. HowTo100M: Learning a Tixt-Vidco Embedding by Watching Hundred Million Narrated Vidco Clips. In IEEE International Conference on Compuler Vision (ICCV), 2019. 2
Lea Müller, Ahmed A. A. Osman, Siyu Tang, Chun-Hao P. Huang, and Michael J. Black. On self-contact and human pose. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: a large-scale speaker identification dataset. Tn INTERSPEECH, 2017. 2, 3
Arsha Nagrani, Paul Hongsuck Sco, Bryan Scybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In European Conference on Computer Vision (ECCV), 2022." 2
16

---

Published in Transactions on Machine Learning Research (08/2024)
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understand- ing the role of wver-parametrisation in generalization of neural networks. ar preprint ar Xin: 1805. 12076, 2018. 4
Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. In IEEE Conference on Compuler Vision and Pottern Recognition (CVPR), 2022. 2, 7, 11
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In British Machine Vision Conference (BMVC), 2015. 2, 3
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2019. 3, 23
P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter. A 3d face model for pose and illumination invariant face recognition. In Proceedings of the 6th IEEE International Conference on Advonced Video and Signal based Surveillance (AVSS) for Securily, Safely and Movilorrig in Smart Encironments, 2009. 3
Amit Raj, Michael Zollhoefer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, and Stephen Tombardi. Pva: Pixel-aligned volumetric avatars. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2020.
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. Generating 3D faces using convolutional mesh autoencoders. In European Conference on Computer Vision (ECCV), 2018. 2, 3, 13
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas. Humor: 3d human motion model for robust pose estimation. In IEEE International Conference on Computer Vision (ICCV), 2021. 10, 11
Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando de Torre, and Yaser Sheikh. Meshtalk: 3d face animation from speech using cross-modality disentanglement. In IEEE International Conference on Computer Vision (ICCV), 2021.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Face- forensics: A large-scale vidco datasct for forgery detection in human faces. arXiv preprint Xiv: 118880009977 2018.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics+++: Learning detect manipulated facial images. In IEEE International Conference on Computer Vision (ICCV), 2019.
Christos Sagonas, Epameinondas Antonakos, Georgios Teimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: Database and results. Image and Vision Computing (IMAVIS), 47, 2016. 9
Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Mathieu Salzmann. Continuous inference in graphical models with polynomial energies. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2013. 5
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2019a. 3, 12
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 7763 -772, 2019b. 12
17

---

Published in Transactions on Machine Learning Research (08/2024)
Jiaxiang Shang, Tianwei Shen, Shiwei Li, Lei Zhou, Mingmin Zhen, Tian Fang, and Long Quan. Self- supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency. In European Conference on Computer Vision (ECCV), 2020. 12
Kihyuk Sohn, Xinchen Yan, and Honglak Lee. Learning structured output representation using deep conditional generative models. In Advances in Newral Information Processing Systems (NeurIPS), 2015. 10
Kim Sung-Bin, Lee Chae-Yeon, Gihun Son, Oh Hyun-Bin, Janghoon Ju, Suekyeong Nam, and Tae-Hyun Oh. Multitalk: Enhancing 3d talking head generation across languages with multilingual video dataset. In INTERSPEECH, 2024a.
Kim Sung- Bin, Lee Hyun, Da Hye Hong, Suekyeong Nam, Janghoon Ju, and Tae-Hyun Oh. Laughtalk: Expressive 3d talking head generation with laughter. In IEEE Winter Conf. on Applications of Computer Vision (WACV), 2024b.
Ayush Tewari, Mohamed Elgharib, Mallikarjun BR, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt. Pie: Portrait image embedding for semantic control. ACM Transactions on Graphics (SIGGRAPH Asia), 39(6), 2020. 1
Justus Thics, M. Zollhöfer, M. Nicßner, L. Valgaerts, M. Stamminger, and C. Theobalt. Real-time expression transfer for facial reenactment. ACM Transactions on Gruphics (SIGGRAPH), 34(6), 2015. 1,
Justus Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016a. 1
Justus Thies, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2016b. 23
Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Niessner. Headon: Real-time reenactment of human portraït videos. ACM Transactions on Graphics (SIGGRAPH), 37(4), jul 2018. ISSN 0730-0301. doi: 10.145/3197751.32213300
Anh Than Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. Regressing robust and discriminative 3d morphable models with very deep neural network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 12
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: largo-seale audio-visual dataset for emotional talking-face generation. In European Conference on Computer Vision (ECCV), 2020. 2, 3, 5, 7, 9, 12, 13, 20
Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang. Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In IEEE International Conference on Computer Vision (ICCV), 2019. 2, 13
Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021. 1, 2
Benjamin Weyrauch, Bernd Hciscle, Jennifer Huang, and Volker Blanz. Component-based face recognition with 3d morphable models. In IEEE Conference Compuler Vision and Pallern Recognition Workshops
(CVPRW), 2004. 1
Erroll Wood, Tadas Baltrusaitis, Charlie Hewitt, Sebastian Dziadzio, Matthew Johnson, Virginia Estellers, Tom Cashman, and Jamie Shotton. Fake it till you make it: Face analysis in the wild using synthetic data alone. In IEEE International Conference on Computer Vision (ICCV), 2021. 1, 3
Erroll Wood, Tadas Baltrusaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavljevic, Daniel Wilde, Stephan Garbin, Chirag Raman, Jamic Shotton, Toby Sharp, Ivan Stojiljkovic, Tom Cashman, and Julien Valentin. 3d face reconstruction with dense landmarks. In European Conference on Computer Vision (FCCV), 2022. 1, 3, 6, 12
18

---

Published in Transactions on Machine Learning Research (08/2024)
Yu- Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming- Yu Liu, and Ting- Chun Wang. Learning to relight portrait images via virtual light stage and synthetic-to-real adaptation. ACM Transactions Graphics (SIGGRAPH), 2022. 1
Jae Shin Yoon, Zhixuan Yu, Jaesik Park, and Hyun Park. Humbi: A large multiview dataset of human body expressions and benchmark challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021. 2, 3
Kim Youwang, Kim Ji- Yoon, and Tao-Hyun Oh. Clip-actor: Text-driven recommendation and stylization for animating human meshes. In European Conference on Compuler Vision (ECCV), 2022. 1
Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll. Paint-it: Text-to-texlure ure synthesis via deep convolutional texture map optimization and rrsscaall-basee rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1,
Jun Zhang, The mean field theory in em procedures for blind markov random field image restoration. 1EEE Transactions on Image Processing (TIP), 2(1):27 40, 1993. 5
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In European Conference on Compuler Vision (ECCV), 2022. 2, 3, 7, 9, 12, 13, 20
Wojeiech Zielonka, Timo Bolkart, and Justus Thies. Towards metrical reconstruction of human faces. In European Conference Computer Vision (ECCV), 2022. 2, 3, 12, 13, 23
Michael Zollhöfer, Justus Thies, Darek Bradley, Pablo Garrido, Thabo Beeler, Patrick Péerez, Marc Stam- minger, Matthias Nießner, and Christian Theobalt. State of the art on monocular 3d face reconstruction, tracking, and applications. Computer Grophics Forum, 2018. 3
19

---

Published in Transactions on Machine Learning Research (08/2024)
Appendix
We present additional details, analysis, results, and experiments that are not included in the main paper due to the space limit. Also, the attached video explains and demonstrates the main idea of NeuFace and video samples for the NeuFace-dataset.
A Implementation details
We provide detailed configurations for implementations and experiments in the main paper.
NeuFace optimization details. NeuFace optimization is composed of a neural network &w and the optimizing part. For the network iww we use a pre-trained DECA (Feng et al., 2021) or a pre-trained EMOCA (Danecek et al., 2022) encoder network. Overall optimization takes about 8 min. for 7 views, 1-2 frames of videos, and about 2.5 min. for view, 122 frames of videos.
NeuFace-dataset acquisition. We provide reliable 3D face mesh annotations for large-scale face video datasets: MEAD (Wang et al., 2020), VoxCeleb2 (Chung et al., 2018), and CelebV-HQ (Zhu et al., 2022). We optimize our full objective (Bq. (1)) to acquire FLAME meshes for the datasets with a multi-view camera setup, e.g., MEAD. Otherwise, we optimize (Eq. (1)) with Aview=0. We automatically discard the sequences if the optimization yields out-of-distributiion shape parameters, i.e,, the L2-norm of shape parameters deviates largely from the pre-built distribution (Li et al., 2017), ||30,>>1.0, or if the 2D landmark detector (Bulat & Trimiropoulss 2017) fails to capture the faces. After subsequent human verification, the NeuFace-dataset achieves <0.1% of failure rate on upon criteria, supporting the reliable quality of our dataset.
NeuFace dataset includes 24 diverse cmotion/cxpression categories. Compared to indoor motion capture datasets (BIWI 3D, COMA, VOCASET), our dalaset has greater variely of imotion//xpresssi categories. Moreover, our dataset contains 3D faces in natural talking scenarios, resulting in higher diversity within these categories. Neul'ace-dttaset contains at least 40 different facial attributes, including hairstyles and accessories such as glasses. While traditional motion capture datasots do not disclose the number of facial attributes, their facial attribute diversity is incomparably limited since they contain a small number of identities, and indoor environments. Unlike traditional motion capture datasets, which are recorded in controlled indoor environments with fixed camera views and lighting, the NeuFacc-dataset uses in-the-wild YouTube vidcos, providing a much greater diversity of backgrounds.
Facial motion prior. As one of our datasel's 's prominent applications, we introduced the learning of 3D facial motion prior, called HuMoR-Face (Sec.5.1 in the main paper). We first pre-process 3D face meshes in the NeulPace-dataset. We compute root orientation, face pose angles, 3D landmark positions, and their velocities, respectively. Then, we represent the state of a moving human face as x (Φ. ~ 0 J,J where Φ, & denotes head root orientation and its velocity, 0 denotes the FLAME face pose parameters, and J, J denotes facial joint and its velocity, respectively.
The generative facial motion prior is trained to predict the facial motion state, X(+1, given the current state X, as a condition. We consider the NeuFace holdout test split as the real motion distribution and compute the FD for the generated motions. We do not consider VOCASET as the rcal motion distribution for computing FD. is limited in diversity and naturalness, which contradicts FD's purpose of measuring the naturalness of generated motions. Our NeuFace holdout test split much larger and more diverse than VOCASET.
Fine-tuning face mesh regressor. As our dataset's another application, we improve the accuracy of the pre-trained DECA model with our NeuFaco-dataset and its 3D annotations. Specifically, we finc-tune the pre-trained DECA parameters with our NeuFace-dalasel and the auxiliary 3D supervisions proposed in the main paper 79795-797). During fine-tuning, we use an adjusted learning rate, 1 x 10-5, which is ten times smaller than training DECA from scratch. Note that there exist the DECA-coarse model and the DECA-detail model. Unfortunately, there are known issues in reproducing detAl due to the absence of VGGFace2 and the training recipe (DECA GitHub issues: /3jj//,,,, bit.ly /3HHVV))..
20

---

Published in Transactions on Machine Learning Research (08/2024)
Table S1: Average vs. Median operation. we conduct an ablation study on a subset the MEAD dataset, using a median operation instead of the average for the temporal consistency loss.
Loss config.
CVD + NME
Temporal loss (Average)
0.0981
3.03
Temporal loss (Median)
0.0985
3.09
Table S2: Ablation the design choices. (a) Optimizing photometric loss (Ephoto) with the NeuFace optimization results in performance degradation. (b) The identity code (ß) gradually converges over the iterations, although we do not manually force shared identity regularization.
Loss config.
CVD 4
MSI f
MSI St
NMFJ
31
NeuFace
0.103
0.206
0.305
2.58
NeuFace+22 phott
0.106
0.205
0.299
2.62
NeuFace-55 photo
0.112
0.195
0.282
3.69
Iterations
(a) Ablation on the photometric loss
(b) Std. of B over the iterations
B Ablation on the design choices
In our proposed NeuFace optimization, the temporal consistency loss employs a temporal moving average to estimate latent target meshes that represent temporally smooth heads. To assess the effectiveness of the temporal moving average operation, we conduct ablation study using a median operation instead of the average subset of the MEAD dataset. In Table S1, both methods temporal moving average and median operation in the temporal Hindow-ddmonstee no significant. difference in the optimization results. This finding suggests that the choice of operation has a negligible impact on the optimization process.
comparable performance in terms of NME and CVD, with
Although our proposed NeuFace optimization utilizes a simple combination of losses, we find that it is surprisingly effective for constructing accurate pseudo 3D face data. For example, we observe that the use of photometric loss (Ephoto) has negligible effect, as demonstrated in Table $2 (n). In fact, Lphoto tends to degrade performance, with larger values of this loss leading to greater degradation. We postulate that this might be due to the impact of shlfsshadows and non Lambertian reflection caused by lighting and noise in video data, which could interfere with robust optimization. Interestingly, we also reveal that without explicit rogularization for shared identities across frames in vidcos, the identity codes tend to converge automatically, as shown in Table $2 (b). This finding suggests that adding shared identity regularization may be unnecessary. Despite the effectiveness of our proposed NeuFace optimization, we believe that exploring additional losses further refine the optimization process could be worthwhile future dircction.
We proposed an EM-style optimization for our NeuFace optimization. In each optimization step I, we compute latent target variables M and a' at E-step using multi-view bootstrapping or temporal smoothing. The latent target variables serve as the supervision for spatio-temporal consistency losses at M-step. Our EM-style losses are as follows (Eq. (3) and Eq. (4) in the main):
Ltemporal,our Eyeweell dll Lmultivlew,ours
Nr.Ny M -M
where J, Nr, and Ny denote the temporal frame index, view index, total number of frames, and total number views, respectively. Also, M, q denotes the mesh vertices and head rotation quaternion, respectively. Please refer Soc. 3.2 in the main draft for more details.
The naive, non EM-style way to implement optimization as follows:
21

---

Published in Transactions on Machine Learning Research (08/2024)
1. Ltemporal.naive Iterate through all the head rotation quaternion and compute frame-wise, Markov chain style head rotation error, without introducing latent larget variable.
2. Luuultiview,aaive Iterate through all the multi-view meshes in the same time frame and compute view-wise vertex error, without introducing latent target variable.
More formally, the losses for non EM-style optimization are as follows:
Nv N
Ns Nv-1 Ny
Lueiporal,naive Σ
Σ
a - 92t1,112: Lmultiview,usive
M -M, M2
We conduct an ablation experiment that compares each optimization performance on a subset the MEAD dataset. In Table S3, our EM-style optimization achieves better mesh quality in terms of temporal smoothness, cross-view vertex consistency, and landmark accuracy compared to the non EM-style optimization baseline.
Table S3: Ablation on EM-stylc optimization
Optim. style
MSI 3D f
MSI 3D f
CVD +
NME +
Non FM--ttll optim.
0.208 0.284
0.333
0.233
4.26
EM-style optim. (ours)
0.462
0.201
3.36
C NeuFace optimization with EMOCA
Recall that we can replace the neural parameterization face meshes with another neural model. Specifically, we use EMOCA (Danecek et al., 2022), which is built upon DECA with an additional expression encoder. We change the neural network from DECA EMOCA and optimize over it with our spatio-temporal and landmark losses. In Table 2 and Sec. 4 in the main paper, we discussed about the quantitative quality of Eeutacc-EEdattset.
Input
U
SUS
EMOCA
RS
C
NeuFace-E
Down
Front
L-30
L-60
R-30
R-60
Top
Figure S1: Multi-view consistency: EMOCA vs. NeuFace-E. We visualize the meshes obtained by EMOCA (Danecek et al., 2022) and NeuFace-E. E. By aligning the meshes to face the same direction, we can clearly notice that NeuFaco-E obtains multi-view consistent meshes compared to EMOCA.
Qualitatively, we visualize the rendered meshes over different views as in Fig. S1. NeuFace-E-datlsel (3rd row) contains multi-view consistent meshes over views compared to the meshes obtained by EMOCA inference (2nd row). Specifically, EMOCA produces huge discropancies mouth area across the views, while our method is more consistent. Moreover, our method reconstructs more accurate meshes, especially for the shape of the nose and face contour.
22

---

Published in Transactions on Machine Learning Research (08/2024)
D NeuFace vs. Video 3D Face Tracking Method
We verify the favorable quality of NeuFace-dataases by comparing it with the meshes obtained by the state-of- the-art method, MICA with a tracker (Zielonka et al., 2022) (MICA+T), among the existing methods (Cao et al., 2013; 2015; Thies et al., 2016b; Zielonka et al., 2022). MICA+T jointly optimizes 3DMM, cameras, and textures with landmark and photometric losses, and statistic regularizers. In Table $4, NeuFace performs better than MICA+T in CVD & NME with comparable MSI. Also, faster optimization makes NeuFace preferable when annotating large-scale videos.
Table S4: Evaluation MEAD: MICA vs. NeuFace
CVD L MSIa f
NME +
Optim. time (7 views)
MICA+T NeuFace-D-dataset
0.049
0.349
2.98
60 min.
0.0094
0.277
2.58
8 min.
E
Analysis on NeuFace
In this section, introduce and validate design choices for NeuFace optimization, through analysis. Specifically, we build a strong baseline and support our choice of "re-parameterized" face mesh optimization method for NeuFace in Sec. E.1. Next, we provide a proof sketch of the provable global minima convergence of NeuFace optimization in Sec. E.2.
E.1 FLAME fitting vs. NeuFace optimization
Recall that NeuFace re-parameterizes the 3DMM, i.e., FLAME (Li et al., 2017) to the neural parameters (represented as DECA (Feng et al., 2021)), then optimizes over them to obtain accurate 3D face meshes for videos. Following the prior arts in the parametric human body reconstruction literat (Bogo al., 2016; Pavlakos al., 2019; Kolotouros et al., 2019), there exists simple method optimize the parametric model; 3DMM parameter fitting. Thus, we implement FLAME fitting as solid bascline and compare the quantitative and qualitative results with NeuFace optimization to analyze and support our choice of neural re-parameterization.
Details baseline FLAME fitting. Given the initial FLAME and camera parameters, [Ob. [bb.Pb] = [rb, Ob, Pb, vh Ph], we implement the direct FLAME optimization as:
10 Pbl arg min LaD + Atemp Ctemprra + Aview Emultiview + XrCr AoLo + Xaha + Ao O..Pb
where the losses LZD, Liemporal, and Lmultiview are identical to the losses discussed in the main paper (Eqs. 2,3,4). We can obtain the initial FLAME parameters for the optimization in two ways: (1) initialize from mean parameters and (2) initialize from pre-trained DECA (Feng et al., 2021) predictions. We empirically found that initialization with mean FLAME parameters frequently fails when the input images contain extreme head poses. Thus, we choose to initialize FLAME parameters from the pre-trained DECA predictions, thus providing a plausible initialization for a fair comparison. Also, following the convention (Li et al., 2017), we optimize FLAME paramoters in coarso-to-fine manner. For the carlier stage, we fix FLAME parameters that control local details, i.e., Ob, Bb, and bb,, and optimize the global head orientation, Tb, and camera parameters Рb. Then we fix camera parameters and optimize other FLAME parameters jointly at later stage fit the local details.
Since we initialize FLAME parameters and camera parameters from the pre-trained DECA predictions, i.e., initial [0,, Pb] in Eq. (5). Accordingly, the meshes obtained by the FLAME fitting achieve better spatio-temporal consistency and 2D landmark accuracy than the meshes obtained by a pre-trained DECA without any post-processing.
Qualitative result. In Fig. S2(a), NeuFace-dataset contains much expressive and imago-aligned meshes, e.g., wrinkles and face boundaries. On the other hand, the meshes obtained by the direct FLAME optimization show
23

---

Published in Transactions on Machine Learning Research (08/2024)
Front
L-30
Input
FLAME fitting (baseline)
NeuFace optim. (ours)
Input
FLAME fitting (baseline)
NeuFace optim. (aurs)
(a)
(b)
Figure S2: FLAME fitting vs. NcuFace optimization. (a) NeuFace optimization obtains much
expressive and pixel-level aligned meshes than the baseline FLAME fitting. (b) We observe NeuFace
optimization induces richer and data-dependent gradients compared the sparse gradients of baseline.
mean biasee 3D faces. We explain such results in terms the inteeiidddeenencyy of the direct FLAME optimization. The baseline method requires several regularization terms based on the prior, pre-built from external 3D datasets, which is intteiindeennentt (Eq. (5)). Such data-independent regularization encourages the optimized FLAME parameters to stay close to the mean of parameter distributions, regardless of the facial characteristics on input images. Balancing such regularization terms with other losses is cumbersome and prone to obtain mean shape faces.
In contrast, recall that NeuFace ro-parameterizes the FLAME parameters as the pre-trained neural network, such as DECA (Feng et al., 2021). Such re-paramotorization allows NeuFace to update face meshes in an input- image-conditioned manner, called data-dependent mesh update (Sec. 3.3 the main paper). Figure S2(b) visualizes the data-dependent gradient of our optimization. While the baseline shows similar gradient patterns throughout three imput images, NeuFace optimization produces diverse gradient maps according to input images. Such rich and data-dependent gradients consider high-dimensional visual features induced from the input RGB values in the optimization process, yielding accurate and expressive 3D faces.
E.2 Proposition 1: Convergence Property to Global Minima
Proposition in this work is straightforward variation of the main result of Allen-Zhu et al. (Allen- Zhu et al., 2019). Before providing the proof sketch of Proposition 1. We describe the assumptions needed to prove Proposition 1. For simplicity, we consider a simple 12-based regression loss and a L- layer fully connected ReLU network o (.) having a uniform weight width size of m. For the training data consisting vector pairs {<,, yf hienni, the network has the batched input of x=(==]s(m), where n is the batch size, x;E(0, 1]din, and of E Rdoue, the output of क (x) E Rdour and the weights W W.ERRxxd W (2.-. Ee.... WLERdowemy
Assumption 1. Without loss of generality, Vi, ||=,|| = and || < O(1)
Assumption 2. The pretrained neural network weights W are assumed to be started from the val- ues distributed normally, i.e., considered as sample instance from a Gaussian distribution. Specifically, L- 1} and [W 1t,5 ~ N(0,
(w("),, Thy ~ (00, 2/ 2/row[W:]) for I the operulor row[-] relurns the row size of the inpul mabria.
for every (i.j), where
Assumption 2 appears to be restrictive by those standard deviations, but it is not. The assumptions cover a fairly broad range of weight distribution scenarios. For larger standard deviations, we can always set a small norm for w's Assumption without loss of generality, and vice versa.
We consider that the biases are included (W) without loss of generality.
24

---

Published in Transactions on Machine Learning Research (08/2024)
Under these assumptions, we restate Proposition in the main paper.
Proposition 1 (Global Convergenec). For any c E (0, de (0. (0,)) given on input data fx,y} and the newral network & ww() (-) over puee-parameterized such that m 2 Ω (poly(n, L, 8 d-1) dout), consider optimizing the non-conwer regression problem: arg min L(W)
yl
Under the above assumptions, with high probability, the gradient descent algorithm with the learning rate p p= θ pely(n,Lim) finds a point W such that L(W") S e in polynomial time.
Proof sketch. We first introduce the following useful leminas needed to prove the proposition.
Lemma 1 (Theorem 3 in (Allen-Zhu et al., 2019)). With probability at least 1 - e 20tt/mmmmm///../m.m L,5 11) it satisfies for every & e [4], every E [n], and every W with ||W w 1l2 S po where [ maxte[r] [Will,
dn? S WI so ((W)).
5m
Lnm d
Ω ((W)
This lemma suggests that, when we are close to the starting point W(O) of the neural network, there is no saddle point or critical point of any order. Specifically, for example, given fixed 6, 6, n and L, when we have the same error L for two different neural networks with respective widths of my and m2, where my < m2, then the lower bound of the gradient with my is larger than that of my with a better chance. This means that neural networks with larger widths are likely to have a lower chance of local minima.
This hints that any local search (e.g-, gradient descent) does not suffer from any local minima saddle points for larger m, which implics a more likely chance of avoiding local minima, i.c., finding global minima. However, the local search does not guarantee to decrease the loss function yet.
With the favorable property of Lemma 1, if we have an additional guarantee of loss decrease with gradient descent, we can prove the convergence to global minima. To derive objective-dcrcass guarantec in optimization theory, a notion of smoothness is typically needed; thus, we introduce the following lemma..
Lemma (Theorem 4 of (Allen-Zhu al., 2019)). With probability at least e -Q(m/ppoyyLLooogm)) m)), we have:
for every Wi E (R with ||W* w S mj]
and for every W' E (Rmxm)L with [ poly(L.lggm) the following inequality holds
L
((W1) (VZ(W)) W
0
0 m12
poly(L) nm log
| 1l2 Wl
Vd
This lemma states the semi-smoothness property of the objective function L w.r.t. & (+) to take into account non-smoothness introduced by ReLU activation in (-). The somi-smoothness looks similar to the Lipschitz
smoothness except for the first order term || [W' Interostingly, when we increase m, the increasing rate of the first order term is much slower than that of the second order lerm; thus, the second order term becomes dominant compared to the first order one, and the smothhesot approaches closer to the Lipschitz smoothness. This means that the neural network is smoother as m goes larger.
Under the assumption that W - W0 is small (will be verified later), the next step is to combine Lemma 2 with gradient descent to derive the loss-decrease guarantee. Denoting V. VL(W"(), the gradient descent update rule is defined as: (1++1)= = W V for a learning rate > 0. Then, from Lemma 2, putting W wi + W and W(O) Wi, i.e., W = - we have
C( ((W(+)) < ((W()) V C 2 pC V. 11 ((W)))
(where C1 = =(( C2
poly L) y nm log
Vd
S ((W1)) -lIV (0CC um PC2v 0 m
(mm
((W())
25

---

Published in Transactions on Machine Learning Research (08/2024)
(I <
max E
Vw
< 0 pm L(W(0) from the upper bound in Lemma 1)
(1+ (2)C3 ( V
(where C3 = ppp00 pu
mm
1 (1-
pôm dn?
L(W1))
(by the gradient lower bound from Lemma and our choice of p, e.g., p = O ==+
When we choose the parameters such that there exists T > such that ((W(T) S c. Examples of convenient paramcter choices of m, P, and T in polynomial orders are suggested in Allen-Zhu et al. (2019) to hold Ω
E(0, 1), we have & ( (1). In other words, dn
pórn
(0,1) and the small value
dn²
of [WW0-Wo00 Wr for every I. This concludes the proof sketch of finding point W -W() such that C( S..
Remark 1: Global optimality, In Proposition 1, set arbitrarily small. With very small e, it suggests that a converged point W is a global minimum.
Remark 2:
The radius condition between the initial weights and updated one. The spectral radius bounds for |[ - W || required in Lemmas and 1 appear to be small, it is sufficiently large enough to completely change the output of the model, considering the large width of size m and the standard of weight entries in Assumption 2.
deviation Vm
Remark 3: Other architectures. Allen- et al. (2019); Du et al. (2019a) present the recipes to convert the L-layer fully connected networks to convolutional neural works and to ResNet by sacrificing the complexity of proof. Thus, the conclusion of the provable guarantee does not change with such architectural changes. Thus, the architectures experimented provably comply with the conclusion of Proposition up to the choice of the parameters, i.e., global convergence.
Remark 4: Other losses. In the above proof sketch, one of the important pieces is the semi-smoothness in Lemma 2. While we discuss only with the simple l2 regression loss function, fortunately, the seothees already encompasses any choice of Lipschitz smooth cases for the loss functions. Thus, as long as the choice of the loss function Lipschitz smooth, the replacement of the loss function does not alter the conclusion of Proposition 1 even for non-convex losses except. the choice of parameters. This hints that our choice of the multi-task loss in Eq. (1) provably complies with the conclusion of Proposition 1 except the choice of the parameters.
Remark 5: L vs. m for the over-parameterization. For designing the over-parameterized architocture, one can control two different. paramelers L and m.
Obviously, the high probability is achieved with larger m rather than L, but more importantly, the local minima smoothing phenomenon suggested in the lower bound of Lemma 1 is independent to L.
26

---

Published in Transactions on Machine Learning Research (08/2024)
F More dataset samples
We present more qualitative samples of NeuFace-dataset, with diverse identities and visual features (see Fig. S3). Since we cannot deliver expressive and temporally smooth facial motion in images, we strongly recommend seeing video visualizations for NeuFace-datasel, the supplementary videos.
CAIS
ELLER
ELLER
WIITH
Leina salammeta con was
Figure S3: NeuFace-dataset is the large-scale face video dataset containing 3DMM annotations for faces with diverse ethnicity, gender, emotions, and actions. See supplementary videos for the dynamic face visualizations.
27


Reviews:
[{"review_id": "3DnZ3BasYl", "reviewer": "Reviewer_D2De", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written and easy to follow. The supplementary video gives a good overview of the work, and the appendix contains a lot of helpful details such as runtime, ablation studies and additional quantitative results.\n- Optimizing w.r.t. network parameters rather than is well motivated through (1) the provided intuition of taking advantage of input image information rather than relying entirely on 2D landmarks (2) the mathematical analysis (3) the visualizations of gradients to non-landmark parts of the face meshes. It's encouraging to see the gradients to update regions like the cheeks which do not traditionally have 2D landmarks but are important for high quality face meshes.\n- When working with video/multi-view it makes sense to have additional loss terms for temporal/multi-view consistency. The ablations in the appendix demonstrate the effectiveness of these additional losses.\n- Quantitative results demonstrate that NeuFace can generate higher quality 3D pseudo labels than off-the-shelf regressors or direct morphable model parameter optimization.\n\nWeaknesses\n- While the face motion generation results are encouraging, the value of the proposed approach and resulting datasets would be even more clear if this data can be used to obtain state of the art face mesh regression performance. The results in Table 3 show a slight improvement when DECA is trained on the new data, but this is not state of the art performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents NeuFace: a new approach for automated labeling of 3D face images with 3D face meshes. Generally, morphable parametric 3D face models such as FLAME can directly be optimized to 2D face landmarks. Instead of this, NeuFace proposes to optimize w.r.t the weights of a neural network that's trained to predict FLAME parameters from an image, rather than the parameters themselves. The paper provides analysis and quantitative evidence for the effectiveness of doing this w.r.t. neural network weights.\n\nIn addition, the paper proposes loss terms for temporal smoothness and multi-view consistency, improving the performance in multi-view and video scenarios. NeuFace is used to generate pseudo-ground truth 3D meshes for large scale 2D video datasets and is shown to improve the performance of the DECA face mesh regressor and allows for training models for long-term 3D face motion generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The draft is in good shape. I would encourage the authors to include the ablations and quantitative results for motion generation in the main draft rather than in the appendix.", "broader_impact_concerns": "The current ethics statement is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zo6lcFj7LU", "reviewer": "Reviewer_BzRa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important problem - the lack of a face video dataset with ground truth 3D geometry. The motivation for this work is valuable.\n\nThe proposed NeuFace optimization scheme leverages temporal and spatial face data to improve face model parameter estimation. This helps alleviate issues like temporal jittering and multi-view inconsistencies, leading to good results in these aspects.\n\nThe authors demonstrate the applications of such a dataset well.\n\nHowever, the paper's claim of \"re-parameterizing\" the FLAME parameters is questionable. The authors appear to simply use a network to predict the original FLAME parameters from input images, which is not technically a re-parameterization.\n\nThe arguments made for why using a network to predict model parameters is better than direct regression are not fully convincing. The gradient of the loss function in Equation 6 does contain information from the input image, which means it is a form of \"data-dependent\" .\n\nThere are also concerns about the evaluation methodology. It is challenging to accurately evaluate 3D mesh accuracy in the dataset, and the proposed methods do not seem to be the best performing in the reported results (Table 3). This raises the question of whether this is the best approach for generating such a dataset, or if alternative methods like RGB-D face video capture could provide more reliable ground truth 3D geometry.\n\nOverall, the paper addresses an important problem, but there are some technical and evaluation-related issues that need to be addressed further.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduce a 3D face video dataset with pseudo ground-truth meshes, built on existing datasets.  To obtain the accurate face meshes,  the authors finetune a pre-trained  DECA / EMOCA  network with the proposed temporal and multi-vew consistency regularization.  Then the meshes are generated from face videos.   The proposed dataset can benefit the face reconstruction and face motion generation tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors are suggested to revise the methodolgy based on the weaknesses part.\n\nMore experiments and disscusions about data validation are  highly recommanded.\n\nThere is a duplicated sentence in page 5:  \"We set the vertices as invisible\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W3BwKKlC7V", "reviewer": "Reviewer_HkHi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- The overall annotation system is constructed based on prior works, including FLAME, DECA, and prior common regularization terms and losses. The temporal consistency loss and Multi-view consistency loss regularization terms are considered to improve the spatial-temporal consistency. Although this annotation system is mainly built on top of prior techniques, the introduced NeuFace optimization pipeline and the constructed dataset could be useful for academia and industry. \n\n- NeuFace optimization could be a good choice for making pseudo annotation considering its effectiveness and efficiency evaluated in the supplementary materials.\n\n- Fig. 3 is nice to visualize the Data-dependent gradient, which is the main motivation of the proposed NeuFace optimization method. It is explained that the over-parameterization method can achieve better global convergence. The proof sketch is also provided in the supplementary materials.  Fig.4 further proves the superiority of  NeuFace optimization over the prior methods.\n\n- NeuFace-dataset achieves about 1,000 times larger number of sequences than existing facial motion capture datasets. Especially, VoxCeleb2 and CelebV-HQ are two datasets captured in-the-wild setting. \n\nCons:\n\n- It is claimed that NeuFace-dataset is the largest 3D face mesh pseudo-labeled dataset in terms of the scale, naturalness, and diversity of facial attributes, emotions, and backgrounds.  However, the detailed statistics are not reported. I believe such statistics could make this paper more comprehensive.\n\n\n- It is claimed that the Expectation-Maximization (EM) optimization style is chosen to ease the loss optimization. More ablations or explanations could be illustrated to provide more insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a robust 3D face mesh pseudo annotation pipeline - NeuFace  - to annotate the 3D face mesh from 2D video. Based on this pipeline and extra manual verification, the largest 3D face mesh pseudo-labeled dataset - NeuFace-dataset - is constructed including the data captured in the wild setting. Furthermore, the performance improvement from this dataset has been demonstrated in two 3D face tasks.   Considering that the data-driven recipe is the most popular way in CV tasks, I believe this dataset could be helpful for 3D face tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The detailed statistics on the dataset should be reported.\nMore ablations or explanations on the EM optimization choice could be illustrated to provide more insights.", "broader_impact_concerns": "Not necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers were positive on the paper after the revision:
- "This work presents a novel approach for automated labelling of 3D face meshes. The key observation is to optimize with respect to network parameters rather than 2D projections of 3D landmarks, and the effectiveness of this is clearly demonstrated. The revised manuscript places an emphasis on video rather than static images, clarifying the contributions of this work."
- "Thanks for the clarification. The author's response has resolved most of my questions, and this paper attempts to address a valuable task."
- "The technical contribution of the proposed NeuFace and the dataset contribution can help to improve the 3D face model estimation. The gradient analysis is also insightful."

The paper makes a good contribution to the area of 3D face processing and analysis.
Decision: Accept (poster)