Expressive Gaussian Human Avatars from Monocular RGB Video
Authors: Hezhen Hu
Zhiwen Fan
Tianhao Walter Wu
Yihan Xi
Seoyoung Lee
Georgios Pavlakos
Zhangyang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: expressiveness
human avatar
monocular RGB video
Primary area: machine_vision

Abstract: Nuanced expressiveness, especially through detailed hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations.
In this work, we aim to learn expressive human avatars from a monocular RGB video; a setting that introduces new challenges in capturing and animating fine-grained details.
To this end, we introduce EVA, a drivable human model that can recover fine details based on 3D Gaussians and an expressive parametric human model, SMPL-X.
Focused on enhancing expressiveness, our work makes three key contributions.
First, we highlight the importance of aligning the SMPL-X model with the video frames for effective avatar learning.
Recognizing the limitations of current methods for estimating SMPL-X parameters from in-the-wild videos, we introduce a reconstruction module that significantly improves the image-model alignment.
Second, we propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds to accommodate the varied granularity across body parts.
Third, we develop a feedback mechanism that predicts per-pixel confidence to better guide the optimization of 3D Gaussians.
Extensive experiments on two benchmarks demonstrate the superiority of our approach both quantitatively and qualitatively, especially on the fine-grained hand and facial details. 
We make our code available at the project website: https://evahuman.github.io.

Expressive Gaussian Human Avatars from Monocular RGB Video
Hezhen Hu'
Zhiwen Fan'
Tianhao Wu² Yihan Xi'
Seoyoung Lee'
Georgios Pavlakos'
Zhangyang Wang!
1 University of Texas at Austin
2 University of Cambridge
Abstract
Nuanced expressiveness, especially through detailed hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations. In this work, we aim to learn expressive human avatars from monocular RGB video; setting that introduces grained details. To this end, we introduce EVA, a drivable human model that can recover fine details based on 3D Gaussians and an expressive parametric human model, SMPL-X. Focused on enhancing expressiveness, our work makes three key contributions. First, highlight the importance of aligning the SMPL-X model with the video frames for effective avatar learning. Recognizing the limitations of current methods for estimating SMPL-X parameters from in-the-wild videos, we introduce a reconstruction module that significantly improves the image-model model alignment. Second, we propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds to accommodate the varied granularity across body parts. Third, we develop a feedback mechanism that predicts per-pixel confidence to better guide the optimization of 3D) Gaussians. Extensive experiments on two benchmarks demonstrate the superiority of our approach both quantitatively and qualitatively, especially on the fine-grained hand and facial details. We make our code available the project website: https: //evahuman, .aiaaannntttbubu. th.
challenges in capturing and animating fine-
Introduction
High-quality digital avatar modeling has a wide range of applications, including AR/VR, movie production, sign language, and more. For digital human representation, capturing nuanced expressions essential for enhancing fidelity and vitality. This is particularly evident in the detailed portrayal of hands and facial expressions, which add emotional depth and interactive expression capabilities to human avatars. In this work, we investigate expressiveness when building human avatars from monocular video. The task involves taking as input a monocular human video and learning an animated human avatar which enables multiple capabilities, such as novel view and pose synthesis.
Reconstructing expressive human avatars is challenging, particularly due to the subtle and complex movements of the hands and face. Compared to the body, hands and faces occupy smaller spatial areas and have distinct characteristics. For example, the hand has many degrees of freedom, intricate textures, and frequent celf-occelusoos. To achieve accurate avatar modeling. it is crucial capture these fine textures from RGB video and ensure effective animation.
Current studies [49, 31, 15, 22, 12] mainly focus learning human avatar the body region and have made remarkable progress. Early works [49, 31, 155 mainly utilize NeRF as an implicit representation but usually have the drawback of low training/inference speed. Recently, more and more works build on top of 3D Gaussian Splatting [19] for its effectiveness and efficiency, which could further speed up rendering to over 100fps. GART [22] utilizes a mixture of moving 3D
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SOTA Method
EVA
Figure 1:
Learning expressive Gaussian human avatars from RGB video. Given a monocular video as input, our proposed approach, EVA, learns an expressive 3D Gaussian avatar. The results outperform the SOTA method [22] on novel pose synthesis, especially for the hand and facial details.
Gaussians approximate human geometry and appearance and enhances fine details with learnable forward skinning and latent bones. GauHuman [12] proposes new density control strategy, e.g. split and clone with KI. divergence and a new merge operation. However, these methods do not consider the fine-grained details of hand and face, which cannot meet the requirement of expressiveness.
In this work, introduce EVA, drivable human model that can represent expressive details using 3D Gaussians and human parametric model. Given a monocular RGB video, we extract the pose and mask information corresponding each frame, which allows to map each frame's 's human observation to a canonical space. Once the avatar is constructed, it can be animated using linear blend skinning given a new pose. followed by rendering the 2D human image. To tackle new challenges introduced by expressiveness, we start by solving the misalignment issue between the SMPL--X model and the RGB frames via reconstruction module. By employing a fitting-based optimization, this module can produce more reliable 3D SMPL-X reconstruction, providing more robust foundation for digital avatar modeling.
Considering the granularity differences across different body parts, we propose a context-aware adap- tive density control strategy for 3D Gaussian optimization. It leverages attributes specific different parts and historical gradient information, to adaptively control Gaussian density. Furthermore, to improve the Gaussian optimization, design feedback mechanism, which adaptively predicts confidence scores based on the rendering result, thereby ensuring the supervision signal effectively transferring the corresponding Gaussian. For evaluation on expressiveness, we build comparison baselines from related body avatar methods with a few modifications and collect new benchmark called UPB containing in-the-win wild upper body videos. The image quality is evaluated with multiple metrics, separately on the full body, as well as the hand and face regions. Our approach, as shown in Figure 1, largely outperforms previous state of the art, with more fidelity in fine-graid -raiined details.
Our contributions are summarized as follows:
We introduce EVA, approach that can build expressive human avatars based on 3D Gaussians, given as input a monocular RGB video. Through extensive experiments on two datasets, we demonstrate the superiority of EVA, particularly the hand and facial details.

---

To enhance expressiveness, we propose a context-aware adaptive density control strategy to accommodate the granularity differences across human parts, followed by a feedback mechanism to better guide the 3D Gaussian optimization.
To handle challenging in-the-wild videos, propose reconstruction module, which critically improves the SMPL-X alignment to the RGB frames compared to off-the-shelf methods. We demonstrate the importance of this improvement in recovering accurate avatars.
2
Related Work
2.1 Human Avatar Modeling from Monocular RGB Video
The problem of building human avatar from a monocular RGB video is challenging due to the dynamic nature of the capture and the partial observations for each frame. Early works [31, 2. 37, 38, 3, 15, 49, 46, 39, 42, 6, 21, 41] mainly resort to the combination of implicit neural representations (e.g., NeRF [27]) and parametric models to represent a human avatar with high fidelity and flexibility, To address the slow computation in implicit models, previous work has proposed various techniques to reduce training [10, 14] or inference time [33, 25, 5]. With the introduction of 3D Gaussian Splatting [19] which can achieve both fast and high-quality rendering.
increasing number of works [22, 24, 11, 12. 20. 23, 32, 13, 18] utilize it as their base representation, jointly with parametric human models like SMPL [26]. More specifically, GaussianAvatar [11] enhances 3D Guassians via two key components for final photorealistic quality, i.e., dynamic properties and joint optimization of motion and appearance. GauHuman [12] leverages human prior and KL divergence to propose new density control strategy. In this work, we systematically explore expressiveness while building human avatars from a monocular RGB video and tackle the new challenges brought by expressiveness.
2.2 Expressive Human Representations
Expressiveness plays a vital role in non-verbal communication, which, besides the body, particularly involves the hand and face regions [44, 48, 51, 8, 9, 28, 16, 35]. To represent expressive humans, various parametric models have been proposed, including SMPL-X [28], Adam [17], GHUM [43], and more. These models usually have predefined topology and can provide a compact mapping from the low-dimensional embedding to the 3D mesh. However, the predefined topology limits their capability of depicting fine-grained texture. Recent works build human avatars via neural 3D representations while leveraging the parametric model to provide human shape priors and animation signals. For example, X-Avatar [35] combines NeRF and SMPL-X to build the avatar. It further proposes part- sampling and initialization strategies to ensure efficient learning from high- quality 3D scans or RGB-D data. GVA [24] integrates the SMPL-X model to improve the rendering quality. AvatarRex [50] proposes a compositional avatar representation to separately model body, hand and faces from multi-view RGB video data. Different from them, aim to relax the input requirements, by building expressive human avatars from a real-world monocular RGB video.
Technical Approach
In this section, we first introduce the preliminary knowledge the SMPL-X human model [28] and 3D Gaussian Splatting [19] and present the general framework design on articulated human modeling. Then, we elaborate on the key technical contributions of our approach.
3.1 Preliminaries
SMPL-X body model. SMPL-X [28] is a parametric human body model, which extends the SMPL body model [26] by modeling both hand articulation and facial expressions. SMPL-X can be defined as mapping function (0, B, $) : Rlo x TRIP x R vl - R3N where 0, B,B.uu vp are the parameters for pose, shape, and facial expression, respectively. The function of SMPL-X is formulated as follows:
M 6, xb) ( 0 +b. J W),
(1)
T(8, 0, xb) T + Bs(B) (x + By)()),
(2)
where Br(+). Bs(-) and B-(() denote pose, shape, and expression blend functions, respectively, and W is set of blend weights. The pose, expression and shape corrective blend shapes, i.e., (0),
3

---

(1) and (8( add corrective vertex displacements to the template human mesh T. After that, linear blend skinning (-)) is applied rotate the vertices in the template mesh around the joints (B), smoothed by the blend weights W. This generates the final human mesh.
3D Gaussian Splatting. Methods based on NeRF [27] model the scene with an implicit representation and render novel views using volume rendering. In contract, 3D Gaussian Splatting [19] (3DGS) models 3D scene with a set of discrete 3D Gaussians and performs rendering through tile-based rasterization operation, which can reach real-time rendering speeds. Specifically, each Gaussian is defined with its central position p. 3D covariance matrix Σ as follows:
Gxz) ezp( )( (x - p)= E (x p)),
where z is an arbitrary position in the 3D scene. The covariance matrix decomposed into two learnable components to make the optimization easier, Σ = RSST R² where R and S denote the rotation matrix and scaling vector, respectively. During rendering, each 3D Gaussian G(z) is first transformed to a 2D Gaussian G' (x) on the image plane. Then, a bile-based rasterizer is designed to efficiently sort the 2D Gaussians and employ a-blending:
C(r) = Σ Go II (1-a)) o,), ar=)((()), KEN
5=1
where r is the queried pixel position and N denotes the number of sorted 2D Gaussians associated with the queried pixel. e; and a; denote the color and opacity of the i-th Gaussian, which are modeled by Spherical harmonics.
Articulated 3D human modeling. Our formulation takes inspiration from works employing 3D Gaussians for modeling articulated objects [12, 22]. We optimize 3D Gaussians in canonical space, which corresponds to a human in a rest pose [15]. The Gaussians are transformed from the canonical space to the frame space via linear blend skinning (LBS):
p' G(J + b(J', Σ = GJ 0l (TT ,,)),
(6)
where p/,2) Σí, pp, E the Gaussian mean and covariance in frame space and canonical space, re- spectively. G(J 0f) Σk= wy G 0²), b(J', 0r, ,5,355 = Ek=n wyb 05, (f) are the rotation and translation, respectively, with respect to the K joints, and G. (J1, 0f), by (J 01. ,1) are the rotation and translation, respectively, with respect to the k-th joint, wN. is the LBS weight.
To perform Linear Blend Skinning, we need important components, i.e., the LBS weights and the input pose parameters. Learning LBS weights from scratch would be inefficient and can lead local optimum in the early stage of the training. Therefore, for each Gaussian, we start with the LBS weight of the nearest SMPL-X vertex and use an MLP fe.. to predict an LBS weight offset w using the positionally encoded [27] Gaussian centers y(p"). Wi is therefore defined as:
clog(u (oex
W/k ΣK e Jon(ig +e)+w²
W4 = Se,, (M()))
(8)
SMPL-XX is the LBS weight of the nearest SMPL-X vertex. We set 10-8, where w
Starting from the input pose GSMPL.X, we further fine-tune it via an MLP-based network fe.. which jointly optimized during the 3D Gaussian optimization process. The actual poses used for optimization and rendering are therefore obtained as follows:
SMPL-X & fep (OSMPL-X).
where @ represents the vector pointwise product.
3.2 SMPL-X Alignment for Real-World Videos
key requirement for learning accurate human avatars is to initialize the optimization process with reliable SMPL-X estimate. However, this can be challenging, particularly in real-world cases,

---

SMPLX Alignment (Roal-wordd Videa)
Monocuter RGB Video
Modeling
Animatte Avatar
Gaussians
Rendered Image Image.
SMPLX Template
Init
Canonical Space
Q
Learnable LBS
Prior
Context-awwre Adaptive Density
Confidence-
Gaussian Avatar Modeling
Figure Overview of our proposed EVA method. Given monocular RGB video, first we estimate SMPL-X mesh that aligns well to the video frames using reconstruction module. Then, EVA utilizes 3D Gaussian Splatting for avatar modeling, while inheriting the human shape prior from the SMPL-X model. To improve the optimization and the quality of the avatar, we propose context-aware adaptive density control and confidence-ave aware loss.
given the limitations of current off-the-shelf methods [29] for SMPL-X fitting. This motivates us to propose robust fitting procedure that leverages multiple sources, including the initial estimation of camera parameter, SMPL-X parameters, 2D keypoints and 3D hand parameters from off-the-shelff tools [30, 45, 1]. Given these initial estimates, the SMPL-X fitting procedure minimizes the following objective:
C( = C2D + Cop + XhpChpp
(10)
The 2D keypoint term, LnD, encourages the projection of the 3D keypoints the avatar align with the detected 2D) keypoints. It is formulated as follows:
L2D Σ WWl (Ro J!P),
(11)
where (+) represents the camera projection under the given camera intrinsic parameter K. <p()) is the robust Geman-McClure eтror function [36] which helps prevent the disturbance from noisy 2D
keypoint detections. Ro denotes the function which rotates the joints J(8) given the pose 0. J2D is the detected 2D keypoints from [45]. y and w represent the predefined weighting parameters and the detection confidences respectively.
Our prior terms contain two components which focus on the coarse- grasinegr body and fine-grained hand, respectively. For the body part, utilize VPoser [28] to filter infeasible body poses. VPoser provides a compact mapping from a low-dimensional embedding v to the rotation matrices of the body pose 0g. To better optimize the low-dimensional embedding 7, the estimated 3DD body joints Jā [1] is treated as guidance, together with the added regularization term. The body prior loss term, Cop, is formulated as follows:
Cop 4b (Rg(())))) T3D) 1|n13
(12)
Similary, for the hands, we utilize the 3D hand joints D& estimated from [30] to provide better hand spatial relationship. This formulated as follows:
Lhp v/ Ro (f) - )AP),
(13)
where the index h represents the joints corresponding to the hand. v-(()) indicates that the robust error function only considers the coordinates along the z-axis coordinates for the error calculation.
3.3 Context-aware Adaptive Density Control
The core of the Gaussian Splatting optimization is adaptive density control, which generally contains two operations, i.e., densification (split and clone) and pruning. The original strategy [19] selects
5

---

fixed constant as the densification criteria. However, utilizing the fixed threshold does not leverage context information, leading to sub-optimal 3D Gaussian representations.
To this end, we propose a context-aware adaptive density control that leverages part attribute and history gradient information. Each Gaussian inherently possesses attributes associated with different body parts. These body parts vary in spatial size and characteristies. For instance, the hand is much smaller in size, compared to the body, exhibiting fine-grained textures and many degrees of freedom. These characteristics also lead to inherent differences in the Gaussian positional gradient changes across different body parts during the Gaussian optimization process. Furthermore, continuous increase in the Gaussian positional gradient indicates that Gaussians need to be densified.
Specifically, first initialize the 3D Gaussians with the vertices of the SMPL-X model. Since SMPL-X has a predefined topology, this initialization provides each Gaussian with the attribute information U on which part it belongs, e-g. body, hands face. After that, considering attributes and gradient history information, the densification threshold for the i-th Gaussian a certain attribute U is formulated as follows:
X
R-R Σ
G a
Σ k=t-R
V i,k
V.
(14)
k=t-2R
where is constant, R represents the densification interval, and V denotes the position gradient of i-th Gaussian at the timestamp k. Note that e and X have different values for different attributes. For the pruning strategy, we remove the points that are far away from the SMPL-X template vertices.
3.4 Objective Functions
The whole framework is optimized under the objective functions follows:
Le + Am + A,CSSIM + AtCLrrrs
(15)
where Am. A. and A are loss weighting terms. The mask loss L.m calculates the consistency between accumulated volume density and the estimated mask. The SSIM [40] loss LSSIAS is adopted to improve the structural similarity between rendered image and input image. The LPIPS [47] loss CLPIPS focuses on the perceptual quality of the rendered image.
Confidence-aware loss L.. It is inevitable that any training video will introduce some form of noise (e.g., misalignment, motion blur), which will also affect the avatar optimization procedure. To address some the effects of the noise, we introduce a feedback module to decide which pixels should be taken into consideration with higher lower weight during training. This module takes as input the rendered image L, and rendered depth D,, and predicts a score for each pixel which represents the confidence value. More specifically:
C= c=p+exp(E(L,,, D,)),
(16)
where μ is constant. Then the confidence serves as an adaptive weighting factor on the per per-pixel consistency, Eventually, we formulate our confidence-aware loss as follows:
= = 1T-1 T/1
(17)
Experiments
In this section, we first introduce experimental setup, including datasets, implementation details and evaluation metrics. Then, we make comparisons with baseline methods both quantitatively and qualitatively. Finally, we perform ablation studies the most important components of our approach.
4.1 Experimental Setup
Datasets. The experiments are conducted on two datasets, XHumans [35] and our collected UPB dataset. XHumans is a dataset captured in a controlled environment. It provides images with resolution of 1200 x 800, along with well-aligned SMPL-X meshes. There are identities (3 male and 3 female) for evaluation. UPB consists of sign language videos from the web, which usually contain complicated hand gestures. The resolution of the videos 1920 1080 and they do not contain any ground truth SMPL-X annotations. UPB includes 4 identities (2 male and female).

---

Table 1: Comparison with three expressive avatar baselines, Le., GART + SMPLX, Splatting + SMPL--X and GauHuman + SMPLX, on the XHumans and UPB dataset. N-GS denotes the number of Gaussians. and represent the higher the better, and the lower the better, respectively.
Full
Hand
Face
Method
N-GS
PSNR SSIM LPIPS SSIM1 LPIPS| PSNR SSIM LPIPS.
Controlled setting: XHumman dataset
3DGS [19] + SMPLX
19,458
28.88
0.9609
44.93
25.28
0.9189
91.37
25.91
0.9087
101.04
GART [22] SMPLX
89,571
27.73 0.9553
50.32
25.42
0.9151
99.53
25.86
0.9013
105.06
Splatting [34] + SMPLX
103,193
29.33 0.9606
44.39
26.19
0.9264
78.53
26.47
0.9103
92.51
GauHuman [12] SMPLX
17,134
29.16
0.9623
41.16
25.69
0.9225
88.16
26.27
0,9124
93.35
EVA
19,993
29.67
0.9632
33.05
26.27
0.9279
72,95
26.56
0.9157
72.30
Real-world setting: UPB dataset
3DGS [19] + SMPLX
21,008
25.31
0,9469
90.80
24.89
0.9425
66.19
24.57
0.9072
136.53
GART [22] SMPLX
90,676
26.20
0,9511
78.90
25.25
0.9411
61.44
26.62
0.9253
93.28
Splatting [34] + SMPLX
257,811
25.13
0.9355
96.16
24.20
0,9298
70.91
24.48
0.8962
127.63
GauHuman [12] + SMPLX
12,372
25.17
0.9455
84.87
24.67
0.9418
67.61
24.33
0,9035
113.13
BVA
20,829
26.78
0.9519
65.07
27.00
0.9524
45.90
26.85
0.9298
65.90
Implementation details. Our framework is implemented on PyTorch and all experiments are performed on NVIDIA A5000. The hyperparameter Xm A, and X are set 0.1, 0.01 and 0.04, respectively. We use SGHM [4] to extract the human mask. The 3D Gaussian optimization lasts for 2,000 iterations with the densification performed between iteration 400 and 1,000. For other parameters, we follow the original settings of [19]. Our method can render 1080p image with an inference speed of 361.02 fps on one RTX A5000. Since our method is reconstruction based, our main requirement is that the areas of interest should be captured in the video. Note that same input RGB frames for all methods to ensure a fair comparison. More details are presented in the Appendices.
the
Evaluation metrics. We evaluate the avatar quality through the rendered views following previous works [22, 12]. We adopt the widely-used Peak Signal-to-Noiss Ratio (PSNR), Structural Similarity Index Measure (SSIM) [40], and Learned Perceptual Similarity (LPIPS) [47] to evaluate the rendered images. For more fine-grained evaluation, we report the above metrics separately for the full body, for the hands region and for the face region.
4.2 Comparison with baselines
The baselines we use for comparison are adopted from related methods with a few modifications. 3DGS + SMPL-X is modified from 3DGS [19]. We add our articulated human modeling mechanism to make it fit the task requirement. GART and GauHuman are originally designed to model body-level avatars. They are animated via the SMPL parametric model, which lacks the capability of modeling hand articulation and facial expressions. Therefore, we replace the driven signal with the SMPL-X model and denote them as GART SMPL-X and GauHuman SMPL-X, respectively. Splatting selects the triangle mesh as the driving signal and we utilize SMPL-X mesh in Splatting SMPL-X. Among these baselines, GauHuman is closer our approach. Our differences are mainly reflected in the adaptive density control strategy, objective functions and the SMPL-X alignment for the real-world videos, which are also consistent with our main technical contributions.
As shown in Table 1, we conduct experiments on the XHumans and UPB datasets to evaluate the effectiveness of our method. XHumans is captured in controlled setting with accurate SMPL-X ground truth, while UPB focuses on real-world video without pose information. Since it is hard to get accurate SMPL-X annotation for real videos, UPB is more challenging and more representative of learning a human avatar in the wild. We observe that our method achieves state-of sate-oof-the-ar the performance on these two datasets. We note that we do not apply our designed alignment module XHumans, since it already contains SMPL-X ground truth. Our method achieves 19.7%, 17.3% and 22.5% relative LPIPS gain on the full, hand and face regions, respectively. For the real-world UPB dataset, our performance gain is much larger, achieving over 25% relative LPIPS gain on the hand region. This indicates that EVA, unlike previous work, handles well the challenges of real-world videos. The qualitative comparison in Figure 3 also validates the effectiveness of our method.
7

---

GT
3DGS+
GART+
GauHuman+
EVA
Figure 3:
Qualitative comparison with baselines. We compare with 3DGS [19] + SMPL-X, GART [22] SMPL--X and GauHuman [12] + SMPL-X. First three rows are from UPB and last row is from the XHumans dataset. EVA exhibits the best visual quality. See the zoomed-in results in the box for comparison of the fine- grained details.
4.3 Ablation Studies
We perform ablation experiments to highlight the important components of EVA, í.e., context-aware adaptive density control, confidence-awart loss and in-the-wild SMPL-X alignment module.
Effectiveness of context-aware adaptive density control (CADC) and confidence-aware loss (CL). These two components are designed for better optimization of Gaussian Avatar Modeling. As shown in Table 2, "w/o CADC" means that we replace with the original density control [19], while "w/o CL" denotes removing the confidence weighting term in the RGB loss calculation. We observe that these two components both improve performance. More specifically, CADC improves performance for the metrics of all regions. Meanwhile, we observed that CL brings another benefit of a more compact representation (e.g., reducing the number Gaussians from 21,00 1,,088 to 19,993).
Effectiveness of in-the-wild SMPL-X alignment. We use our SMPL-X alignment module for real-world videos without accurate SMPL-X ground truth, so we conduct the corresponding ablation study on UPB dataset. As shown in Table 2, "w/o Align" means that we directly utilize the SMPL-X mesh extracted from current SOTA estimation method [1]. Notably, our proposed alignment module brings notable performance gains on all metrics of all regions. It also demonstrates the importance of the SMPL-X pose quality for final human avatar modeling. Without accurate SMPL-X alignment, the following Gaussian model will need to deal with inconsistent textures for specific region across frames, leading to lower avatar quality. Furthermore, we visualize SMPL-X alignments in Figure and present qualitative comparison with -fate-of-the-art estimation methods in Figure 5, including fitting-based AMPLify-XX and regression- based SMPLer-X. Visible improvements are clearly observed specifically for the hands.

---

Table 2: Effect of context-aware adaptive density control (CADC), confidence-aware loss (CL) and SMPL-X alignment. and represent "higher the better", and "lower the better", respectively.
Full
Hand
Face
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
XHumans dataset
w/o CADC
28.92
0.9611
35.14
25.23
0.9175
84.00
26.08
0.9080
80.76
w/o CL.
29.63
0.9634
34.08
26.27
0.9279
74.02
26.58
0.9166
73.19
EVA
29.66
0.9632
33.05
26.27
0.9279
72.95
26.56
0.9156
72.30
UPB dataset
w/o Align
25.02
0.9435
73.82
24.64
0.9396
63.64
24.27
0,9009
93.56
w/o CADC
26.53
0.9504
68.76
26.43
0.9494
50.59
26.45
0.9265
71.17
w/o CL
26.74
0.9507
67.90
26.74
0,9507
48.54
26.69
0.9293
69.60
EVA
26.72
0.9519
65.37
26.90
0.9523
46.11
26.75
0.9298
66.41
GT
w/o Align
EVA
GT
w/o Align
EVA
Figure 4: Effect our SMPL-X alignment module. We can estimate SMPL-X mesh that aligns well with the RGB frame, especially for the fine-grained hand regions.
4.4 More Discussion
In summary, we apply EVA on variety of monocular RGB videos, including both controlled captures and videos collected from the Internet. The visual data contain large body motions (e.g., playing basketball, weight lifting, dancing), along with fine-grained motions (e.g. finger counting, using tools, sign language). EVA can handles well self-occlusions, which usually exist in every frame, especially for the hand areas. This can be attributed to proposed SMPL-X fitting method which provides reliable correspondences from the pixels to the canonical space. Besides, we expect that EVA could also handle low lighting conditions (as long the lighting is not changing significantly), since our SMPL-X fitting method should be robust to low lighting conditions. Given accurate SMPL-X estimates, we expect the avatar learning stage will perform well.
Limitations. The focus of this work is to capture expressive human avatars, by designing pipeline that can take raw web videos as the input. Besides capturing the expressive details, other factors are challenging, yet worth investigating elements like clothes and hair. It is also worth extending avatar reconstruction more challenging input sources, e.g. occlusions or few-shot scenarios.
further improve the quality of the avatar, e.g. non-rigid
Failure Cases. We show some representative ex- amples of our failure cases in Figure 6. For the fine-grained expressive areas, the failure cases happen when the video includes hand interac- tions. The failure is mainly caused by incor- rect SMPL-X reconstruction (e.g.. the estimated SMPL-X model that drives the reconstruction implausible interpenetration between hands). Another failure case is the existence of floaters in the reconstruction which is a common issue of 3DGS modeling.
Figure Illustration of failure cases.

---

SMPLer-X
SMPLify-X
Ours
Figure 5: Comparison among methods for SMPL-- alignment. We compare our results with a regression-baseer method (SMPLer-XX, and ätting-based method (SMPLify-X).
Broader Impact. this work, our proposed framework helps push the boundaries of what is possible with human avatar modeling, allowing for more expressive results than previously achievable. However, with these advancements come certain risks, particularly concerning the potential misuse of the technology for deceptive practices harassment. By advancing SOTA, our work makes these potential risks more pressing, as the barrier to creating such deceptive or harmful content is lowered. This highlights the need for careful consideration of the ethical implications and the development of strategies to mitigate these risks.
Conclusion
In this work, we present EVA, a drivable expressive human avatar learned from a real- world monocular RGB video. EVA is built on 3D Gaussians, in coordination with the human prior introduced by SMPL-X. To deal with the challenges brought by expressiveness, we first utilize a reconstruction module to solve the misalignment between SMPL-X model and RGB frames. During the optimization of the 3D Gaussians, we propose context- aware adaptive density control, which leverages attribute and historical gradient information to accommodate the varied granularity across body parts. A feedback mechanism is jointly designed to further guide learning. Extensive experiments on two benchmarks demonstrate that our method outperforms baselines both quantitatively and qualitatively, especially on the fine-graine hand and facial details.
Acknowledgments. This project and partially supported by ARL grants W91 INF20-2-0158 and cooperative A212 program. It was also in part supported by NSF AI Institute for Foundations of Machine Learning (IFML). GP has received a research gift from Google.
supported by LUCI program under the Basic Research Office under the
References
Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang. Haiyi Mei, Mingyuan Zhang, Lei Zhang, al. Smpler-x: Scaling up expressive human pose and shape estimation. NewrIPS, pages 1-15, 2024.
[2] Xu Chen, Tianjian Jiang, Jie Song. Max Rietmann, Andreas Geiger, Michael J. Black, and Otmar Hilliges. Fast-snarf: A fast deformer for articulated neural fields. IEEE TPAMI, 2023.
[3] Xu Chen. Yufeng Zheng. Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, 2021.
10

---

[4] Xiangguang Chen, Ye Zhu, Yu Li, Bingtao Fu, Lei Sun, Ying Shan, and Shan Liu. Robust human matting via semantic guidance. In ACCV. pages 2984-2999, 2022.
[5] Zhaoxi Chen, Fangzhou Hong, Haiyi Mci, Guangcong Wang, Lei Yang, and Ziwei Liu. Primdiffusion: Volumetric primitives diffusion for 3d human generation. In NeurIPS, pages 1-14, 2023. [6] Junting Dong. Qi Fang. Yudong Guo, Sida Peng. Qing Shuai, Xiaowei Zhou, and Hujun Bao. Totalselfscan: Lccaming full-body avatars from self-portrait vidcos of faces, hands, and bodies. In NewrIPS, pages 13654-13677, 2022.
[7] Maria-Paola Forte. Peter Kulits, Chun-Hao P Huang, Vasileios Choutas, Dimitrios Tzionas, Katherine J Kuchenbecker, and Michael J Black. Reconstructing signing avatars from video using linguistic priors. In CVPR, pages 12791-12801, 2023.
[8] Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, and Ziwei
Liu. Stylegan- human: A data-centric odyssey of human generation. In ECCV, pages 1-19, 2022.
[9] Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Wayne Wu, and Ziwei Liu. Unitedhuman: Harnessing multi-source data for high-resolution human generation. In ICCV, pages 7301-731 1, 2023.
[10] Chen Geng, Sida Peng. Zhen Xu, Hujun Bao, and Xiaowei Zhou. Learning neural volumetric representa- tions dynamic humans in minutes. In CVPR, pages 8759-8770, 2023.
[11] Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from single video via animatable 3d gaussians. arXiv, pages 1-13, 2023.
[12] Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting for real-time 3d human rendering.
In CVPR, pages 1-16, 2024.
[13] Rohit Jena. Ganesh Subramanian Iyer, Siddharth Choudhary, Brandon Smith, Pratik Chaudhari, and James Gee. Splatarmor: Articulated gaussian splatting for animatable humans from monocular rgb videos. arXiv, pages 1-11, 2023.
[14] Tianjian Jiang. Xu Chen, Jie Song, and Otmar Hilliges. Instantavatar: Learning avatars from monocular video in 60 seconds. CVPR. pages 16922- 16932,, 2023.
[15] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. In ECCV, pages 402 018. 2022.
[16] Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong. Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. In CVPR, pages 1-12, 2024.
[17] Hanbyul Joo, Tomas Simon, and Yaser Sheikh, Total capture: A 3d deformation model for tracking faces, hands, and bodics. In CVPR, pages 8320 8029,,32,,200 2018.
[18] HyunJun Jung, Nikolas Brasch, Jifei Song, Eduardo Perez-Pellitero. Yiren Zhou, Zhihao Li, Nassir Navab, and Benjamin Busam. Deformable 3d gaussian splatting for animatable human avatars. arXiv, pages 1-15, 2023.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 1244)):-14, 2023.
[20] Muhammed Kocabas, Jen-Has Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. Hugs:
Human gaussian splats. In CVPR, pages 1-16, 2023.
[21] Youngjoong Kwon, Lingjie Liu, Henry Fuchs, Marc Habermann, and Christian Theobalt. Deliffas: Deformable light fields for fast avatar synthesis. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine, editors, NeurlPS, pages 40944-40966 2023.
[22] Jiahui Lei. Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilidis. Gart: Gaussian articulated template models, arXiv, pages 1-13, 2023.
[23] Mingwei Li, Jiachen Tao, Zongxin Yang, and Yi Yang. Human101: Training 100+ fps human gaussians in 100s from 1 view. arXiv, pages 1-20, 2023.
[24] Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Chen Zhao, Haocheng Feng, Errui Ding, and Jingdong
Wang. Gva: Reconstructing vivid 3d gaussian avatars from monocular videos. arxiv, pages 1-13, 2024.
[25] Stephen Lombardi, Tomas Simon. Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih.
Mixture of volumetric primitives for efficient neural rendering. ACM TOG, 40(4):1-13, 2021.
[26] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL. A skinned multi-person linear model. ACM TOG, 34(6):1-16. 2015.
[27] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020.
[28] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, pages 10975-10985, 2019.
[29] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black, Expressive body capture: 3d hands, face, and body from single image. In CVPR, pages 10975-10985, 2019.

---

[30]
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In CVPR, pages 1-11, 2023.
[31] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR. pages 9054-9063. 2021.
[32] Zhiyin Qian, Shaofei Wang, Marko Mihajlovic. Andreas Geiger. and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In CVPR. pages 1-19, 2024.
[33] Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito, Chenglei Wu. Tomas Simon, Shih-En Wei, Kaiwen Guo, Zhe Cao. Fabian Prada, Jason Saragih, et al. Drivable volumetric avatars using texel-aligned features.
In SIGGRAPH. pages 1-9, 2022.
[34] Zhijing Shao, Zhaolong Wang. Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang. Mingming Fan, and Zeyu Wang. Splattingavatar: Realistic real-time human avatars with mesh-embedded gaussian splatting. In CVPR, pages 1606-1616, 2024.
[35] Kaiyue Shen, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Julien Valentin. Jie Song, and Otmar Hilliges. X-avatar: Expressive human avatars, In CVPR, pages 16911-16921, 2023.
[36] Geman Stuart and E. McClure Donald. Statistical methods for tomographic image reconstruction. Bulletin of the International Statistical Institute, 52:5-21, 1987.
[37] Shih- Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In NewrlPS, pages 12278-12291, 2021.
[38] David Svitov, Pietro Morerio, Lourdes Agapito, and Alessio Del Bue. Haha: Highly articulated gaussian human avatars with textured mesh prior. arXiv, pages 1-23, 2024.
[39] Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In ECCV, pages 1-19, 2022.
[40] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE TIP. 13:600-612, 2004.
[41] Chung- Yi Weng. Brian Curless, Pratul P. Srinivasan. Jonathan T. Barron, and Ira Kemelmacher Shhizeremaa....m...... HumanNeRF: viee-vit rendering of moving people from monocular video. In CVPR, pages 16210-16220, 2022.
[42] Hongyi Xu. Thiemo Alldieck. and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. NeurlPS, 34:14955- 14966, 2021.
[43] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR, pages 6184-6193, 2020.
[44] Zhongcong Xu, Jianteng Zhang, Junhao Liew, Jiashi Feng, and Mike Zheng Shou. Xagen: 3d expressive human avatars generation. In NeurlPS, pages 34852-34866 2023.
[45] Zhendong Yang. Ailing Zeng. Chun Yuan, and Yu Effective whole-body pose estimation with two-stages distillation. In ICCV, pages 4210-4220, 2023.
[46] Zhengming Yu. Wei Cheng. Xian Liu, Wayne Wu, and Kwan-Yee Lin. Monohuman: Animatable human neural field from monocular video. CVPR. pages 16943-16953, 2023.
[47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, pages 586-555655555 2018.
[48] Zechuan Zhang, Li Sun, Zongxin Yang. Ling Chen, and Yi Yang. Global-correlated 3d-decoupling transformer for clothed avatar reconstruction. NewrIPS, pages 78300, 2023.
[49] Fuqiang Zhao, Wei Yang. Jiakai Zhang, Pei Lin, Yingliang Zhang. Jingyi Yu, and Lan Xu, Humannerf: Efficiently generated human radiance field from sparse inputs. In CVPR, pages 7743-7753, 2022. [50] Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu. Avatarrex: Real-time expressive full-body avatars. ACM TOG, 42:1-19, 2023.
[51] Wojeiech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv, pages 1-11, 2023.
12

---

Appendices
This technical appendices provides more details which are not included in the main paper due to space limitations. For more visualization results, please refer to the project webpage.
Training/Testing split. For each identity in XHumans dataset, split, where the other videos are marked as testing. During training, we utilize all of the frames (150 frames). We sample 20 frames for each testing video with the sampling rate of 5. For UPB dataset, we uniformly sample the frames with the interval as 1 to split the training and testing frames. The number of training and testing frames are both 140.
video is selected the training
Additional implementation details. 1l is set to 1. A is set as -9.0, -4.5 and -6.3 for body, hand, and face parts, respectively. We set for body, hand, and face parts 2c-4, le-4 and 1.4e-4, respectively. The feedback module E-)) consists of two 2D convolutional networks. For SMPL-X alignment, we utilize the L-BFGS optimizer with the Wolfe line search. The optimization contains three stages with different loss weighting factors. The first stage is designed to initialize the human body embedding to the initial estimation from the method [1]. Then the second stage mainly aims to get better spatial hand relationship leveraging the prediction from the method [30]. The third stage performs fine-tuning with more emphasize the fine-grained hand and face regions.
Comparison baselines. The baselines we use for comparison are adopted from related methods with few modifications. 3DGS SMPL-X is modified from 3DGS [19]. Since the original 3DGS does not have the capability of animation, we add the same articulated human modeling in Section 3.1 as ours. For fair comparison, it also utilizes the same optimization schedule ours, which lasts 2,000 iterations with the densification performed between 400 and 1,000 iterations. GART and GauHuman originally designed to model body- avatars. They animated via the SMPL parametric model, which lacks the capability of modeling hand articulation and facial expressions. Therefore, replace the driven signal with the SMPL-X model and denote them as GART + SMPL-X and GauHuman + SMPL-X, respectively. Since Splatting is embedded on a triangle mesh, we utilize the SMPL.-X human mesh its driving signal and denote it as Splatting SMPL-X. We do modify the optimization schedules of these three methods.
Future works. We outline the potential future works as follows,
Modeling capability on non-rigid elements, such as loose cloth (dress). Modeling cloth on the avatar has been a challenging topic, even separately studied in several works. A potential solution could be parameterizing the cloth deformation or adding more prior on cloth type, so as to provide more driving signals.
Generalizable human avatar from monocular RGB video. Current methods in this topic mostly need per-subject optimization, which needs to be re-trained to any new given subjects. It is worth exploring if we could get an avatar from monocular RGB video of any given subject with single ferwafo pass.
Robustness. is worth exploring if we could build the human avatar well with more limited source inputs, c.g. a few images with even mutual occlusion, single image, etc.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly stated the contributions and scope in Abstract and Introduction.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We describe the limitations in the Section 4.4.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have presented the information in Section 4.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our code after our paper gets accepted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details specified in Section 4.1.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We use the fixed seed for easy reproducing.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We report the type of resources used in the experiments in Section 4.1.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http///rrr...cco//l/////ccccccie
//
14

---

Answer: [Yes]
Justification: The research conducted in the paper conform the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discusses the broader impact in Section 4.4.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The released model does not have a high risk for misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite corresponding papers for the asserts use in Section 4.1.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
15


Reviews:
[{"review_id": "zk60tmNJNk", "reviewer": "Reviewer_1sDc", "summary": "- Given an RGB human video, the paper focuses on photorealistic reconstruction of the human in 3D using SMPL-X and Gaussian rendering.\n- The main idea is to align the expressive SMPL-X with the video evidence followed by Gaussian Avatar modeling with better gaussian density control and loss. \n- Technical contributions: Context-aware adaptive density control, confidence-aware loss, SMPL-X alignment.\n- Baselines: 3DGS, GART, GauHuman.\n- Evaluation are done on two datasets, XHuman and UPB. Metrics: PSNR, SSIM, LPIPS.\n- The results show that the proposed method EVA consistently outperforms the baselines.", "questions": "Given the lack of technical novelty, it is important to establish the empirical usefulness of the proposed method. \nIn this context, I am asking for a fair evaluation with prior methods (mostly using SMPL) on popular benchmarks.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written, organized and easy to follow.\n- The focus problem is of great importance for the field of human digitization.\n- The ablative studies are informative, giving us insights into the importance of the technical components.", "weaknesses": "- Weak technical contributions: In comparison to GauHuman, CVPR 2024, the proposed method EVA makes three changes. 1) Context-aware adaptive density (CADC), 2) Confidence-aware loss (CL), 3) SMPL-X alignment. CADC is a nice idea for human modeling, and is inspired by the original 3DGS paper. However, it is a slight change to the originally proposed heuristic. CL is frequently used in the field of pose-estimation and 3D modeling (eg. DUST3R, CVPR 2024). SMPL-X alignment is very simply mesh fitting to 2D keypoints and is well explored in SMPLify-X, CVPR 2019 and its related works. Overall, the main technical ideas proposed lack technical novelty.\n\n- XHuman over ZJU_MoCap for evaluations: Is there a reason why metrics were reported on XHuman and not ZJU-MoCap dataset? The missing metrics on the popular ZJU-MoCap benchmark makes it hard to fairly compare EVA with existing. I am guessing this because ZJU-MoCap does not have SMPL-X annotations. In this context, can we use the same setup as UPB and still evaluate using predicted SMPL-X parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarification on technical contributions**.\n\nA1: Our proposed three technical contributions are well-motivated and well-formulated to meet the requirement of expressiveness in human avatar modeling, which are validated by extensive experiments and have been acknowledged by Reviewer 4SGk ('the paper introduces three significant innovations'). \n\nOur proposed fitting aims at mitigating the misalignment issues, especially on the fine-grained hands, which designs effective loss terms leveraging multiple pseudo ground truths (meshes and 2D keypoints predicted by off-the-shelf methods). As shown in Figure 1 of the rebuttal pdf, our fitting results are clearly better than both SOTA regression-based method SMPLer-X and optimization-based fitting (SMPLify-X). Its effectiveness has also been acknowledged by Reviewer g7vJ ('significantly improves the alignment between SMPL-X and the RGB frames').\n\n**Q2: Reasons for not reporting results on ZJU-Mocap.**\n\nA2: The main reason for not choosing ZJU-Mocap is that it is not suitable for evaluating expressiveness. It has limited diversity on hand gestures/facial expressions (only clenched fist or open palm), and insufficient hand resolutions following the practice of previous methods. \nMeanwhile, we also prioritize the experiment on ZJU_Mocap and have spent over 24 hours trying to solve it.\n1) The first option (proposed by the reviewer) is that we use the same setup as UPB and still evaluate using predicted SMPL-X parameters, the input SMPLX quality will be worse than the SMPL counterpart given in ZJU-Mocap, which will lead to an unfair comparison.\n2) Another option is that we try to convert the given SMPL annotation to the form of SMPL-X. However, we still find there remains the projection discrepancy which will also lead to an unfair comparison.\n\nWe have conducted the experiments on two benchmarks to perform a fair comparison with previous methods, which is acknowledged by Reviewer ZKQ6 ('It is commendable that the authors added SMPL-X to existing methods for fair and thorough comparisons').\n\n\n**Q3: Clarification on the empirical usefulness of the proposed method.**\n\nA3: The main empirical usefulness is that our method could build the expressive avatar from monocular RGB video while getting rid of time-consuming SMPLX annotation. From the application perspective, it could be utilized in sign language production. We have demonstrated this application by conducting experiments on the UPB benchmark (videos collected from the Web).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6wepdEg9Z", "reviewer": "Reviewer_4SGk", "summary": "This paper proposes a solution to generate expressive human avatars from monocular RGB videos. The main focus is to improve the expressiveness. To this end, a few ideas such as combining 3d Gaussian splatting and SMPL-X (SMPL+parametric hands), minimizing 2d reprojection error,  finegrained density control, etc. \n\nThere are a few similar papers in citation. To better understand the novelty of this paper, here's a high level comparison:\n+ SMPLer-X [1]: Representation is SMPL-X only. Input is monocular RGB videos. Losses: SMPL-X keypoint losses.\n+ GART[23]:  Representation is 3d Gaussians+SMPL. Input is monocular RGB videos.  Losses: photometric, perceptual.\n+ 3DGS-Avatar [33]: Representation is 3d Gaussians+SMPL; Input is monocular RGB videos; Losses: photometric, perceptual, mask, etc; \n+ This paper: Representation is 3d Gaussians+LBS+SMPL-X. Input is monocular RGB videos. Losses: SMPL-X keypoint losses, perceptual.", "questions": "What's 'E' in Eq (16)? It looks like the so-called feedback module. But what's the design? Is it a learning-based predictor?\n\nPlease also address the novelty questions in the weakness part.", "limitations": "Limitations are discussed. It'd be good to discuss the failure cases.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "From the high level comparison, the paper does find a unique and timely combination of 3d Gaussians and SMPL-X, which is well-motivated by improving the expressiveness. Some of the ideas such as adaptive density control are therefore new as it leverages the fine-grained details of SMPL-X.", "weaknesses": "The concept of model and image alignment by minimizing reprojection losses is not new. For example, the keypoints losses (Sec. 3.2) are very common in literature [1, 30]. \nSimilarly mask loss and perceptual loss can be found in [33].\n\nThe adaptive density control part seems novel, as it's initialized from body parts and leveraging the fine-grained topology of SMPL-X.\n\nThe confidence-aware loss seems new but only provides marginal improvement as in Table 2.\n\nLack of hyperparameter ablation. Not even in supplementary. \n\nMinor issue: `video` is countable. Please explicitly use `videos` or `a video` in the paper. Actually it's not clear how many videos of one subject are used to train one avatar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarification on reprojection losses.**\n\nA1: We want to clarify that the reprojection losses (keypoint, mask, perceptual loss) are not claimed as our contributions. Instead, we include them in our paper to describe the necessary components of our framework and make sure our paper is self-contained.\n\n**Q2: Clarification on confidence-aware losses.**\n\nA2: We observed that the Confidence-aware loss helped with perceptually improving the results, which is also reflected to the consistent performance improvement on the LPIPS metrics. Moreover, another benefit of using the Confidence-aware loss is that it led to a more compact representation (e.g., reducing the number of Gaussians from 21,038 to 19,993). We also present some qualitative results (w/o and w confidence-aware loss) in Figure 3 of the rebuttal pdf.\n\n**Q3: Ablation on hyperparameter.**\n\nA3: We observed that one of the most crucial hyperparameters was the value $\\lambda_{t}$ in the context-aware adaptive density control. We perform an ablation on this hyperparameter.\n\n| **$\\lambda_{t}$** | **PSNR (Full)** | **SSIM (Full)** | **LPIPS (Full)** | **PSNR (Hand)** | **SSIM (Hand)** | **LPIPS (Hand)** | **PSNR (Face)** | **SSIM (Face)** | **LPIPS (Face)** |\n|-------------------|--------------------|--------------------|-----------------------|--------------------|--------------------|-----------------------|--------------------|--------------------|-----------------------|\n| 0.0               | 28.92              | 0.9611             | 35.14                 | 25.23              | 0.9175             | 84.00                 | 26.08              | 0.9080             | 80.76                 |\n| -6.0              | 29.28              | 0.9619             | 33.71                 | 25.69              | 0.9221             | 77.81                 | 26.33              | 0.9114             | 74.21                 |\n| **-9.0**              | **29.66**          | **0.9632**             | **33.05**             | **26.27**          | **0.9279**         | **72.95**             | **26.56**              | **0.9156**             | **72.30**             |\n| -12.0             | 29.29              | 0.9619             | 33.85                 | 25.67              | 0.9219             | 78.75                 | 26.34              | 0.9113             | 74.29                 |\n\n\n\n**Q3: Typo on 'video'.**\n\nA3: Thanks for pointing out this issue. We will revise the usage of 'video' in the manuscript. We clarify that only one monocular RGB video of one subject is used to train the corresponding avatar.\n\n**Q4: What's 'E' in Eq (16)? It looks like the so-called feedback module. But what's the design? Is it a learning-based predictor?**\n\nA4: 'E' is the feedback module. It mainly consists of two convolutional layers and is a learning-based predictor. We show the pseudo code of this module below.\n\n```\nclass E_feedback(nn.Module):\n    def __init__(self, num_feat=4):\n        super(E_feedback, self).__init__()\n        self.conv1 = nn.Conv2d(num_feat, 3, 1, 1, 0)\n        self.conv2 = nn.Conv2d(num_feat + 3, 1, 1, 1, 0)\n        self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)\n    def forward(self, x):\n        x1 = self.lrelu(self.conv1(x))\n        x2 = self.conv2(torch.cat((x, x1), 1))\n```\n\n**Q5: Suggestion on discussing the failure cases.**\n\nA5: Thanks for pointing out this issue. For the fine-grained expressive areas, the failure cases happen when hand interaction exists, which is mainly caused by incorrect SMPLX reconstruction (e.g. the estimated SMPLX model that drives the reconstruction has implausible interpenetration between hands). Another failure case is the existence of floaters in the reconstruction which is a common issue of 3DGS modeling. We show some representative examples in Figure 2 of the rebuttal pdf.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2eIELIBbFc", "reviewer": "Reviewer_fFxo", "summary": "The paper presents EVA, a model that can recover expressive human avatars from monocular RGB videos. The primary focus is enhancing fine-grained hand and facial expressions using 3D Gaussian Splatting and the SMPL-X model. EVA introduces three key contributions: a plug-and-play module for better SMPL-X alignment, a context-aware adaptive density control strategy, and a feedback mechanism for optimizing 3D Gaussian learning. Extensive experiments on two benchmarks demonstrate EVA's superiority in capturing detailed expressiveness compared to previous methods.", "questions": "1. How does EVA perform with videos that have significant occlusions or low lighting conditions?\n2. Are there any specific preprocessing steps required for the input monocular RGB videos?\n3. Can the proposed method be extended to capture other expressive elements, such as cloth or hair movements?", "limitations": "1. The performance heavily relies on the quality of the datasets used. Real-world applications might present challenges not covered by the current benchmarks.\n2. The method requires significant computational power, which might limit its applicability in real-time scenarios or on devices with limited resources.\n3. While the method focuses on hand and facial details, other expressive elements like body language, clothing, and hair dynamics are not addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper introduces three significant innovations: a plug-and-play alignment module, a context-aware adaptive density control strategy, and a feedback mechanism for 3D Gaussian optimization.\n2. Extensive quantitative and qualitative evaluations on two benchmarks show EVA's effectiveness in capturing fine-grained details, particularly in hand and facial expressions.\n3. The proposed method addresses the challenging task of creating expressive avatars from monocular RGB video, which has practical applications in VR/AR, movie production, and video games.\n4. The paper comprehensively explains the technical approach, including SMPL-X alignment, Gaussian optimization, and adaptive density control.", "weaknesses": "1. The proposed method's complexity might pose implementation challenges for those not well-versed in the field. Simplifying some aspects or providing more intuitive explanations could help. \n2. While the method shows superior performance on the provided datasets, the generalizability to other types of monocular videos or environments is not thoroughly discussed.\n3. Although the paper compares EVA with previous SOTA methods, it would be beneficial to include more diverse baselines or variants to demonstrate the robustness of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Suggestion on simplifying some aspects or providing more intuitive explanations.**\n\nA1: Thanks for this suggestion. We commit to releasing our code upon acceptance. Our work makes a step towards building an expressive avatar from the real-world video. In short, our method aims to answer two key questions below,\n\n1) How to generate a well-aligned SMPLX mesh to provide accurate correspondence for expressive avatar modeling? We propose a fitting-based alignment method explicitly focusing on fine-grained details (e.g. hand).\n\n2) How to enhance the expressiveness during avatar modeling? We propose context-aware adaptive density control, in coordination with the feedback mechanism.\n\n**Q2: More discussion on the generalizability to other types of monocular videos or environments.**\n\nA2: Currently, our work has included various types of monocular RGB videos from controlled capturing setting to those collected from the Internet. The involved human poses contain large body motion (e.g., playing basketball, weight lifting, do kicks, dancing), along with fine-grained motions (e.g. finger counting, using tools, sign language). Among them, considering the complexity and practicability of sign language, we separately involve it to evaluate expressiveness and collect a new benchmark (UPB) from the Internet. We acknowledge that there are more types of videos and environments that could be further explored (such as when the people are occluded by other objects, changing light conditions) and will add this discussion as promising future work.\n\n**Q3: Suggestion on including more diverse baselines or variants to demonstrate the robustness of the proposed approach.**\n\nA3: Thanks for your suggestion! We have used the Splatting Avatar method from CVPR 24 and evaluated it on the datasets we used. The results are shown in A1 of General Response. We will add these experiment results in Table 1.\n\n\n**Q4: How does EVA perform with videos that have significant occlusions or low lighting conditions?**\n\nA4: EVA can deal well with self-occlusions, which usually exist in each frame, especially on the hand areas. This can be attributed to our proposed SMPL-X fitting method which could provide reliable correspondence to map the pixels to the canonical space. Both qualitative and quantitative results demonstrate the effectiveness of our method. Besides, we expect that EVA could also handle low lighting conditions (as long as the lighting is not changing significantly), since our SMPL-X fitting method should be robust to low lighting conditions and with good SMPL-X estimates, we expect the Gaussian Splatting stage to perform well.\n\n**Q5: Are there any specific preprocessing steps required for the input monocular RGB videos?**\n\nA5: Since our method is reconstruction-based, our main requirement is that the areas of interest should be captured in the video. For example, for the full-body X_Human dataset, the monocular RGB video includes observations around the person of interest. On the other hand, for the upper-body UPB dataset, since the area of interest for sign language is only on the upper body, we only need to capture the corresponding appearance of the person. We note that we keep the input RGB frames of all comparison methods consistent for fair comparison.\n\n**Q6: Can the proposed method be extended to capture other expressive elements, such as cloth or hair movements?**\n\nA6: We believe our work could be extended to capture other expressive elements. One possible solution is to embed the driving signal with more modeling capabilities considering the other expressive elements that we need to incorporate. In this way, our method could benefit from explicit signals to perform modeling. However, this is beyond the scope of our current work.\n\n\n**Q7: The performance heavily relies on the quality of the datasets used. Real-world applications might present challenges not covered by the current benchmarks.**\n\nA7: The high-quality dataset is needed to ensure sufficient resolution on fine-grained expressive areas for training and evaluation. With the popularization of low-cost consumer cameras, high-quality data (e.g. 1080p) becomes much easier to capture, even via personal phones. With that being said, our proposed SMPLX fitting simplifies and robustifies the whole avatar generation pipeline, such that it works with sufficient robustness in Internet videos (e.g. see the UPB videos that come from YouTube).\n\n**Q8: The method requires significant computational power, which might limit its applicability in real-time scenarios or on devices with limited resources.**\n\nA8: We only need one GPU for both training and testing. Once our model is trained, it is efficient to run, which could render the 1080p image at 361.02 fps on one RTX A5000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "op5ar9mmkl", "reviewer": "Reviewer_g7vJ", "summary": "The paper introduces a 3DGS-based human avatar generation method from a monocular RGB video. Specifically, the authors first optimize the SMPL-X model to better align with the RGB frames. Then, they propose an adaptive adjustment method for different body parts and use per-pixel confidence to guide 3DGS. It is reported that the proposed method achieves state-of-the-art performance. Visual results show fine-grained hand and facial details.", "questions": "1. What are the specific requirements for videos in terms of the method, such as the length of the video and the range of camera angles?\n2. How does the method perform when the magnitude of the action being driven is out-of-distribution?\n3. What is the inference speed of the method?", "limitations": "This paper has involved discussions about limitations and broader impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Generating the human upper body videos is a critical task. It is more expressive and more challenging than facial generation. This method overall demonstrates fine-grained generation details.\n2. The existing SMPL-X model has alignment issues when estimating the human body. The optimized SMPL-X model in this method significantly improves the alignment between SMPL-X and the RGB frames.\n3. The paper is overall easy to follow, and the experimental results demonstrate the method's superiority in numerical metrics.", "weaknesses": "1. In the paper, the confidence scores for CL are all learned from the rendered results. How can the effectiveness of this confidence be ensured? In Table 2, the face area without CL also shows better results.\n2. [11] also demonstrates photorealistic 3D human body generation, and the authors need to include more discussions with it.\n3. The paper has many writing issues, such as a too-brief related work section, inconsistent formula definitions (e.g., $\\Pi_K$ in Eq. 11), and some missing punctuation in formulas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: How can the effectiveness of this confidence be ensured?**\n\nA1: The predictor could leverage the information contained in rendered RGB and depth to adaptively learn the confidence in an end-to-end data-driven manner. The experimental results have validated the effectiveness of CL and we also present some visualization in Figure 3 of the rebuttal pdf. Another benefit of CL is to reduce the number of Gaussians from 21,038 to 19,993.\n\n**Q2: GaussianAvatar also demonstrates photorealistic 3D human body generation, and the authors need to include more discussions with it.**\n\nA2: GaussianAvatar is one of the pioneering animatable 3D Gaussian models which is learned from a monocular RGB video. Its representation is further enhanced via two key components for final photorealistic quality. Dynamic properties are designed to support pose-dependent appearance modeling, while joint optimization of motion and appearance helps tackle inaccurate motion estimation. We will add this discussion in related work.\n\n**Q3: Suggestion on revising some writing issues, such as a too-brief related work section, inconsistent formula definitions (e.g., in Eq. 11), and some missing punctuation in formulas.**\n\nA3: Thanks for pointing out these issues. We will discuss more works in the related work section, e.g. GaussianAvatar, fix the inconsistent formula definitions, and add the missing commas in Eq. 7, 8, 9, 10 and 13. We will incorporate these modifications in the revised manuscript.\n\n**Q4: Specific requirements for videos in terms of the method.**\n\nA4: Since our method is reconstruction-based, our main requirement is that the areas of interest should be captured in the video. For example, for the full-body X_Human dataset, the monocular RGB video includes observations around the person of interest. On the other hand, for the upper-body UPB dataset, since the area of interest for sign language is only on the upper body, we only need to capture the corresponding appearance of the person. We note that we keep the input RGB frames of all comparison methods consistent for fair comparison.\n\n**Q5: How does the method perform when the magnitude of the action being driven is out-of-distribution?**\n\nA5: It is hard to quantify the magnitude of out-of-distribution actions in a principal way (although if the reviewer would like to indicate a specific metric/evaluation, we would be happy to explore it). With that being said, on the UPB benchmark for instance, our evaluation is conducted on novel poses that are not used during training, so our evaluation indicates the performance on the novel pose setting. Moreover, qualitatively, we demonstrate the results of our avatar driven by an in-the-wild SMPL-X sequence with unseen poses and unseen identities. By inspecting the results, the expressive details are quite faithful. However, if the motion used at test time is completely different, it is likely we will observe artifacts in the reconstruction.\n\n**Q6: Inference speed of the method.**\n\nA6: Our method could render the 1080p image with the inference speed of 361.02 fps on one RTX A5000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0ZMjUd5j5", "reviewer": "Reviewer_ZKQ6", "summary": "The paper focuses on improving the expressiveness of digital human avatars, particularly through detailed hand and facial expressions, learned from monocular RGB video. The main contributions are:\n- SMPL-X Alignment improves the alignment of the SMPL-X model with RGB frames and aids in accurately recovering avatars. \n- Context-Aware Adaptive Density Control Strategy adjusts gradient thresholds to handle the varied granularity across different body parts, enhancing the expressiveness of the avatars.\n- Feedback Mechanism that predicts per-pixel confidence to better guide the learning and optimization of 3D Gaussians.\n\nOverall, the paper presents a comprehensive framework that enhances the realism and expressiveness of digital human representations, validated by substantial quantitative and qualitative experiments.", "questions": "- **Project Page for Visualizations:** Is there a project page available that shows visualizations of the rendered method, such as a 360-degree rotating video of the generated person and animation to better illustrate the quality? Specifically, could visualizations zoom into detailed areas such as the hands and face?\n- Is e and λt in Equation 14 learned or fixed? If learned, it would be advantageous to list the values to demonstrate that the learned parameters effectively handle different resolution details for different body parts.\n- Discussing failure cases and outlining potential future work would provide a more comprehensive view of the method's limitations and areas for improvement.\n- Information on the optimization time compared to previous methods and the resource requirements for running the proposed method would be valuable for assessing its efficiency and scalability.", "limitations": "While the use of CADC is interesting, the other two contributions are of limited novelty. SMPL-X usage and alignment have been incorporated by previous works. The improvement of CL is not significant. The addition of project page for visualizations would be helpful to demonstrate the improvements made by the methods. In addition, more quantitative results should be added to Tables 1, 2 and 3.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is written in a clear and understandable manner. Comprehensive ablation studies and clear visualizations help to demonstrate the effectiveness of different components of the method. The paper also compares the new method with previous methods and shows that it achieves better performance. It is commendable that the authors added SMPL-X to existing methods for fair and thorough comparisons.", "weaknesses": "- One of the contributions of using and optimizing SMPL-X for expressive avatar is not novel. This has been done in previous works [1]. \n- The paper lacks a comparison to recent methods such as Splatting Avatar [2]. Including this comparison in Table 1 can help to distinguish the improvement made by the proposed approach. \n- There is an inconsistent ablation study across the datasets. Table 2 should illustrate how the alignment affects the baseline method by comparing scenarios with and without CADC, CL, and alignment. Similarly, Table 3 should also show the method with CADC and CL included.\n- PSNR, SSIM, and LPIPS should be calculated from four novel views to ensure the robustness of the method across different perspectives. Additionally, animation and novel pose metrics can be included to display differences between methods.\n- From Table 2, it seems that CL does not have a significant impact. This should be addressed or clarified to understand its role in the framework. It would be good to provide more qualitative visualization to demonstrate its effectiveness. \n\n\n[1] (Liu et al., 2024) GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos. \n\n[2] (Shao et al., 2024) SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: One of the contributions of using and optimizing SMPL-X for expressive avatar is not novel. This has been done in previous works GVA.**\n\nA1: GVA is a concurrent work and has not been accepted. Compared with GVA, our fitting design is quite different, since ours explicitly focuses on the fine-grained areas (especially on hand) to meet the requirement of expressiveness. Considering GVA is not open-sourced, we are not able to compare with it. However, we have visualized that our fitting method outperforms the current SOTA SMPL-X estimation method in both Figure 4 (main paper) and Figure 1 (rebuttal pdf).\n\n**Q2: More comparison with recent methods such as SplattingAvatar.**\n\nA2: Thanks for this suggestion. The results of SplattingAvatar are shown in A1 of General Response. We will add these experiment results in Table 1.\n\n\n**Q3: Clarification on inconsistent ablation study across the datasets.**\n\nA3: Table 2 demonstrates the ablation results on the XHumans dataset. For XHumans, we directly utilize its provided accurate SMPLX annotation. That is to say, we do not need the SMPL-X fitting involved. For Table3, we further update the ablation results.\n\n| **Method** | **PSNR (Full)** | **SSIM (Full)** | **LPIPS (Full)** | **PSNR (Hand)** | **SSIM (Hand)** | **LPIPS (Hand)** | **PSNR (Face)** | **SSIM (Face)** | **LPIPS (Face)** |\n|-------------------|--------------------|--------------------|-----------------------|--------------------|--------------------|-----------------------|--------------------|--------------------|-----------------------|\n| w/o Align  | 25.02 | 0.9435 | 73.82 | 24.64 | 0.9396 | 63.64 | 24.27 | 0.9009 | 93.56\n| w/o CADC   | 26.53 | 0.9504 | 68.76 | 26.43 | 0.9494 | 50.59 | 26.45 | 0.9265 | 71.17\n| w/o CL     | 26.74 | 0.9507 | 67.90 | 26.74 | 0.9507 | 48.54 | 26.69 | 0.9293 | 69.60 \n| EVA        | 26.72 | 0.9519 | 65.37 | 26.90 | 0.9523 | 46.11 | 26.75 | 0.9298 | 66.41 \n\n\n\n**Q4: PSNR, SSIM, and LPIPS should be calculated from four novel views to ensure the robustness of the method across different perspectives. Additionally, animation and novel pose metrics can be included to display differences between methods.**\n\nA4: We calculate PSNR, SSIM and LPIPS from different viewpoints. To our best knowledge, we cannot find any animation and novel pose metrics reported in relevant works (GART, GauHuman, SplattingAvatar, etc.)\n\n**Q5: Clarification on CL role in the framework.**\n\nA5: We observed that CL helped with perceptually improving the results, which is also reflected in the consistent performance improvement on the LPIPS metrics. Moreover, another benefit of CL is that it led to a more compact representation (e.g., reducing the number of Gaussians from 21,038 to 19,993). We further present some qualitative results (w/o and w confidence-aware loss) in Figure 3 of the rebuttal pdf.\n\n**Q6: Project page for visualization including a 360-degree rotating video, animation, and zooming.**\n\nA6: Due to the NeurIPS policy (\"all the texts you post (rebuttal, discussion and PDF) should not contain any links to external pages\"), we are unable to provide the external link to our built project page. In fact, we have included 360-degree rotating video (Supp video) and animation (Supp video) and zooming results (both Supp video and figures in main paper). We promise to release the project page afterward.\n\n**Q7: Is e and λt in Equation 14 learned or fixed?**\n\nA7: Both e and λt are fixed.\n\n**Q8: Discussion on failure cases and outlining potential future work**.\n\nA8: Thanks for pointing out this issue. For the fine-grained expressive areas, the failure cases happen when hand interaction exists, which is mainly caused by the false driven SMPLX signal (e.g. the driven SMPLX has implausible interpenetration between hands). From the holistic view, some 'floaters' may exist sometimes, where general 3DGS modeling also exists.\n\nWe outline the potential future works as follows,\n\n1) Modeling capability on non-rigid elements, such as loose cloth (dress). Modeling cloth on the avatar has been a challenging topic, even separately studied in several works. A potential solution could be parameterizing the cloth deformation or adding more prior on cloth type, so as to provide more driving signals.\n\n2) Generalizable human avatar from monocular RGB video. Current methods in this topic mostly need per-subject optimization, which needs to be re-trained to any new given subjects. It is worth exploring if we could get an avatar from a monocular RGB video of any given subject with a single feed-forward pass.\n\n3) Robustness. It is worth exploring if we could build the human avatar well with more limited source inputs, e.g. a few images with even mutual occlusion, a single image, etc.\n\n**Q9: Information on the optimization time compared to previous methods and the resource requirements.**\n\nA9: For the optimization time, we generally divide it into two stages. Take a real-world 1080p video as an example, the first stage (data preprocessing) needs 10.2 minutes, while the second stage (avatar modeling) needs 7.5 minutes on RTX A5000. Compared with the avatar modeling stage of EVA, 3DGS+SMPLX needs 7.5min, GART+SMPLX needs 12.9 min, GauHuman+SMPLX needs 4.5min, and SplattingAvatar needs 45.2 min. The resource requirement could be further lowered down to consumer GPU cards like 3090, but at the expense of speed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**

We sincerely appreciate the reviewers for their insightful and constructive comments. We are encouraged by their positive comments and the recognition of the merits of our work. More specifically, the reviewers have appreciated the great importance and challenges introduced by expressiveness in human avatar modeling (R-1sDc, R-fFxo, and R-g7vJ). Moreover, they have highlighted the novelty (R-fFxo, R-4SGk, and R-g7vJ) and strong performance (R-fFxo, R-1sDc, R-g7vJ, and R-ZKQ6) of the proposed EVA framework for capturing expressiveness. Finally, the reviewers have positively commented on our comprehensive experiments (R-fFxo, R-1sDc and R-ZKQ6), the clear visualizations (R-fFxo, R-ZKQ6, and R-g7vJ), and the well-written and easy-to-follow paper (R-1sDc, R-fFxo, R-g7vJ and R-ZKQ6).

In the following, we first address the common concern, and then the concerns of each reviewer. We will revise the manuscript accordingly.

**Q1: Suggestion on including more comparison methods.**

A1: We have used the Splatting Avatar method from CVPR 24 and evaluated it on the datasets we used. The results are shown below. Our approach outperforms Splatting Avatar.
| **Method** | **Dataset** | **N-GS**   | **PSNR (Full)** | **SSIM (Full)** | **LPIPS (Full)** | **PSNR (Hand)** | **SSIM (Hand)** | **LPIPS (Hand)** | **PSNR (Face)** | **SSIM (Face)** | **LPIPS (Face)** |
|------------|-------------|------------|-----------------|-----------------|------------------|-----------------|-----------------|------------------|-----------------|-----------------|------------------|
| SplattingAvatar   |  UPB     | 257,811  | 25.13           | 0.9355          | 96.16            | 24.20           | 0.9298          | 70.91            | 24.48           | 0.8962          | 127.63           |
| EVA |  UPB     | 20,829  | 26.78           | 0.9519          | 65.07            | 27.00           | 0.9524          | 45.90            | 26.85           | 0.9298         | 65.90           |
| SplattingAvatar   | XHumans | 103,193  | 29.33           | 0.9606          | 44.39            | 26.19           | 0.9264          | 78.53            | 26.47           | 0.9103          | 92.51            |
| EVA | XHumans | 19,993 | 29.67 | 0.9632 | 33.05 | 26.27 | 0.9279 | 72.95 | 26.56 | 0.9157 | 72.30 |

We will add the discussion on this paper in related work and the experiment results in Table 1.

Meta-review: This paper proposes a new method called EVA for expressive 3D human avatar creation from monocular videos. To solve this task, the work proposes three innovations: improved SMPL-X estimation, context aware densification of Gaussians and context aware pixel-level feedback, which all improve quality. The method is evaluated on the X-Humans dataset, but more importantly on a new challenging UPB dataset of in-the-wild videos curated from the Internet. The proposed method shows clear improvements over the existing state of the art methods. Reviewers, appreciated the importance of the challenging problem addresses, and the novelty and good results of the proposed method. They initially raised concerns about the lack of comparisons to some baselines and the lack of ablations. The reviewers' concerns were addressed during the rebuttal phase and all reviewers were in agreement in recommending acceptance of this work. The AC concurs and recommends acceptance. The authors should include the changes that they have promised in the rebuttal into the final manuscript.
Decision: Accept (poster)