Expressive Gaussian Human Avatars from Monocular RGB Video

Authors: hezhen  hu
zhiwen  fan
tianhao  wu
yihan  xi
seoyoung  lee
georgios  pavlakos
zhangyang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 06:07:40

Abstract: Nuanced expressiveness, especially through detailed hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations.
In this work, we aim to learn expressive human avatars from a monocular RGB video; a setting that introduces new challenges in capturing and animating fine-grained details.
To this end, we introduce EVA, a drivable human model that can recover fine details based on 3D Gaussians and an expressive parametric human model, SMPL-X.
Focused on enhancing expressiveness, our work makes three key contributions.
First, we highlight the importance of aligning the SMPL-X model with the video frames for effective avatar learning.
Recognizing the limitations of current methods for estimating SMPL-X parameters from in-the-wild videos, we introduce a reconstruction module that significantly improves the image-model alignment.
Second, we propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds to accommodate the varied granularity across body parts.
Third, we develop a feedback mechanism that predicts per-pixel confidence to better guide the optimization of 3D Gaussians.
Extensive experiments on two benchmarks demonstrate the superiority of our approach both quantitatively and qualitatively, especially on the fine-grained hand and facial details. 
We make our code available at the project website: https://evahuman.github.io.

Main paper: 
 introduction: 
 High-quality digital avatar modeling has a wide range of applications, including AR/VR, movie production, sign language, and more. For digital human representation, capturing nuanced expressions is essential for enhancing fidelity and vitality. This is particularly evident in the detailed portrayal of hands and facial expressions, which add emotional depth and interactive expression capabilities to human avatars. In this work, we investigate expressiveness when building human avatars from monocular video. The task involves taking as input a monocular human video and learning an animated human avatar which enables multiple capabilities, such as novel view and pose synthesis.Reconstructing expressive human avatars is challenging, particularly due to the subtle and complex movements of the hands and face. Compared to the body, hands and faces occupy smaller spatial areas and have distinct characteristics. For example, the hand has many degrees of freedom, intricate textures, and frequent self-occlusions. To achieve accurate avatar modeling, it is crucial to capture these fine textures from RGB video and ensure effective animation.Current studies [49, 31, 15, 22, 12] mainly focus on learning human avatar on the body region and have made remarkable progress. Early works [49, 31, 15] mainly utilize NeRF as an implicit representation but usually have the drawback of low training/inference speed. Recently, more and more works build on top of 3D Gaussian Splatting [19] for its effectiveness and efficiency, which could further speed up rendering to over 100fps. GART [22] utilizes a mixture of moving 3DFigure 1 : Learning expressive Gaussian human avatars from RGB video. Given a monocular video as input, our proposed approach, EVA, learns an expressive 3D Gaussian avatar. The results outperform the SOTA method [22] on novel pose synthesis, especially for the hand and facial details.Gaussians to approximate human geometry and appearance and enhances fine details with learnable forward skinning and latent bones. GauHuman [12] proposes a new density control strategy, e.g. split and clone with KL divergence and a new merge operation. However, these methods do not consider the fine-grained details of hand and face, which cannot meet the requirement of expressiveness.In this work, we introduce EVA, a drivable human model that can represent expressive details using 3D Gaussians and a human parametric model. Given a monocular RGB video, we extract the pose and mask information corresponding to each frame, which allows us to map each frame's human observation to a canonical space. Once the avatar is constructed, it can be animated using linear blend skinning given a new pose, followed by rendering to the 2D human image. To tackle new challenges introduced by expressiveness, we start by solving the misalignment issue between the SMPL-X model and the RGB frames via a reconstruction module. By employing a fitting-based optimization, this module can produce more reliable 3D SMPL-X reconstruction, providing a more robust foundation for digital avatar modeling.Considering the granularity differences across different body parts, we propose a context-aware adaptive density control strategy for 3D Gaussian optimization. It leverages attributes specific to different parts and historical gradient information, to adaptively control Gaussian density. Furthermore, to improve the Gaussian optimization, we design a feedback mechanism, which adaptively predicts confidence scores based on the rendering result, thereby ensuring the supervision signal effectively transferring to the corresponding Gaussian. For evaluation on expressiveness, we build comparison baselines from related body avatar methods with a few modifications and collect a new benchmark called UPB containing in-the-wild upper body videos. The image quality is evaluated with multiple metrics, separately on the full body, as well as the hand and face regions. Our approach, as shown in Figure 1 , largely outperforms previous state of the art, with more fidelity in fine-grained details.Our contributions are summarized as follows:• We introduce EVA, an approach that can build expressive human avatars based on 3D Gaussians, given as input a monocular RGB video. Through extensive experiments on two datasets, we demonstrate the superiority of EVA, particularly on the hand and facial details.• To enhance expressiveness, we propose a context-aware adaptive density control strategy to accommodate the granularity differences across human parts, followed by a feedback mechanism to better guide the 3D Gaussian optimization. • To handle challenging in-the-wild videos, we propose a reconstruction module, which critically improves the SMPL-X alignment to the RGB frames compared to off-the-shelf methods. We demonstrate the importance of this improvement in recovering accurate avatars.2 Related Work 
human avatar modeling from monocular rgb video: 
 The problem of building a human avatar from a monocular RGB video is challenging due to the dynamic nature of the capture and the partial observations for each frame. Early works [31, 2, 37, 38, 3, 15, 49, 46, 39, 42, 6, 21, 41] mainly resort to the combination of implicit neural representations (e.g., NeRF [27] ) and parametric models to represent a human avatar with high fidelity and flexibility.To address the slow computation in implicit models, previous work has proposed various techniques to reduce training [10, 14] or inference time [33, 25, 5] . With the introduction of 3D Gaussian Splatting [19] which can achieve both fast and high-quality rendering, an increasing number of works [22, 24, 11, 12, 20, 23, 32, 13, 18] utilize it as their base representation, jointly with parametric human models like SMPL [26] . More specifically, GaussianAvatar [11] enhances 3D Guassians via two key components for final photorealistic quality, i.e., dynamic properties and joint optimization of motion and appearance. GauHuman [12] leverages human prior and KL divergence to propose a new density control strategy. In this work, we systematically explore expressiveness while building human avatars from a monocular RGB video and tackle the new challenges brought by expressiveness.Expressiveness plays a vital role in non-verbal communication, which, besides the body, particularly involves the hand and face regions [44, 48, 51, 8, 9, 28, 16, 7, 35] . To represent expressive humans, various parametric models have been proposed, including SMPL-X [28] , Adam [17] , GHUM [43] , and more. These models usually have predefined topology and can provide a compact mapping from the low-dimensional embedding to the 3D mesh. However, the predefined topology limits their capability of depicting fine-grained texture. Recent works build human avatars via neural 3D representations while leveraging the parametric model to provide human shape priors and animation signals. For example, X-Avatar [35] combines NeRF and SMPL-X to build the avatar. It further proposes part-aware sampling and initialization strategies to ensure efficient learning from highquality 3D scans or RGB-D data. GVA [24] integrates the SMPL-X model to improve the rendering quality. AvatarRex [50] proposes a compositional avatar representation to separately model body, hand and faces from multi-view RGB video data. Different from them, we aim to relax the input requirements, by building expressive human avatars from a real-world monocular RGB video. 
technical approach: 
 In this section, we first introduce the preliminary knowledge on the SMPL-X human model [28] and 3D Gaussian Splatting [19] and present the general framework design on articulated human modeling. Then, we elaborate on the key technical contributions of our approach.SMPL-X body model. SMPL-X [28] is a parametric human body model, which extends the SMPL body model [26] by modeling both hand articulation and facial expressions. SMPL-X can be defined as a mapping function M (θ θ θ, β β β, ψ ψ ψ) :R |θ θ θ| × R |β β β| × R |ψ ψ ψ| → R 3N, where θ θ θ, β β β, ψ ψ ψ are the parameters for pose, shape, and facial expression, respectively. The function of SMPL-X is formulated as follows:EQUATIONEQUATIONwhere B P (•), B S (•) and B E (•) denote pose, shape, and expression blend functions, respectively, and W is a set of blend weights. The pose, expression and shape corrective blend shapes, i.e., B P (θ θ θ), B E (ψ ψ ψ) and B S (β β β), add corrective vertex displacements to the template human mesh T. After that, linear blend skinning W (•) is applied to rotate the vertices in the template mesh around the joints J(β β β), smoothed by the blend weights W. This generates the final human mesh. 3D Gaussian Splatting. Methods based on NeRF [27] model the scene with an implicit representation and render novel views using volume rendering. In contract, 3D Gaussian Splatting [19] (3DGS) models a 3D scene with a set of discrete 3D Gaussians and performs rendering through a tile-based rasterization operation, which can reach real-time rendering speeds. Specifically, each Gaussian is defined with its central position p, 3D covariance matrix Σ as follows:EQUATIONwhere x is an arbitrary position in the 3D scene. The covariance matrix Σ is decomposed into two learnable components to make the optimization easier, Σ = RSS T R T , where R and S denote the rotation matrix and scaling vector, respectively. During rendering, each 3D Gaussian G(x) is first transformed to a 2D Gaussian G ′ (x) on the image plane. Then, a tile-based rasterizer is designed to efficiently sort the 2D Gaussians and employ α-blending:EQUATION)where r is the queried pixel position and N denotes the number of sorted 2D Gaussians associated with the queried pixel. c i and α i denote the color and opacity of the i-th Gaussian, which are modeled by Spherical harmonics.Articulated 3D human modeling. Our formulation takes inspiration from works employing 3D Gaussians for modeling articulated objects [12, 22] . We optimize 3D Gaussians in a canonical space, which corresponds to a human in a rest pose [15] . The Gaussians are transformed from the canonical space to the frame space via linear blend skinning (LBS):EQUATIONEQUATIONwhere p f , Σ f , p c , Σ c are the Gaussian mean and covariance in frame space and canonical space, respectively.G(J f , θ f ) = K k=1 w k G k (J f , θ f ), b(J f , θ f , β f ) = K k=1 w k b k (J f , θ f , β f) are the rotation and translation, respectively, with respect to the K joints, andG k (J f , θ f ), b k (J f , θ f , β f )are the rotation and translation, respectively, with respect to the k-th joint. w k is the LBS weight.To perform Linear Blend Skinning, we need two important components, i.e., the LBS weights and the input pose parameters. Learning LBS weights w k from scratch would be inefficient and can lead to a local optimum in the early stage of the training. Therefore, for each Gaussian, we start with the LBS weight of the nearest SMPL-X vertex and use an MLP f Θw to predict an LBS weight offset w ′ k using the positionally encoded [27] Gaussian centers γ(p c ). w k is therefore defined as:EQUATIONEQUATIONwhere w SMPL-X k is the LBS weight of the nearest SMPL-X vertex. We set ϵ = 10 -8 .Starting from the input pose θ SMPL-X , we further fine-tune it via an MLP-based network f Θ θ , which is jointly optimized during the 3D Gaussian optimization process. The actual poses θ used for optimization and rendering are therefore obtained as follows:EQUATIONwhere ⊗ represents the vector pointwise product.A key requirement for learning accurate human avatars is to initialize the optimization process with a reliable SMPL-X estimate. However, this can be challenging, particularly in real-world cases, Figure 2 : Overview of our proposed EVA method. Given a monocular RGB video, first we estimate a SMPL-X mesh that aligns well to the video frames using a reconstruction module. Then, EVA utilizes 3D Gaussian Splatting for avatar modeling, while inheriting the human shape prior from the SMPL-X model. To improve the optimization and the quality of the avatar, we propose context-aware adaptive density control and a confidence-aware loss.given the limitations of current off-the-shelf methods [29] for SMPL-X fitting. This motivates us to propose a robust fitting procedure that leverages multiple sources, including the initial estimation of camera parameter, SMPL-X parameters, 2D keypoints and 3D hand parameters from off-the-shelf tools [30, 45, 1] . Given these initial estimates, the SMPL-X fitting procedure minimizes the following objective:EQUATIONThe 2D keypoint term, L 2D , encourages the projection of the 3D keypoints of the avatar to align with the detected 2D keypoints. It is formulated as follows:EQUATIONwhere Π K (•) represents the camera projection under the given camera intrinsic parameter K. ψ(•) is the robust Geman-McClure error function [36] which helps prevent the disturbance from noisy 2D keypoint detections. R θ θ θ (•) denotes the function which rotates the joints J(β β β) given the pose θ θ θ. J 2D is the detected 2D keypoints from [45] . γ and ω represent the predefined weighting parameters and the detection confidences respectively.Our prior terms contain two components which focus on the coarse-grained body and fine-grained hand, respectively. For the body part, we utilize VPoser [28] to filter infeasible body poses. VPoser provides a compact mapping from a low-dimensional embedding η to the rotation matrices of the body pose θ b . To better optimize the low-dimensional embedding η, the estimated 3D body joints J 3Db [1] is treated as guidance, together with the added regularization term. The body prior loss term, L bp , is formulated as follows:EQUATION)Similary, for the hands, we utilize the 3D hand joints J 3D h estimated from [30] to provide a better hand spatial relationship. This is formulated as follows:EQUATIONwhere the index h represents the joints corresponding to the hand. ψ z (•) indicates that the robust error function only considers the coordinates along the z-axis coordinates for the error calculation.The core of the Gaussian Splatting optimization is adaptive density control, which generally contains two operations, i.e., densification (split and clone) and pruning. The original strategy [19] selects a fixed constant as the densification criteria. However, utilizing the fixed threshold does not leverage context information, leading to sub-optimal 3D Gaussian representations.To this end, we propose a context-aware adaptive density control that leverages part attribute and history gradient information. Each Gaussian inherently possesses attributes associated with different body parts. These body parts vary in spatial size and characteristics. For instance, the hand is much smaller in size, compared to the body, exhibiting fine-grained textures and many degrees of freedom. These characteristics also lead to inherent differences in the Gaussian positional gradient changes across different body parts during the Gaussian optimization process. Furthermore, a continuous increase in the Gaussian positional gradient indicates that Gaussians need to be densified.Specifically, we first initialize the 3D Gaussians with the vertices of the SMPL-X model. Since SMPL-X has a predefined topology, this initialization provides each Gaussian with the attribute information U on which part it belongs, e.g. body, hands or face. After that, considering attributes and gradient history information, the densification threshold for the i-th Gaussian in a certain attribute U is formulated as follows:EQUATION)where e is a constant, R represents the densification interval, and ∇ k denotes the position gradient of i-th Gaussian at the timestamp k. Note that e and λ t have different values for different attributes. For the pruning strategy, we remove the points that are far away from the SMPL-X template vertices.The whole framework is optimized under the objective functions as follows:EQUATIONwhere λ m , λ s , and λ l are loss weighting terms. The mask loss L m calculates the consistency between accumulated volume density and the estimated mask. The SSIM [40] loss L SSIM is adopted to improve the structural similarity between rendered image and input image. The LPIPS [47] loss L LP IP S focuses on the perceptual quality of the rendered image.Confidence-aware loss L c . It is inevitable that any training video will introduce some form of noise (e.g., misalignment, motion blur), which will also affect the avatar optimization procedure. To address some of the effects of the noise, we introduce a feedback module to decide which pixels should be taken into consideration with higher or lower weight during training. This module takes as input the rendered image I r and rendered depth D r , and predicts a score for each pixel which represents the confidence value. More specifically:EQUATIONwhere µ is a constant. Then the confidence serves as an adaptive weighting factor on the per-pixel consistency. Eventually, we formulate our confidence-aware loss as follows:EQUATION 
experiments: 
 In this section, we first introduce our experimental setup, including datasets, implementation details and evaluation metrics. Then, we make comparisons with baseline methods both quantitatively and qualitatively. Finally, we perform ablation studies on the most important components of our approach.Datasets. The experiments are conducted on two datasets, XHumans [35] and our collected UPB dataset. XHumans is a dataset captured in a controlled environment. It provides images with resolution of 1200 × 800, along with well-aligned SMPL-X meshes. There are 6 identities (3 male and 3 female) for evaluation. UPB consists of sign language videos from the web, which usually contain complicated hand gestures. The resolution of the videos is 1920 × 1080 and they do not contain any ground truth SMPL-X annotations. UPB includes 4 identities (2 male and 2 female). Implementation details. Our framework is implemented on PyTorch and all experiments are performed on NVIDIA A5000. The hyperparameter λ m , λ s and λ l are set to 0.1, 0.01 and 0.04, respectively. We use SGHM [4] to extract the human mask. The 3D Gaussian optimization lasts for 2,000 iterations with the densification performed between iteration 400 and 1,000. For other parameters, we follow the original settings of [19] . Our method can render a 1080p image with an inference speed of 361.02 fps on one RTX A5000. Since our method is reconstruction-based, our main requirement is that the areas of interest should be captured in the video. Note that we use the same input RGB frames for all methods to ensure a fair comparison. More details are presented in the Appendices.Evaluation metrics. We evaluate the avatar quality through the rendered views following previous works [22, 12] . We adopt the widely-used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [40] , and Learned Perceptual Similarity (LPIPS) [47] to evaluate the rendered images. For a more fine-grained evaluation, we report the above metrics separately for the full body, for the hands region and for the face region.The baselines we use for comparison are adopted from related methods with a few modifications. 3DGS + SMPL-X is modified from 3DGS [19] . We add our articulated human modeling mechanism to make it fit the task requirement. GART and GauHuman are originally designed to model body-level avatars. They are animated via the SMPL parametric model, which lacks the capability of modeling hand articulation and facial expressions. Therefore, we replace the driven signal with the SMPL-X model and denote them as GART + SMPL-X and GauHuman + SMPL-X, respectively. Splatting selects the triangle mesh as the driving signal and we utilize SMPL-X mesh in Splatting + SMPL-X. Among these baselines, GauHuman is closer to our approach. Our differences are mainly reflected in the adaptive density control strategy, objective functions and the SMPL-X alignment for the real-world videos, which are also consistent with our main technical contributions.As shown in Table 1 , we conduct experiments on the XHumans and UPB datasets to evaluate the effectiveness of our method. XHumans is captured in a controlled setting with accurate SMPL-X ground truth, while UPB focuses on real-world video without pose information. Since it is hard to get accurate SMPL-X annotation for real videos, UPB is more challenging and more representative of learning a human avatar in the wild. We observe that our method achieves state-of-the-art performance on these two datasets. We note that we do not apply our designed alignment module in XHumans, since it already contains SMPL-X ground truth. Our method achieves 19.7%, 17.3% and 22.5% relative LPIPS gain on the full, hand and face regions, respectively. For the real-world UPB dataset, our performance gain is much larger, achieving over 25% relative LPIPS gain on the hand region. This indicates that EVA, unlike previous work, handles well the challenges of real-world videos. The qualitative comparison in Figure 3 also validates the effectiveness of our method. We perform ablation experiments to highlight the important components of EVA, i.e., context-aware adaptive density control, confidence-aware loss and in-the-wild SMPL-X alignment module.Effectiveness of context-aware adaptive density control (CADC) and confidence-aware loss (CL). These two components are designed for better optimization of Gaussian Avatar Modeling. As shown in Table 2 , "w/o CADC" means that we replace it with the original density control [19] , while "w/o CL" denotes removing the confidence weighting term in the RGB loss calculation. We observe that these two components both improve performance. More specifically, CADC improves performance for the metrics of all regions. Meanwhile, we observed that CL brings another benefit of a more compact representation (e.g., reducing the number of Gaussians from 21,038 to 19,993).Effectiveness of in-the-wild SMPL-X alignment. We use our SMPL-X alignment module for real-world videos without accurate SMPL-X ground truth, so we conduct the corresponding ablation study on UPB dataset. As shown in Table 2 , "w/o Align" means that we directly utilize the SMPL-X mesh extracted from current SOTA estimation method [1] . Notably, our proposed alignment module brings notable performance gains on all metrics of all regions. It also demonstrates the importance of the SMPL-X pose quality for final human avatar modeling. Without accurate SMPL-X alignment, the following Gaussian model will need to deal with inconsistent textures for a specific region across frames, leading to lower avatar quality. Furthermore, we visualize the SMPL-X alignments in Figure 4 and present qualitative comparison with state-of-the-art estimation methods in Figure 5 , including fitting-based SMPLify-X and regression-based SMPLer-X. Visible improvements are clearly observed specifically for the hands. We can estimate a SMPL-X mesh that aligns well with the RGB frame, especially for the fine-grained hand regions.In summary, we apply EVA on a variety of monocular RGB videos, including both controlled captures and videos collected from the Internet. The visual data contain large body motions (e.g., playing basketball, weight lifting, dancing), along with fine-grained motions (e.g. finger counting, using tools, sign language). EVA can handles well self-occlusions, which usually exist in every frame, especially for the hand areas. This can be attributed to our proposed SMPL-X fitting method which provides reliable correspondences from the pixels to the canonical space. Besides, we expect that EVA could also handle low lighting conditions (as long as the lighting is not changing significantly), since our SMPL-X fitting method should be robust to low lighting conditions. Given accurate SMPL-X estimates, we expect the avatar learning stage will perform well.Limitations. The focus of this work is to capture expressive human avatars, by designing a pipeline that can take raw web videos as the input. Besides capturing the expressive details, other factors are challenging, yet worth investigating to further improve the quality of the avatar, e.g. non-rigid elements like clothes and hair. It is also worth extending our avatar reconstruction to more challenging input sources, e.g. occlusions or few-shot scenarios. Failure Cases. We show some representative examples of our failure cases in Figure 6 . For the fine-grained expressive areas, the failure cases happen when the video includes hand interactions. The failure is mainly caused by incorrect SMPL-X reconstruction (e.g., the estimated SMPL-X model that drives the reconstruction has implausible interpenetration between hands). Another failure case is the existence of floaters in the reconstruction which is a common issue of 3DGS modeling.Figure 5 : Comparison among methods for SMPL-X alignment. We compare our results with a regression-based method (SMPLer-X), and a fitting-based method (SMPLify-X).Broader Impact. In this work, our proposed framework helps push the boundaries of what is possible with human avatar modeling, allowing for more expressive results than previously achievable. However, with these advancements come certain risks, particularly concerning the potential misuse of the technology for deceptive practices or harassment. By advancing SOTA, our work makes these potential risks more pressing, as the barrier to creating such deceptive or harmful content is lowered. This highlights the need for careful consideration of the ethical implications and the development of strategies to mitigate these risks. 
conclusion: 
 In this work, we present EVA, a drivable expressive human avatar learned from a real-world monocular RGB video. EVA is built on 3D Gaussians, in coordination with the human prior introduced by SMPL-X. To deal with the challenges brought by expressiveness, we first utilize a reconstruction module to solve the misalignment between SMPL-X model and RGB frames. During the optimization of the 3D Gaussians, we propose context-aware adaptive density control, which leverages attribute and historical gradient information to accommodate the varied granularity across body parts. A feedback mechanism is jointly designed to further guide learning. Extensive experiments on two benchmarks demonstrate that our method outperforms baselines both quantitatively and qualitatively, especially on the fine-grained hand and facial details. 
 
 Appendix: This technical appendices provides more details which are not included in the main paper due to space limitations. For more visualization results, please refer to the project webpage.Training/Testing split. For each identity in XHumans dataset, one video is selected as the training split, where the other videos are marked as testing. During training, we utilize all of the frames (150 frames). We sample 20 frames for each testing video with the sampling rate of 5. For UPB dataset, we uniformly sample the frames with the interval as 1 to split the training and testing frames. The number of training and testing frames are both 140.Additional implementation details. µ is set to 1. λ t is set as -9.0, -4.5 and -6.3 for body, hand, and face parts, respectively. We set e for body, hand, and face parts as 2e-4, 1e-4 and 1.4e-4, respectively. The feedback module E(•) consists of two 2D convolutional networks. For SMPL-X alignment, we utilize the L-BFGS optimizer with the Wolfe line search. The optimization contains three stages with different loss weighting factors. The first stage is designed to initialize the human body embedding to the initial estimation from the method [1] . Then the second stage mainly aims to get better spatial hand relationship leveraging the prediction from the method [30] . The third stage performs fine-tuning with more emphasize on the fine-grained hand and face regions.Comparison baselines. The baselines we use for comparison are adopted from related methods with a few modifications. 3DGS + SMPL-X is modified from 3DGS [19] . Since the original 3DGS does not have the capability of animation, we add the same articulated human modeling in Section 3.1 as ours. For fair comparison, it also utilizes the same optimization schedule as ours, which lasts 2,000 iterations with the densification performed between 400 and 1,000 iterations. GART and GauHuman are originally designed to model body-level avatars. They are animated via the SMPL parametric model, which lacks the capability of modeling hand articulation and facial expressions. Therefore, we replace the driven signal with the SMPL-X model and denote them as GART + SMPL-X and GauHuman + SMPL-X, respectively. Since Splatting is embedded on a triangle mesh, we utilize the SMPL-X human mesh as its driving signal and denote it as Splatting + SMPL-X. We do not modify the optimization schedules of these three methods.Future works. We outline the potential future works as follows,• Modeling capability on non-rigid elements, such as loose cloth (dress). Modeling cloth on the avatar has been a challenging topic, even separately studied in several works. A potential solution could be parameterizing the cloth deformation or adding more prior on cloth type, so as to provide more driving signals. • Generalizable human avatar from monocular RGB video. Current methods in this topic mostly need per-subject optimization, which needs to be re-trained to any new given subjects. It is worth exploring if we could get an avatar from a monocular RGB video of any given subject with a single feed-forward pass. • Robustness. It is worth exploring if we could build the human avatar well with more limited source inputs, e.g. a few images with even mutual occlusion, a single image, etc.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper proposes a new method called EVA for expressive 3D human avatar creation from monocular videos. To solve this task, the work proposes three innovations: improved SMPL-X estimation, context aware densification of Gaussians and context aware pixel-level feedback, which all improve quality. The method is evaluated on the X-Humans dataset, but more importantly on a new challenging UPB dataset of in-the-wild videos curated from the Internet. The proposed method shows clear improvements over the existing state of the art methods. Reviewers, appreciated the importance of the challenging problem addresses, and the novelty and good results of the proposed method. They initially raised concerns about the lack of comparisons to some baselines and the lack of ablations. The reviewers' concerns were addressed during the rebuttal phase and all reviewers were in agreement in recommending acceptance of this work. The AC concurs and recommends acceptance. The authors should include the changes that they have promised in the rebuttal into the final manuscript.

Reviews: review: paper_summary: paper_summary: - Given an RGB human video, the paper focuses on photorealistic reconstruction of the human in 3D using SMPL-X and Gaussian rendering.
- The main idea is to align the expressive SMPL-X with the video evidence followed by Gaussian Avatar modeling with better gaussian density control and loss. 
- Technical contributions: Context-aware adaptive density control, confidence-aware loss, SMPL-X alignment.
- Baselines: 3DGS, GART, GauHuman.
- Evaluation are done on two datasets, XHuman and UPB. Metrics: PSNR, SSIM, LPIPS.
- The results show that the proposed method EVA consistently outperforms the baselines.
strength_weakness: strength_weakness: - Weak technical contributions: In comparison to GauHuman, CVPR 2024, the proposed method EVA makes three changes. 1) Context-aware adaptive density (CADC), 2) Confidence-aware loss (CL), 3) SMPL-X alignment. CADC is a nice idea for human modeling, and is inspired by the original 3DGS paper. However, it is a slight change to the originally proposed heuristic. CL is frequently used in the field of pose-estimation and 3D modeling (eg. DUST3R, CVPR 2024). SMPL-X alignment is very simply mesh fitting to 2D keypoints and is well explored in SMPLify-X, CVPR 2019 and its related works. Overall, the main technical ideas proposed lack technical novelty.

- XHuman over ZJU_MoCap for evaluations: Is there a reason why metrics were reported on XHuman and not ZJU-MoCap dataset? The missing metrics on the popular ZJU-MoCap benchmark makes it hard to fairly compare EVA with existing. I am guessing this because ZJU-MoCap does not have SMPL-X annotations. In this context, can we use the same setup as UPB and still evaluate using predicted SMPL-X parameters?
questions: questions: Given the lack of technical novelty, it is important to establish the empirical usefulness of the proposed method. 
In this context, I am asking for a fair evaluation with prior methods (mostly using SMPL) on popular benchmarks.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a solution to generate expressive human avatars from monocular RGB videos. The main focus is to improve the expressiveness. To this end, a few ideas such as combining 3d Gaussian splatting and SMPL-X (SMPL+parametric hands), minimizing 2d reprojection error,  finegrained density control, etc. 

There are a few similar papers in citation. To better understand the novelty of this paper, here's a high level comparison:
+ SMPLer-X [1]: Representation is SMPL-X only. Input is monocular RGB videos. Losses: SMPL-X keypoint losses.
+ GART[23]:  Representation is 3d Gaussians+SMPL. Input is monocular RGB videos.  Losses: photometric, perceptual.
+ 3DGS-Avatar [33]: Representation is 3d Gaussians+SMPL; Input is monocular RGB videos; Losses: photometric, perceptual, mask, etc; 
+ This paper: Representation is 3d Gaussians+LBS+SMPL-X. Input is monocular RGB videos. Losses: SMPL-X keypoint losses, perceptual.
strength_weakness: strength_weakness: The concept of model and image alignment by minimizing reprojection losses is not new. For example, the keypoints losses (Sec. 3.2) are very common in literature [1, 30]. 
Similarly mask loss and perceptual loss can be found in [33].

The adaptive density control part seems novel, as it's initialized from body parts and leveraging the fine-grained topology of SMPL-X.

The confidence-aware loss seems new but only provides marginal improvement as in Table 2.

Lack of hyperparameter ablation. Not even in supplementary. 

Minor issue: `video` is countable. Please explicitly use `videos` or `a video` in the paper. Actually it's not clear how many videos of one subject are used to train one avatar.
questions: questions: What's 'E' in Eq (16)? It looks like the so-called feedback module. But what's the design? Is it a learning-based predictor?

Please also address the novelty questions in the weakness part.
limitations: limitations: Limitations are discussed. It'd be good to discuss the failure cases.
paper_summary: paper_summary: The paper presents EVA, a model that can recover expressive human avatars from monocular RGB videos. The primary focus is enhancing fine-grained hand and facial expressions using 3D Gaussian Splatting and the SMPL-X model. EVA introduces three key contributions: a plug-and-play module for better SMPL-X alignment, a context-aware adaptive density control strategy, and a feedback mechanism for optimizing 3D Gaussian learning. Extensive experiments on two benchmarks demonstrate EVA's superiority in capturing detailed expressiveness compared to previous methods.
strength_weakness: strength_weakness: 1. The proposed method's complexity might pose implementation challenges for those not well-versed in the field. Simplifying some aspects or providing more intuitive explanations could help. 
2. While the method shows superior performance on the provided datasets, the generalizability to other types of monocular videos or environments is not thoroughly discussed.
3. Although the paper compares EVA with previous SOTA methods, it would be beneficial to include more diverse baselines or variants to demonstrate the robustness of the proposed approach.
questions: questions: 1. How does EVA perform with videos that have significant occlusions or low lighting conditions?
2. Are there any specific preprocessing steps required for the input monocular RGB videos?
3. Can the proposed method be extended to capture other expressive elements, such as cloth or hair movements?
limitations: limitations: 1. The performance heavily relies on the quality of the datasets used. Real-world applications might present challenges not covered by the current benchmarks.
2. The method requires significant computational power, which might limit its applicability in real-time scenarios or on devices with limited resources.
3. While the method focuses on hand and facial details, other expressive elements like body language, clothing, and hair dynamics are not addressed.
paper_summary: paper_summary: The paper introduces a 3DGS-based human avatar generation method from a monocular RGB video. Specifically, the authors first optimize the SMPL-X model to better align with the RGB frames. Then, they propose an adaptive adjustment method for different body parts and use per-pixel confidence to guide 3DGS. It is reported that the proposed method achieves state-of-the-art performance. Visual results show fine-grained hand and facial details.
strength_weakness: strength_weakness: 1. In the paper, the confidence scores for CL are all learned from the rendered results. How can the effectiveness of this confidence be ensured? In Table 2, the face area without CL also shows better results.
2. [11] also demonstrates photorealistic 3D human body generation, and the authors need to include more discussions with it.
3. The paper has many writing issues, such as a too-brief related work section, inconsistent formula definitions (e.g., $\Pi_K$ in Eq. 11), and some missing punctuation in formulas.
questions: questions: 1. What are the specific requirements for videos in terms of the method, such as the length of the video and the range of camera angles?
2. How does the method perform when the magnitude of the action being driven is out-of-distribution?
3. What is the inference speed of the method?
limitations: limitations: This paper has involved discussions about limitations and broader impact.
paper_summary: paper_summary: The paper focuses on improving the expressiveness of digital human avatars, particularly through detailed hand and facial expressions, learned from monocular RGB video. The main contributions are:
- SMPL-X Alignment improves the alignment of the SMPL-X model with RGB frames and aids in accurately recovering avatars. 
- Context-Aware Adaptive Density Control Strategy adjusts gradient thresholds to handle the varied granularity across different body parts, enhancing the expressiveness of the avatars.
- Feedback Mechanism that predicts per-pixel confidence to better guide the learning and optimization of 3D Gaussians.

Overall, the paper presents a comprehensive framework that enhances the realism and expressiveness of digital human representations, validated by substantial quantitative and qualitative experiments.
strength_weakness: strength_weakness: - One of the contributions of using and optimizing SMPL-X for expressive avatar is not novel. This has been done in previous works [1]. 
- The paper lacks a comparison to recent methods such as Splatting Avatar [2]. Including this comparison in Table 1 can help to distinguish the improvement made by the proposed approach. 
- There is an inconsistent ablation study across the datasets. Table 2 should illustrate how the alignment affects the baseline method by comparing scenarios with and without CADC, CL, and alignment. Similarly, Table 3 should also show the method with CADC and CL included.
- PSNR, SSIM, and LPIPS should be calculated from four novel views to ensure the robustness of the method across different perspectives. Additionally, animation and novel pose metrics can be included to display differences between methods.
- From Table 2, it seems that CL does not have a significant impact. This should be addressed or clarified to understand its role in the framework. It would be good to provide more qualitative visualization to demonstrate its effectiveness. 


[1] (Liu et al., 2024) GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos. 

[2] (Shao et al., 2024) SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.
questions: questions: - **Project Page for Visualizations:** Is there a project page available that shows visualizations of the rendered method, such as a 360-degree rotating video of the generated person and animation to better illustrate the quality? Specifically, could visualizations zoom into detailed areas such as the hands and face?
- Is e and λt in Equation 14 learned or fixed? If learned, it would be advantageous to list the values to demonstrate that the learned parameters effectively handle different resolution details for different body parts.
- Discussing failure cases and outlining potential future work would provide a more comprehensive view of the method's limitations and areas for improvement.
- Information on the optimization time compared to previous methods and the resource requirements for running the proposed method would be valuable for assessing its efficiency and scalability.
limitations: limitations: While the use of CADC is interesting, the other two contributions are of limited novelty. SMPL-X usage and alignment have been incorporated by previous works. The improvement of CL is not significant. The addition of project page for visualizations would be helpful to demonstrate the improvements made by the methods. In addition, more quantitative results should be added to Tables 1, 2 and 3.

Comments: title: Response to Authors
Thanks for the responses
comment: Thank you for recognizing our efforts in addressing your concerns. We feel very grateful that you will recommend our work for acceptance. We will revise the manuscript as you suggested.
We are glad that we could address your concerns and feel very grateful that you raised the score and recommended our work for acceptance. We will incorporate our responses in the revised manuscript.
Thank you for recognizing our efforts in addressing your concerns. We are glad that our responses have been helpful and will incorporate these responses into the revised manuscript as you suggested. We are always open to discussions that could further improve our paper. If there are no additional concerns, we would greatly appreciate it if you could kindly consider raising the rating.

Thank you again for your valuable comments.
Thank you for recognizing our efforts in addressing your concerns. We are pleased that our responses have been helpful and will incorporate the additional experimental results and responses into the main paper, as per your suggestions.

We are always open to further discussion that could help enhance our paper. If there are no further concerns, we would greatly appreciate it if you could kindly consider raising the rating.

Thank you again for your valuable input.
Thank you for the rebuttal. Appreciate your efforts on incorporating ZJU-Mocap. Overall, my concerns are addressed.
I am recommending the work for acceptance.
Thank you for your insightful feedback and for highlighting the important ethical and privacy considerations regarding the use of videos to train AI models. We appreciate the opportunity to address these concerns and outline our commitment to ethical research practices.

**Data Consent**

Note that our research is conducted for social good and is strictly only permitted for non-commercial use. We follow the fair use rule (a legal doctrine that says use of copyright-protected material under certain circumstances is allowed without permission from the copyright holder). We source our videos from YouTube and some are licensed under Creative Commons. We currently cannot release external links during the rebuttal due to the NeurIPS policy. Note that we will give credit to content creators. We further take two steps to tackle the potential privacy/licensing issues,

1. Verification of Consent: We will make every effort to verify that the individuals in the videos have consented to the use of their images and data for research and development purposes. We have reached out to the content creators and individuals depicted in the videos to obtain explicit consent and await their response.
2. Alternative Datasets: In cases where consent cannot be obtained, we will remove these data points from our dataset. We are seeking alternative data (CC licensed) where consent and copyright permissions are clear and unequivocal as backup. Under initial testing, replacing the original default-license videos with those CC-license videos does not have major influence on our comparison presented.


**Mitigating Misrepresentation and Stereotypes**

We understand the potential risks of misrepresentation and the reinforcement of stereotypes associated with AI avatars trained on this data. To mitigate these risks:

1. Rigorous Validation: We can implement rigorous validation processes to ensure that the AI models accurately represent sign language communication. This includes collaborating with experts in the field of sign language to review and validate the outputs of our models.
2. Ethical Guidelines: one option is developing a set of ethical guidelines to govern the use of AI avatars, ensuring that they do not perpetuate harmful stereotypes or misrepresent the related communities.


**Avoiding Reputational Harm**

We are acutely aware of the potential for reputational harm if AI avatars are used in unforeseen commercial or public settings. To prevent this:

1. Controlled Deployment:  
   1. Access Control: We can implement strict access controls to ensure that only authorized researchers have access to our AI models and datasets. This will involve the use of secure authentication mechanisms and role-based access controls.  
   2. Licensing and Usage Agreements: One option is ensuring that any use of the AI avatars is governed by proper licensing agreements that explicitly define the permissible use cases. These agreements will include clauses that prohibit the use of AI avatars in commercial or public settings without additional review and consent.
2. Built-in Safeguards: Another option is to integrate safeguards directly into the technology, such as digital watermarks, or limiting the generation of certain types of content.
3. Transparent Communication: We can maintain transparent communication with all stakeholders, including the individuals depicted in the videos, to ensure they are fully informed about how their data is being used and the contexts in which AI avatars may be deployed. Joint efforts are needed from authors, communities and regulations to prevent the technology from being misused. We are open to further discussing more strategies if the reviewer has any other suggestions.
Thank you for your valuable comments. We appreciate the opportunity to address these concerns, which are listed below.

**Q1: Suggestion on clarifying how technology changes SOTA with respect to potential issues of deception and harassment.**

A1: In this work, our proposed framework helps push the boundaries of what is possible with human avatar modeling, allowing for more expressiveness than previously achievable. However, with these advancements come certain risks, particularly concerning the potential misuse of the technology for deceptive practices or harassment. By advancing SOTA, our work makes these potential risks more pressing, as the barrier to creating such deceptive or harmful content is lowered. This highlights the need for careful consideration of the ethical implications and the development of strategies to mitigate these risks.

**Q2: Discuss/consider potential mitigation strategies such as restricted access to the technology.**

A2: Potential mitigation strategies could be taken from multiple aspects. We can provide restricted access to our code and request the users utilizing their institutional email addresses to apply. Another option is to integrate safeguards directly into our technology, such as digital watermarks, or limiting the generation of certain types of content. Joint efforts are needed from authors, communities and regulations to prevent the technology from being misused. We are open to further discussing more strategies if the reviewer has any other suggestions.
Dear Reviewer,

The discussion period with the authors is coming to a close in less than 24 hours, at 11:59 PM AOE Aug 13, 2024. There is still time to review the authors' response to your questions and engage in a discussion with them. It would great if you could acknowledge having read the authors' response and update your rating if it changes your mind.

Best,
AC
Thank you for the detailed responses. I also recommend including the fixed values of e and λt​ in the paper, along with a justification for how these values were determined. Given the overall quality of this submission, I will recommend the acceptance of this paper. Please include the additional experimental results in your revision.
Thanks for the clarification. Please include these in the revision.
Thanks for the detailed responses. Given the overall quality of this submission, I will recommend the acceptance of this paper.