Title: Expressive Gaussian Human Avatars from Monocular RGB Video
Venue: NeurIPS 2024
Accepted: yes
Keywords: expressiveness

human avatar

monocular RGB video
Abstract: Nuanced expressiveness, especially through detailed hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations.
In this work, we aim to learn expressive human avatars from a monocular RGB video; a setting that introduces new challenges in capturing and animating fine-grained details.
To this end, we introduce EVA, a drivable human model that can recover fine details based on 3D Gaussians and an expressive parametric human model, SMPL-X.
Focused on enhancing expressiveness, our work makes three key contributions.
First, we highlight the importance of aligning the SMPL-X model with the video frames for effective avatar learning.
Recognizing the limitations of current methods for estimating SMPL-X parameters from in-the-wild videos, we introduce a reconstruction module that significantly improves the image-model alignment.
Second, we propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds to accommodate the varied granularity across body parts.
Third, we develop a feedback mechanism that predicts per-pixel confidence to better guide the optimization of 3D Gaussians.
Extensive experiments on two benchmarks demonstrate the superiority of our approach both quantitatively and qualitatively, especially on the fine-grained hand and facial details. 
We make our code available at the project website: https://evahuman.github.io.

Expressive Gaussian Human Avatars from Monocular RGB Video

Abstract

Nuanced expressiveness, especially through detailed hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations. In this work, we aim to learn expressive human avatars from a monocular RGB video; a setting that introduces new challenges in capturing and animating finegrained details. To this end, we introduce EV A, a drivable human model that can recover fine details based on 3D Gaussians and an expressive parametric human model, SMPL-X. Focused on enhancing expressiveness, our work makes three key contributions. First, we highlight the importance of aligning the SMPL-X model with the video frames for effective avatar learning. Recognizing the limitations of current methods for estimating SMPL-X parameters from in-the-wild videos, we introduce a reconstruction module that significantly improves the image-model alignment. Second, we propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds to accommodate the varied granularity across body parts. Third, we develop a feedback mechanism that predicts per-pixel confidence to better guide the optimization of 3D Gaussians. Extensive experiments on two benchmarks demonstrate the superiority of our approach both quantitatively and qualitatively, especially on the fine-grained hand and facial details. We make our code available at the project website: https: //evahuman.github.io .

1 Introduction

High-quality digital avatar modeling has a wide range of applications, including AR/VR, movie production, sign language, and more. For digital human representation, capturing nuanced expressions is essential for enhancing fidelity and vitality. This is particularly evident in the detailed portrayal of hands and facial expressions, which add emotional depth and interactive expression capabilities to human avatars. In this work, we investigate expressiveness when building human avatars from monocular video. The task involves taking as input a monocular human video and learning an animated human avatar which enables multiple capabilities, such as novel view and pose synthesis.

Reconstructing expressive human avatars is challenging, particularly due to the subtle and complex movements of the hands and face. Compared to the body, hands and faces occupy smaller spatial areas and have distinct characteristics. For example, the hand has many degrees of freedom, intricate textures, and frequent self-occlusions. To achieve accurate avatar modeling, it is crucial to capture these fine textures from RGB video and ensure effective animation.

Current studies [49, 31, 15, 22, 12] mainly focus on learning human avatar on the body region and have made remarkable progress. Early works [49, 31, 15] mainly utilize NeRF as an implicit representation but usually have the drawback of low training/inference speed. Recently, more and more works build on top of 3D Gaussian Splatting [19] for its effectiveness and efficiency, which could further speed up rendering to over 100fps. GART [22] utilizes a mixture of moving 3D

EVA

SOTA Method

小

一

Gaussians to approximate human geometry and appearance and enhances fine details with learnable forward skinning and latent bones. GauHuman [12] proposes a new density control strategy, e.g. split and clone with KL divergence and a new merge operation. However, these methods do not consider the fine-grained details of hand and face, which cannot meet the requirement of expressiveness.

In this work, we introduce EVA, a drivable human model that can represent expressive details using 3D Gaussians and a human parametric model. Given a monocular RGB video, we extract the pose and mask information corresponding to each frame, which allows us to map each frame's human observation to a canonical space. Once the avatar is constructed, it can be animated using linear blend skinning given a new pose, followed by rendering to the 2D human image. To tackle new challenges introduced by expressiveness, we start by solving the misalignment issue between the SMPL-X model and the RGB frames via a reconstruction module. By employing a fitting-based optimization, this module can produce more reliable 3D SMPL-X reconstruction, providing a more robust foundation for digital avatar modeling.

Considering the granularity differences across different body parts, we propose a context-aware adaptive density control strategy for 3D Gaussian optimization. It leverages attributes specific to different parts and historical gradient information, to adaptively control Gaussian density. Furthermore, to improve the Gaussian optimization, we design a feedback mechanism, which adaptively predicts confidence scores based on the rendering result, thereby ensuring the supervision signal effectively transferring to the corresponding Gaussian. For evaluation on expressiveness, we build comparison baselines from related body avatar methods with a few modifications and collect a new benchmark called UPB containing in-the-wild upper body videos. The image quality is evaluated with multiple metrics, separately on the full body, as well as the hand and face regions. Our approach, as shown in Figure 1, largely outperforms previous state of the art, with more fidelity in fine-grained details.

Our contributions are summarized as follows:

We introduce EVA, an approach that can build expressive human avatars based on 3D Gaussians, given as input a monocular RGB video. Through extensive experiments on two datasets, we demonstrate the superiority of EVA, particularly on the hand and facial details.

To enhance expressiveness, we propose a context-aware adaptive density control strategy to accommodate the granularity differences across human parts, followed by a feedback mechanism to better guide the 3D Gaussian optimization.

To handle challenging in-the-wild videos, we propose a reconstruction module, which critically improves the SMPL-X alignment to the RGB frames compared to off-the-shelf methods. We demonstrate the importance of this improvement in recovering accurate avatars.

2 Related Work

2.1 Human Avatar Modeling from Monocular RGB Video

The problem of building a human avatar from a monocular RGB video is challenging due to the dynamic nature of the capture and the partial observations for each frame. Early works [31, 2, 37, 38, 3, 15, 49, 46, 39, 42, 6, 21, 41] mainly resort to the combination of implicit neural representations ( e.g., NeRF [27]) and parametric models to represent a human avatar with high fidelity and flexibility. To address the slow computation in implicit models, previous work has proposed various techniques to reduce training [10, 14] or inference time [33, 25, 5]. With the introduction of 3D Gaussian Splatting [19] which can achieve both fast and high-quality rendering, an increasing number of works [22, 24, 11, 12, 20, 23, 32, 13, 18] utilize it as their base representation, jointly with parametric human models like SMPL [26]. More specifically, GaussianAvatar [11] enhances 3D Guassians via two key components for final photorealistic quality, i.e., dynamic properties and joint optimization of motion and appearance. GauHuman [12] leverages human prior and KL divergence to propose a new density control strategy. In this work, we systematically explore expressiveness while building human avatars from a monocular RGB video and tackle the new challenges brought by expressiveness.

2.2 Expressive Human Representations

Expressiveness plays a vital role in non-verbal communication, which, besides the body, particularly involves the hand and face regions [44, 48, 51, 8, 9, 28, 16, 7, 35]. To represent expressive humans, various parametric models have been proposed, including SMPL-X [28], Adam [17], GHUM [43], and more. These models usually have predefined topology and can provide a compact mapping from the low-dimensional embedding to the 3D mesh. However, the predefined topology limits their capability of depicting fine-grained texture. Recent works build human avatars via neural 3D representations while leveraging the parametric model to provide human shape priors and animation signals. For example, X-Avatar [35] combines NeRF and SMPL-X to build the avatar. It further proposes part-aware sampling and initialization strategies to ensure efficient learning from highquality 3D scans or RGB-D data. GVA [24] integrates the SMPL-X model to improve the rendering quality. AvatarRex [50] proposes a compositional avatar representation to separately model body, hand and faces from multi-view RGB video data. Different from them, we aim to relax the input requirements, by building expressive human avatars from a real-world monocular RGB video.

3 Technical Approach

In this section, we first introduce the preliminary knowledge on the SMPL-X human model [28] and 3D Gaussian Splatting [19] and present the general framework design on articulated human modeling. Then, we elaborate on the key technical contributions of our approach.

3.1 Preliminaries

SMPL-X body model. SMPL-X [28] is a parametric human body model, which extends the SMPL body model [26] by modeling both hand articulation and facial expressions. SMPL-X can be defined as a mapping function M ( θ,β,ψ ) : R | θ | × R | β | × R | ψ | → R 3 N , where θ,β,ψ are the parameters for pose, shape, and facial expression, respectively. The function of SMPL-X is formulated as follows:

where B P ( · ) , B S ( · ) and B E ( · ) denote pose, shape, and expression blend functions, respectively, and W is a set of blend weights. The pose, expression and shape corrective blend shapes, i.e., B P ( θ ) ,

B E ( ψ ) and B S ( β ) , add corrective vertex displacements to the template human mesh ¯ T . After that, linear blend skinning W ( · ) is applied to rotate the vertices in the template mesh around the joints J ( β ) , smoothed by the blend weights W . This generates the final human mesh.

3D Gaussian Splatting. Methods based on NeRF [27] model the scene with an implicit representation and render novel views using volume rendering. In contract, 3D Gaussian Splatting [19] (3DGS) models a 3D scene with a set of discrete 3D Gaussians and performs rendering through a tile-based rasterization operation, which can reach real-time rendering speeds. Specifically, each Gaussian is defined with its central position p , 3D covariance matrix Σ as follows:

where x is an arbitrary position in the 3D scene. The covariance matrix Σ is decomposed into two learnable components to make the optimization easier, Σ = RSS T R T , where R and S denote the rotation matrix and scaling vector, respectively. During rendering, each 3D Gaussian G ( x ) is first transformed to a 2D Gaussian G ′ ( x ) on the image plane. Then, a tile-based rasterizer is designed to efficiently sort the 2D Gaussians and employ α -blending:

where r is the queried pixel position and N denotes the number of sorted 2D Gaussians associated with the queried pixel. c i and α i denote the color and opacity of the i -th Gaussian, which are modeled by Spherical harmonics.

Articulated 3D human modeling. Our formulation takes inspiration from works employing 3D Gaussians for modeling articulated objects [12, 22]. We optimize 3D Gaussians in a canonical space, which corresponds to a human in a rest pose [15]. The Gaussians are transformed from the canonical space to the frame space via linear blend skinning (LBS):

where p f , Σ f , p c , Σ c are the Gaussian mean and covariance in frame space and canonical space, respectively. G ( J f , θ f ) = ∑ K k =1 w k G k ( J f , θ f ) , b ( J f , θ f , β f ) = ∑ K k =1 w k b k ( J f , θ f , β f ) are the rotation and translation, respectively, with respect to the K joints, and G k ( J f , θ f ) , b k ( J f , θ f , β f ) are the rotation and translation, respectively, with respect to the k -th joint. w k is the LBS weight.

To perform Linear Blend Skinning, we need two important components, i.e., the LBS weights and the input pose parameters. Learning LBS weights w k from scratch would be inefficient and can lead to a local optimum in the early stage of the training. Therefore, for each Gaussian, we start with the LBS weight of the nearest SMPL-X vertex and use an MLP f Θ w to predict an LBS weight offset w ′ k using the positionally encoded [27] Gaussian centers γ ( p c ) . w k is therefore defined as:

where w SMPL-X k is the LBS weight of the nearest SMPL-X vertex. We set ϵ = 10 -8 .

Starting from the input pose θ SMPL-X , we further fine-tune it via an MLP-based network f Θ θ , which is jointly optimized during the 3D Gaussian optimization process. The actual poses θ used for optimization and rendering are therefore obtained as follows:

where ⊗ represents the vector pointwise product.

3.2 SMPL-X Alignment for Real-World Videos

A key requirement for learning accurate human avatars is to initialize the optimization process with a reliable SMPL-X estimate. However, this can be challenging, particularly in real-world cases,

Canonical Space

Gaussians

Context-aware

Adaptive Density Control

Learnable

LBS

Rendered Image

Confidence-aware Loss

GT Image

∇

p

R

u

p

S

SMPLX Template

Init

Prior

SMPLX Alignment

(Real-world Video)

Modeling

Monocular RGB Video

Animate Avatar

c

u

α

given the limitations of current off-the-shelf methods [29] for SMPL-X fitting. This motivates us to propose a robust fitting procedure that leverages multiple sources, including the initial estimation of camera parameter, SMPL-X parameters, 2D keypoints and 3D hand parameters from off-the-shelf tools [30, 45, 1]. Given these initial estimates, the SMPL-X fitting procedure minimizes the following objective:

The 2D keypoint term, L 2 D , encourages the projection of the 3D keypoints of the avatar to align with the detected 2D keypoints. It is formulated as follows:

where Π K ( · ) represents the camera projection under the given camera intrinsic parameter K . ψ ( · ) is the robust Geman-McClure error function [36] which helps prevent the disturbance from noisy 2D keypoint detections. R θ ( · ) denotes the function which rotates the joints J ( β ) given the pose θ . J 2 D is the detected 2D keypoints from [45]. γ and ω represent the predefined weighting parameters and the detection confidences respectively.

Our prior terms contain two components which focus on the coarse-grained body and fine-grained hand, respectively. For the body part, we utilize VPoser [28] to filter infeasible body poses. VPoser provides a compact mapping from a low-dimensional embedding η to the rotation matrices of the body pose θ b . To better optimize the low-dimensional embedding η , the estimated 3D body joints J 3 D b [1] is treated as guidance, together with the added regularization term. The body prior loss term, L bp , is formulated as follows:

Similary, for the hands, we utilize the 3D hand joints J 3 h D estimated from [30] to provide a better hand spatial relationship. This is formulated as follows:

where the index h represents the joints corresponding to the hand. ψ z ( · ) indicates that the robust error function only considers the coordinates along the z-axis coordinates for the error calculation.

3.3 Context-aware Adaptive Density Control

The core of the Gaussian Splatting optimization is adaptive density control, which generally contains two operations, i.e., densification (split and clone) and pruning. The original strategy [19] selects a

fixed constant as the densification criteria. However, utilizing the fixed threshold does not leverage context information, leading to sub-optimal 3D Gaussian representations.

To this end, we propose a context-aware adaptive density control that leverages part attribute and history gradient information. Each Gaussian inherently possesses attributes associated with different body parts. These body parts vary in spatial size and characteristics. For instance, the hand is much smaller in size, compared to the body, exhibiting fine-grained textures and many degrees of freedom. These characteristics also lead to inherent differences in the Gaussian positional gradient changes across different body parts during the Gaussian optimization process. Furthermore, a continuous increase in the Gaussian positional gradient indicates that Gaussians need to be densified.

Specifically, we first initialize the 3D Gaussians with the vertices of the SMPL-X model. Since SMPL-X has a predefined topology, this initialization provides each Gaussian with the attribute information U on which part it belongs, e.g. body, hands or face. After that, considering attributes and gradient history information, the densification threshold for the i -th Gaussian in a certain attribute U is formulated as follows:

where e is a constant, R represents the densification interval, and ∇ k denotes the position gradient of i -th Gaussian at the timestamp k . Note that e and λ t have different values for different attributes. For the pruning strategy, we remove the points that are far away from the SMPL-X template vertices.

3.4 Objective Functions

The whole framework is optimized under the objective functions as follows:

where λ m , λ s , and λ l are loss weighting terms. The mask loss L m calculates the consistency between accumulated volume density and the estimated mask. The SSIM [40] loss L SSIM is adopted to improve the structural similarity between rendered image and input image. The LPIPS [47] loss L LPIPS focuses on the perceptual quality of the rendered image.

Confidence-aware loss L c . It is inevitable that any training video will introduce some form of noise ( e.g. , misalignment, motion blur), which will also affect the avatar optimization procedure. To address some of the effects of the noise, we introduce a feedback module to decide which pixels should be taken into consideration with higher or lower weight during training. This module takes as input the rendered image I r and rendered depth D r , and predicts a score for each pixel which represents the confidence value. More specifically:

where µ is a constant. Then the confidence serves as an adaptive weighting factor on the per-pixel consistency. Eventually, we formulate our confidence-aware loss as follows:

4 Experiments

In this section, we first introduce our experimental setup, including datasets, implementation details and evaluation metrics. Then, we make comparisons with baseline methods both quantitatively and qualitatively. Finally, we perform ablation studies on the most important components of our approach.

4.1 Experimental Setup

Datasets. The experiments are conducted on two datasets, XHumans [35] and our collected UPB dataset. XHumans is a dataset captured in a controlled environment. It provides images with resolution of 1200 × 800, along with well-aligned SMPL-X meshes. There are 6 identities (3 male and 3 female) for evaluation. UPB consists of sign language videos from the web, which usually contain complicated hand gestures. The resolution of the videos is 1920 × 1080 and they do not contain any ground truth SMPL-X annotations. UPB includes 4 identities (2 male and 2 female).

Implementation details. Our framework is implemented on PyTorch and all experiments are performed on NVIDIA A5000. The hyperparameter λ m , λ s and λ l are set to 0.1, 0.01 and 0.04, respectively. We use SGHM [4] to extract the human mask. The 3D Gaussian optimization lasts for 2,000 iterations with the densification performed between iteration 400 and 1,000. For other parameters, we follow the original settings of [19]. Our method can render a 1080p image with an inference speed of 361.02 fps on one RTX A5000. Since our method is reconstruction-based, our main requirement is that the areas of interest should be captured in the video. Note that we use the same input RGB frames for all methods to ensure a fair comparison. More details are presented in the Appendices.

Evaluation metrics. We evaluate the avatar quality through the rendered views following previous works [22, 12]. We adopt the widely-used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [40], and Learned Perceptual Similarity (LPIPS) [47] to evaluate the rendered images. For a more fine-grained evaluation, we report the above metrics separately for the full body, for the hands region and for the face region.

4.2 Comparison with baselines

The baselines we use for comparison are adopted from related methods with a few modifications. 3DGS + SMPL-X is modified from 3DGS [19]. We add our articulated human modeling mechanism to make it fit the task requirement. GART and GauHuman are originally designed to model body-level avatars. They are animated via the SMPL parametric model, which lacks the capability of modeling hand articulation and facial expressions. Therefore, we replace the driven signal with the SMPL-X model and denote them as GART + SMPL-X and GauHuman + SMPL-X, respectively. Splatting selects the triangle mesh as the driving signal and we utilize SMPL-X mesh in Splatting + SMPL-X. Among these baselines, GauHuman is closer to our approach. Our differences are mainly reflected in the adaptive density control strategy, objective functions and the SMPL-X alignment for the real-world videos, which are also consistent with our main technical contributions.

As shown in Table 1, we conduct experiments on the XHumans and UPB datasets to evaluate the effectiveness of our method. XHumans is captured in a controlled setting with accurate SMPL-X ground truth, while UPB focuses on real-world video without pose information. Since it is hard to get accurate SMPL-X annotation for real videos, UPB is more challenging and more representative of learning a human avatar in the wild. We observe that our method achieves state-of-the-art performance on these two datasets. We note that we do not apply our designed alignment module in XHumans, since it already contains SMPL-X ground truth. Our method achieves 19.7%, 17.3% and 22.5% relative LPIPS gain on the full, hand and face regions, respectively. For the real-world UPB dataset, our performance gain is much larger, achieving over 25% relative LPIPS gain on the hand region. This indicates that EV A, unlike previous work, handles well the challenges of real-world videos. The qualitative comparison in Figure 3 also validates the effectiveness of our method.

GT

3DGS+

GART+

GauHuman+

EVA

4.3 Ablation Studies

We perform ablation experiments to highlight the important components of EVA, i.e. , context-aware adaptive density control, confidence-aware loss and in-the-wild SMPL-X alignment module.

Effectiveness of context-aware adaptive density control (CADC) and confidence-aware loss (CL). These two components are designed for better optimization of Gaussian Avatar Modeling . As shown in Table 2, 'w/o CADC' means that we replace it with the original density control [19], while 'w/o CL' denotes removing the confidence weighting term in the RGB loss calculation. We observe that these two components both improve performance. More specifically, CADC improves performance for the metrics of all regions. Meanwhile, we observed that CL brings another benefit of a more compact representation (e.g., reducing the number of Gaussians from 21,038 to 19,993).

Effectiveness of in-the-wild SMPL-X alignment. We use our SMPL-X alignment module for real-world videos without accurate SMPL-X ground truth, so we conduct the corresponding ablation study on UPB dataset. As shown in Table 2, 'w/o Align' means that we directly utilize the SMPL-X mesh extracted from current SOTA estimation method [1]. Notably, our proposed alignment module brings notable performance gains on all metrics of all regions. It also demonstrates the importance of the SMPL-X pose quality for final human avatar modeling. Without accurate SMPL-X alignment, the following Gaussian model will need to deal with inconsistent textures for a specific region across frames, leading to lower avatar quality. Furthermore, we visualize the SMPL-X alignments in Figure 4 and present qualitative comparison with state-of-the-art estimation methods in Figure 5, including fitting-based SMPLify-X and regression-based SMPLer-X. Visible improvements are clearly observed specifically for the hands.

GT

w/o Align

EVA

GT

w/o Align

EVA

4.4 More Discussion

In summary, we apply EVA on a variety of monocular RGB videos, including both controlled captures and videos collected from the Internet. The visual data contain large body motions (e.g., playing basketball, weight lifting, dancing), along with fine-grained motions (e.g. finger counting, using tools, sign language). EVA can handles well self-occlusions, which usually exist in every frame, especially for the hand areas. This can be attributed to our proposed SMPL-X fitting method which provides reliable correspondences from the pixels to the canonical space. Besides, we expect that EV A could also handle low lighting conditions (as long as the lighting is not changing significantly), since our SMPL-X fitting method should be robust to low lighting conditions. Given accurate SMPL-X estimates, we expect the avatar learning stage will perform well.

Limitations. The focus of this work is to capture expressive human avatars, by designing a pipeline that can take raw web videos as the input. Besides capturing the expressive details, other factors are challenging, yet worth investigating to further improve the quality of the avatar, e.g. non-rigid elements like clothes and hair. It is also worth extending our avatar reconstruction to more challenging input sources, e.g. occlusions or few-shot scenarios.

Failure Cases. Weshow some representative examples of our failure cases in Figure 6. For the fine-grained expressive areas, the failure cases happen when the video includes hand interactions. The failure is mainly caused by incorrect SMPL-X reconstruction (e.g., the estimated SMPL-X model that drives the reconstruction has implausible interpenetration between hands). Another failure case is the existence of floaters in the reconstruction which is a common issue of 3DGS modeling.

(a) False driving signal

SMPLer-X

SMPLify-X

Ours

Broader Impact. In this work, our proposed framework helps push the boundaries of what is possible with human avatar modeling, allowing for more expressive results than previously achievable. However, with these advancements come certain risks, particularly concerning the potential misuse of the technology for deceptive practices or harassment. By advancing SOTA, our work makes these potential risks more pressing, as the barrier to creating such deceptive or harmful content is lowered. This highlights the need for careful consideration of the ethical implications and the development of strategies to mitigate these risks.

5 Conclusion

In this work, we present EV A, a drivable expressive human avatar learned from a real-world monocular RGB video. EVA is built on 3D Gaussians, in coordination with the human prior introduced by SMPL-X. To deal with the challenges brought by expressiveness, we first utilize a reconstruction module to solve the misalignment between SMPL-X model and RGB frames. During the optimization of the 3D Gaussians, we propose context-aware adaptive density control, which leverages attribute and historical gradient information to accommodate the varied granularity across body parts. A feedback mechanism is jointly designed to further guide learning. Extensive experiments on two benchmarks demonstrate that our method outperforms baselines both quantitatively and qualitatively, especially on the fine-grained hand and facial details.

Acknowledgments. This project was supported by LUCI program under the Basic Research Office and partially supported by ARL grants W911NF20-2-0158 and W911NF-21-2-0104 under the cooperative A2I2 program. It was also in part supported by NSF AI Institute for Foundations of Machine Learning (IFML). GP has received a research gift from Google.

References

Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation. NeurIPS , pages 1-15, 2024.

Xu Chen, Tianjian Jiang, Jie Song, Max Rietmann, Andreas Geiger, Michael J. Black, and Otmar Hilliges. Fast-snarf: A fast deformer for articulated neural fields. IEEE TPAMI , 2023.

Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In ICCV , 2021.

Xiangguang Chen, Ye Zhu, Yu Li, Bingtao Fu, Lei Sun, Ying Shan, and Shan Liu. Robust human matting via semantic guidance. In ACCV , pages 2984-2999, 2022.

Zhaoxi Chen, Fangzhou Hong, Haiyi Mei, Guangcong Wang, Lei Yang, and Ziwei Liu. Primdiffusion: Volumetric primitives diffusion for 3d human generation. In NeurIPS , pages 1-14, 2023.

Junting Dong, Qi Fang, Yudong Guo, Sida Peng, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Totalselfscan: Learning full-body avatars from self-portrait videos of faces, hands, and bodies. In NeurIPS , pages 13654-13667, 2022.

Maria-Paola Forte, Peter Kulits, Chun-Hao P Huang, Vasileios Choutas, Dimitrios Tzionas, Katherine J Kuchenbecker, and Michael J Black. Reconstructing signing avatars from video using linguistic priors. In CVPR , pages 12791-12801, 2023.

Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, and Ziwei Liu. Stylegan-human: A data-centric odyssey of human generation. In ECCV , pages 1-19, 2022.

Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Wayne Wu, and Ziwei Liu. Unitedhuman: Harnessing multi-source data for high-resolution human generation. In ICCV , pages 7301-7311, 2023.

Chen Geng, Sida Peng, Zhen Xu, Hujun Bao, and Xiaowei Zhou. Learning neural volumetric representations of dynamic humans in minutes. In CVPR , pages 8759-8770, 2023.

Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. arXiv , pages 1-13, 2023.

Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting for real-time 3d human rendering. In CVPR , pages 1-16, 2024.

Rohit Jena, Ganesh Subramanian Iyer, Siddharth Choudhary, Brandon Smith, Pratik Chaudhari, and James Gee. Splatarmor: Articulated gaussian splatting for animatable humans from monocular rgb videos. arXiv , pages 1-11, 2023.

Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Instantavatar: Learning avatars from monocular video in 60 seconds. In CVPR , pages 16922-16932, 2023.

Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In ECCV , pages 402-418, 2022.

Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. In CVPR , pages 1-12, 2024.

Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In CVPR , pages 8320-8329, 2018.

HyunJun Jung, Nikolas Brasch, Jifei Song, Eduardo Perez-Pellitero, Yiren Zhou, Zhihao Li, Nassir Navab, and Benjamin Busam. Deformable 3d gaussian splatting for animatable human avatars. arXiv , pages 1-15, 2023.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG , 42(4):1-14, 2023.

Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. Hugs: Human gaussian splats. In CVPR , pages 1-16, 2023.

Youngjoong Kwon, Lingjie Liu, Henry Fuchs, Marc Habermann, and Christian Theobalt. Deliffas: Deformable light fields for fast avatar synthesis. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, NeurIPS , pages 40944-40962, 2023.

Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilidis. Gart: Gaussian articulated template models. arXiv , pages 1-13, 2023.

Mingwei Li, Jiachen Tao, Zongxin Yang, and Yi Yang. Human101: Training 100+ fps human gaussians in 100s from 1 view. arXiv , pages 1-20, 2023.

Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Chen Zhao, Haocheng Feng, Errui Ding, and Jingdong Wang. Gva: Reconstructing vivid 3d gaussian avatars from monocular videos. arxiv , pages 1-13, 2024.

Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM TOG , 40(4):1-13, 2021.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM TOG , 34(6):1-16, 2015.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , pages 405-421, 2020.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR , pages 10975-10985, 2019.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR , pages 10975-10985, 2019.

Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In CVPR , pages 1-11, 2023.

Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR , pages 9054-9063, 2021.

Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In CVPR , pages 1-19, 2024.

Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito, Chenglei Wu, Tomas Simon, Shih-En Wei, Kaiwen Guo, Zhe Cao, Fabian Prada, Jason Saragih, et al. Drivable volumetric avatars using texel-aligned features. In SIGGRAPH , pages 1-9, 2022.

Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. Splattingavatar: Realistic real-time human avatars with mesh-embedded gaussian splatting. In CVPR , pages 1606-1616, 2024.

Kaiyue Shen, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Julien Valentin, Jie Song, and Otmar Hilliges. X-avatar: Expressive human avatars. In CVPR , pages 16911-16921, 2023.

Geman Stuart and E. McClure Donald. Statistical methods for tomographic image reconstruction. Bulletin of the International Statistical Institute , 52:5-21, 1987.

Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In NeurIPS , pages 12278-12291, 2021.

David Svitov, Pietro Morerio, Lourdes Agapito, and Alessio Del Bue. Haha: Highly articulated gaussian human avatars with textured mesh prior. arXiv , pages 1-23, 2024.

Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In ECCV , pages 1-19, 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP , 13:600-612, 2004.

Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizerman. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In CVPR , pages 16210-16220, 2022.

Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. NeurIPS , 34:14955-14966, 2021.

Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR , pages 6184-6193, 2020.

Zhongcong Xu, Jianfeng Zhang, Junhao Liew, Jiashi Feng, and Mike Zheng Shou. Xagen: 3d expressive human avatars generation. In NeurIPS , pages 34852-34865, 2023.

Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In ICCV , pages 4210-4220, 2023.

Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, and Kwan-Yee Lin. Monohuman: Animatable human neural field from monocular video. In CVPR , pages 16943-16953, 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , pages 586-595, 2018.

Zechuan Zhang, Li Sun, Zongxin Yang, Ling Chen, and Yi Yang. Global-correlated 3d-decoupling transformer for clothed avatar reconstruction. NeurIPS , pages 7818-7830, 2023.

Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In CVPR , pages 7743-7753, 2022.

Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu. Avatarrex: Real-time expressive full-body avatars. ACM TOG , 42:1-19, 2023.

Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv , pages 1-11, 2023.

Appendices

This technical appendices provides more details which are not included in the main paper due to space limitations. For more visualization results, please refer to the project webpage.

Training/Testing split. For each identity in XHumans dataset, one video is selected as the training split, where the other videos are marked as testing. During training, we utilize all of the frames (150 frames). We sample 20 frames for each testing video with the sampling rate of 5. For UPB dataset, we uniformly sample the frames with the interval as 1 to split the training and testing frames. The number of training and testing frames are both 140.

Additional implementation details. µ is set to 1. λ t is set as -9.0, -4.5 and -6.3 for body, hand, and face parts, respectively. We set e for body, hand, and face parts as 2e-4, 1e-4 and 1.4e-4, respectively. The feedback module E ( · ) consists of two 2D convolutional networks. For SMPL-X alignment, we utilize the L-BFGS optimizer with the Wolfe line search. The optimization contains three stages with different loss weighting factors. The first stage is designed to initialize the human body embedding to the initial estimation from the method [1]. Then the second stage mainly aims to get better spatial hand relationship leveraging the prediction from the method [30]. The third stage performs fine-tuning with more emphasize on the fine-grained hand and face regions.

Comparison baselines. The baselines we use for comparison are adopted from related methods with a few modifications. 3DGS + SMPL-X is modified from 3DGS [19]. Since the original 3DGS does not have the capability of animation, we add the same articulated human modeling in Section 3.1 as ours. For fair comparison, it also utilizes the same optimization schedule as ours, which lasts 2,000 iterations with the densification performed between 400 and 1,000 iterations. GART and GauHuman are originally designed to model body-level avatars. They are animated via the SMPL parametric model, which lacks the capability of modeling hand articulation and facial expressions. Therefore, we replace the driven signal with the SMPL-X model and denote them as GART + SMPL-X and GauHuman + SMPL-X, respectively. Since Splatting is embedded on a triangle mesh, we utilize the SMPL-X human mesh as its driving signal and denote it as Splatting + SMPL-X. We do not modify the optimization schedules of these three methods.

Future works. We outline the potential future works as follows,

Modeling capability on non-rigid elements, such as loose cloth (dress). Modeling cloth on the avatar has been a challenging topic, even separately studied in several works. A potential solution could be parameterizing the cloth deformation or adding more prior on cloth type, so as to provide more driving signals.

Generalizable human avatar from monocular RGB video. Current methods in this topic mostly need per-subject optimization, which needs to be re-trained to any new given subjects. It is worth exploring if we could get an avatar from a monocular RGB video of any given subject with a single feed-forward pass.

Robustness. It is worth exploring if we could build the human avatar well with more limited source inputs, e.g. a few images with even mutual occlusion, a single image, etc.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the contributions and scope in Abstract and Introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Answer: [Yes]

Justification: We describe the limitations in the Section 4.4.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have presented the information in Section 4.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release our code after our paper gets accepted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are specified in Section 4.1.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We use the fixed seed for easy reproducing.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the type of resources used in the experiments in Section 4.1.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discusses the broader impact in Section 4.4.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The released model does not have a high risk for misuse.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite corresponding papers for the asserts we use in Section 4.1.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on improving the expressiveness of digital human avatars, particularly through detailed hand and facial expressions, learned from monocular RGB video. The main contributions are:\n- SMPL-X Alignment improves the alignment of the SMPL-X model with RGB frames and aids in accurately recovering avatars. \n- Context-Aware Adaptive Density Control Strategy adjusts gradient thresholds to handle the varied granularity across different body parts, enhancing the expressiveness of the avatars.\n- Feedback Mechanism that predicts per-pixel confidence to better guide the learning and optimization of 3D Gaussians.\n\nOverall, the paper presents a comprehensive framework that enhances the realism and expressiveness of digital human representations, validated by substantial quantitative and qualitative experiments.\n\nSTRENGTHS:\nThe paper is written in a clear and understandable manner. Comprehensive ablation studies and clear visualizations help to demonstrate the effectiveness of different components of the method. The paper also compares the new method with previous methods and shows that it achieves better performance. It is commendable that the authors added SMPL-X to existing methods for fair and thorough comparisons.\n\nWEAKNESSES:\n- One of the contributions of using and optimizing SMPL-X for expressive avatar is not novel. This has been done in previous works [1]. \n- The paper lacks a comparison to recent methods such as Splatting Avatar [2]. Including this comparison in Table 1 can help to distinguish the improvement made by the proposed approach. \n- There is an inconsistent ablation study across the datasets. Table 2 should illustrate how the alignment affects the baseline method by comparing scenarios with and without CADC, CL, and alignment. Similarly, Table 3 should also show the method with CADC and CL included.\n- PSNR, SSIM, and LPIPS should be calculated from four novel views to ensure the robustness of the method across different perspectives. Additionally, animation and novel pose metrics can be included to display differences between methods.\n- From Table 2, it seems that CL does not have a significant impact. This should be addressed or clarified to understand its role in the framework. It would be good to provide more qualitative visualization to demonstrate its effectiveness. \n\n\n[1] (Liu et al., 2024) GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos. \n\n[2] (Shao et al., 2024) SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.\n\nQUESTIONS:\n- **Project Page for Visualizations:** Is there a project page available that shows visualizations of the rendered method, such as a 360-degree rotating video of the generated person and animation to better illustrate the quality? Specifically, could visualizations zoom into detailed areas such as the hands and face?\n- Is e and λt in Equation 14 learned or fixed? If learned, it would be advantageous to list the values to demonstrate that the learned parameters effectively handle different resolution details for different body parts.\n- Discussing failure cases and outlining potential future work would provide a more comprehensive view of the method's limitations and areas for improvement.\n- Information on the optimization time compared to previous methods and the resource requirements for running the proposed method would be valuable for assessing its efficiency and scalability.\n\nLIMITATIONS:\nWhile the use of CADC is interesting, the other two contributions are of limited novelty. SMPL-X usage and alignment have been incorporated by previous works. The improvement of CL is not significant. The addition of project page for visualizations would be helpful to demonstrate the improvements made by the methods. In addition, more quantitative results should be added to Tables 1, 2 and 3.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a 3DGS-based human avatar generation method from a monocular RGB video. Specifically, the authors first optimize the SMPL-X model to better align with the RGB frames. Then, they propose an adaptive adjustment method for different body parts and use per-pixel confidence to guide 3DGS. It is reported that the proposed method achieves state-of-the-art performance. Visual results show fine-grained hand and facial details.\n\nSTRENGTHS:\n1. Generating the human upper body videos is a critical task. It is more expressive and more challenging than facial generation. This method overall demonstrates fine-grained generation details.\n2. The existing SMPL-X model has alignment issues when estimating the human body. The optimized SMPL-X model in this method significantly improves the alignment between SMPL-X and the RGB frames.\n3. The paper is overall easy to follow, and the experimental results demonstrate the method's superiority in numerical metrics.\n\nWEAKNESSES:\n1. In the paper, the confidence scores for CL are all learned from the rendered results. How can the effectiveness of this confidence be ensured? In Table 2, the face area without CL also shows better results.\n2. [11] also demonstrates photorealistic 3D human body generation, and the authors need to include more discussions with it.\n3. The paper has many writing issues, such as a too-brief related work section, inconsistent formula definitions (e.g., $\\Pi_K$ in Eq. 11), and some missing punctuation in formulas.\n\nQUESTIONS:\n1. What are the specific requirements for videos in terms of the method, such as the length of the video and the range of camera angles?\n2. How does the method perform when the magnitude of the action being driven is out-of-distribution?\n3. What is the inference speed of the method?\n\nLIMITATIONS:\nThis paper has involved discussions about limitations and broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents EVA, a model that can recover expressive human avatars from monocular RGB videos. The primary focus is enhancing fine-grained hand and facial expressions using 3D Gaussian Splatting and the SMPL-X model. EVA introduces three key contributions: a plug-and-play module for better SMPL-X alignment, a context-aware adaptive density control strategy, and a feedback mechanism for optimizing 3D Gaussian learning. Extensive experiments on two benchmarks demonstrate EVA's superiority in capturing detailed expressiveness compared to previous methods.\n\nSTRENGTHS:\nThe paper introduces three significant innovations: a plug-and-play alignment module, a context-aware adaptive density control strategy, and a feedback mechanism for 3D Gaussian optimization.\n2. Extensive quantitative and qualitative evaluations on two benchmarks show EVA's effectiveness in capturing fine-grained details, particularly in hand and facial expressions.\n3. The proposed method addresses the challenging task of creating expressive avatars from monocular RGB video, which has practical applications in VR/AR, movie production, and video games.\n4. The paper comprehensively explains the technical approach, including SMPL-X alignment, Gaussian optimization, and adaptive density control.\n\nWEAKNESSES:\n1. The proposed method's complexity might pose implementation challenges for those not well-versed in the field. Simplifying some aspects or providing more intuitive explanations could help. \n2. While the method shows superior performance on the provided datasets, the generalizability to other types of monocular videos or environments is not thoroughly discussed.\n3. Although the paper compares EVA with previous SOTA methods, it would be beneficial to include more diverse baselines or variants to demonstrate the robustness of the proposed approach.\n\nQUESTIONS:\n1. How does EVA perform with videos that have significant occlusions or low lighting conditions?\n2. Are there any specific preprocessing steps required for the input monocular RGB videos?\n3. Can the proposed method be extended to capture other expressive elements, such as cloth or hair movements?\n\nLIMITATIONS:\n1. The performance heavily relies on the quality of the datasets used. Real-world applications might present challenges not covered by the current benchmarks.\n2. The method requires significant computational power, which might limit its applicability in real-time scenarios or on devices with limited resources.\n3. While the method focuses on hand and facial details, other expressive elements like body language, clothing, and hair dynamics are not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a solution to generate expressive human avatars from monocular RGB videos. The main focus is to improve the expressiveness. To this end, a few ideas such as combining 3d Gaussian splatting and SMPL-X (SMPL+parametric hands), minimizing 2d reprojection error,  finegrained density control, etc. \n\nThere are a few similar papers in citation. To better understand the novelty of this paper, here's a high level comparison:\n+ SMPLer-X [1]: Representation is SMPL-X only. Input is monocular RGB videos. Losses: SMPL-X keypoint losses.\n+ GART[23]:  Representation is 3d Gaussians+SMPL. Input is monocular RGB videos.  Losses: photometric, perceptual.\n+ 3DGS-Avatar [33]: Representation is 3d Gaussians+SMPL; Input is monocular RGB videos; Losses: photometric, perceptual, mask, etc; \n+ This paper: Representation is 3d Gaussians+LBS+SMPL-X. Input is monocular RGB videos. Losses: SMPL-X keypoint losses, perceptual.\n\nSTRENGTHS:\nFrom the high level comparison, the paper does find a unique and timely combination of 3d Gaussians and SMPL-X, which is well-motivated by improving the expressiveness. Some of the ideas such as adaptive density control are therefore new as it leverages the fine-grained details of SMPL-X.\n\nWEAKNESSES:\nThe concept of model and image alignment by minimizing reprojection losses is not new. For example, the keypoints losses (Sec. 3.2) are very common in literature [1, 30]. \nSimilarly mask loss and perceptual loss can be found in [33].\n\nThe adaptive density control part seems novel, as it's initialized from body parts and leveraging the fine-grained topology of SMPL-X.\n\nThe confidence-aware loss seems new but only provides marginal improvement as in Table 2.\n\nLack of hyperparameter ablation. Not even in supplementary. \n\nMinor issue: `video` is countable. Please explicitly use `videos` or `a video` in the paper. Actually it's not clear how many videos of one subject are used to train one avatar.\n\nQUESTIONS:\nWhat's 'E' in Eq (16)? It looks like the so-called feedback module. But what's the design? Is it a learning-based predictor?\n\nPlease also address the novelty questions in the weakness part.\n\nLIMITATIONS:\nLimitations are discussed. It'd be good to discuss the failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Given an RGB human video, the paper focuses on photorealistic reconstruction of the human in 3D using SMPL-X and Gaussian rendering.\n- The main idea is to align the expressive SMPL-X with the video evidence followed by Gaussian Avatar modeling with better gaussian density control and loss. \n- Technical contributions: Context-aware adaptive density control, confidence-aware loss, SMPL-X alignment.\n- Baselines: 3DGS, GART, GauHuman.\n- Evaluation are done on two datasets, XHuman and UPB. Metrics: PSNR, SSIM, LPIPS.\n- The results show that the proposed method EVA consistently outperforms the baselines.\n\nSTRENGTHS:\n- The paper is well written, organized and easy to follow.\n- The focus problem is of great importance for the field of human digitization.\n- The ablative studies are informative, giving us insights into the importance of the technical components.\n\nWEAKNESSES:\n- Weak technical contributions: In comparison to GauHuman, CVPR 2024, the proposed method EVA makes three changes. 1) Context-aware adaptive density (CADC), 2) Confidence-aware loss (CL), 3) SMPL-X alignment. CADC is a nice idea for human modeling, and is inspired by the original 3DGS paper. However, it is a slight change to the originally proposed heuristic. CL is frequently used in the field of pose-estimation and 3D modeling (eg. DUST3R, CVPR 2024). SMPL-X alignment is very simply mesh fitting to 2D keypoints and is well explored in SMPLify-X, CVPR 2019 and its related works. Overall, the main technical ideas proposed lack technical novelty.\n\n- XHuman over ZJU_MoCap for evaluations: Is there a reason why metrics were reported on XHuman and not ZJU-MoCap dataset? The missing metrics on the popular ZJU-MoCap benchmark makes it hard to fairly compare EVA with existing. I am guessing this because ZJU-MoCap does not have SMPL-X annotations. In this context, can we use the same setup as UPB and still evaluate using predicted SMPL-X parameters?\n\nQUESTIONS:\nGiven the lack of technical novelty, it is important to establish the empirical usefulness of the proposed method. \nIn this context, I am asking for a fair evaluation with prior methods (mostly using SMPL) on popular benchmarks.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}