Animatable 3D Gaussian: Fast and High-Quality Reconstruction of Multiple Human Avatars

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 05:00:34

Abstract: Neural radiance fields are capable of reconstructing high-quality drivable human avatars but are expensive to train and render and not suitable for multi-human scenes with complex shadows. To reduce consumption, we propose Animatable 3D Gaussian, which learns human avatars from input images and poses. We extend 3D Gaussians to dynamic human scenes by modeling a set of skinned 3D Gaussians and a corresponding skeleton in canonical space and deforming 3D Gaussians to posed space according to the input poses. We introduce a multi-head hash encoder for pose-dependent shape and appearance and a time-dependent ambient occlusion module to achieve high-quality reconstructions in scenes containing complex motions and dynamic shadows. On both novel view synthesis and novel pose synthesis tasks, our method achieves higher reconstruction quality than InstantAvatar with less training time (1/60), less GPU memory (1/4), and faster rendering speed ($7\times$). Our method can be easily extended to multi-human scenes and achieve comparable novel view synthesis results on a scene with ten people in only 25 seconds of training. We will release the code and dataset.

Main paper: 
 introduction: 
 Real-time rendering and fast reconstruction of a high-quality digital human has a variety of applications in various fields, such as virtual reality, gaming, sports broadcasting, and telepresence. Existing methods often take considerable time for training, and none of them can achieve high-quality reconstruction and real-time rendering in a very short training time.Recent methods [5, 26, 37] implicitly reconstruct high-quality human avatars using neural radiance fields [35] . However, implicit neural radiance fields inevitably lead to artifacts in the synthesized novel views as they require modeling the entire space including empty space. Some of these methods [5, 37] require high memory and time consumption due to multilayer perceptron and complex point sampling for each ray. When it comes to datasets with dynamic illumination and shadow, the reconstruction quality of these methods [5, 26, 37] is significantly degraded. This decline is attributed to the inherent incapacity of these methods to accommodate the dynamic alterations in illumination and shadow.In this paper, we aim to acquire high-fidelity human avatars from a monocular or sparse-view video sequence in seconds and to render high-quality novel views and poses at interactive rates (170 FPS for 540 × 540 resolution). To this end, we introduce a novel neural representation using 3D Gaussians (3D-GS [28] ) for dynamic humans, named Animatable 3D Gaussian, overcoming the problem of artifacts in implicit neural radiance fields. In order to deform 3D Gaussians to the posed space, we model a set of skinned 3D Gaussians and a corresponding skeleton in canonical space. We use the multi-head hash encoder to encode the shape and appearance of 3D Gaussians to accelerate convergence speed, avoid overfitting, and acquire pose-dependent deformation. For low memory and time consumption of rendering, we use the 3D Gaussian rasterizer [28] to rasterize the deformed 3D Gaussians instead of volume rendering. To handle the dynamic illumination and shadow, we suggest modeling time-dependent ambient occlusion for each timestamp.Since the public dataset [1, 37] contains few pose and shadow changes, we create a new dataset named GalaBasketball in order to show the performance of our method under complex motion and dynamic shadows. We evaluate our method on both public and our created dataset and compare it with state-of-the-art methods [5, 26] . Our method is able to reconstruct better-quality human avatars in a shorter time. Moreover, our method can be extended to multihuman scenes and performs well.In summary, the major contributions of our work are:• We propose Animatable 3D Gaussian, a novel neural representation using 3D Gaussians for dynamic humans, which enables 3D Gaussians to reconstruct human avatars from scratch without any preprocessing such as SFM [41] . • We present a novel pipeline for human reconstruction that can acquire higher-fidelity human avatars with lower memory and time consumption (few seconds) than state-ofthe-art methods [5, 26] . • We propose a time-dependent ambient occlusion module to reconstruct the dynamic shadows, which allows our method to obtain high-fidelity human avatars from multi-human scenes with complex motions and dynamic shadows. 
related work: 
 3D Human Reconstruction. Reconstructing 3D human has been a popular research topic in recent years. Traditional methods achieve high-fidelity reconstruction by means of depth sensors [9, 14, 19, 42] and dense camera arrays [11, 12] , but expensive hardware requirements limit the application of these methods. Recent methods [1, 2, 18, 20, 21, 47, 48] utilized parametric mesh templates as a prior, such as SMPL [33] . By optimizing mesh templates, these methods are able to reconstruct 3D human bodies of different shapes. However, they have limitations in reconstructing details such as hair and fabric.The emergence of neural radiance fields [35] has made neural representations popular in the field of human reconstruction. Many works [3, 5-8, 10, 13, 15, 22-27, 29, 32, 34, 37, 39, 40] have used neural representations to model 3D human shapes and appearances in a canonical space and then used deformation fields to deform the model into a posed space for rendering. These approaches achieve high-quality reconstruction results but require high memory and time consumption, as they typically require complex implicit representations, deformation algorithms, and volume rendering. One of these works InstantAvatar [26] is fast but suffers from artifacts in synthetic images. Our approach solves the problems of previous methods by combining parametric templates and implicit expressions to achieve fast and high-quality human reconstruction.Accelerating Neural Rendering. Since the rendering speed of vanilla NeRF [35] is very slow, it takes several seconds to obtain an image and more time to train. Recent works [4, 16, 17, 28, 30, 31, 36, 36, 38, [43] [44] [45] 49] have been devoted to improving the speed of neural rendering. Plenoxels [16] proposed the utilization of a sparse volume, accompanied by density and spherical harmonic coefficients, for rendering purposes. TensoRF [4] decomposed the voxel grid into an aggregate of vector coefficients, aiming to reduce the size of the model for efficiency. Instant-NGP [36] introduced multiresolution hash encoding to accomplish fast rendering. 3D-GS [28] represented each scene as a collection of scalable semi-transparent ellipsoids to achieve real-time rendering. These methods are mainly used for the reconstruction of static scenes. For dynamic scenes, FPO [45] presented a novel combination of NeRF, PlenOctree [49] representation, volumetric fusion, and Fourier transform to tackle efficient neural modeling and real-time rendering of dynamic scenes. InstantAvatar [26] applied hash encoding on 3d avatar reconstruction tasks for fast rendering. Such methods require high memory costs and suffer from artifacts. In this paper, we extend 3D-GS from static scenes to dynamic human scenes. 
preliminary: 
 In this section, we briefly review the representation and pipeline of 3D Gaussian rasterization [28] in Sec. 3.1, and pose-based deformation in Sec. 3.2.Kerbl et al. [28] proposed to represent each scene as a collection of 3D Gaussians. Each 3D Gaussian 𝑃 is defined as:EQUATIONwhere 𝑥 0 represents the geometric center of a 3D Gaussian distribution, 𝑅 is a rotation matrix, 𝑆 is a scaling matrix that scales the Gaussian in three dimensions, 𝛼 0 denotes the opacity of the center, and 𝑆𝐻 refers to a set of spherical harmonic coefficients used for modeling view-dependent color distribution following standard practice [16] .The opacity of position 𝑥 in the vicinity of a 3D Gaussian 𝑃 is defined as:EQUATIONwhere the covariance matrix Σ is decomposed into the rotation matrix 𝑅 and the scaling matrix 𝑆:EQUATIONFollowing the approach of Zwicker et al. [52] , 3D Gaussians are projected to 2D image space as follows:EQUATION)where 𝑊 is a viewing transformation and 𝑃 is the Jacobian of the affine approximation of the projective transformation. Subsequently, the projected Gaussians are sorted based on their depths and rasterized to neighboring pixels according to Eq. (2) . Each pixel receives a depth-sorted list of colors 𝑐 𝑖 and opacities 𝛼 𝑖 . The final pixel color 𝐶 is computed by blending N ordered points overlapping the pixel:EQUATION)Figure 2 : Overview. The proposed animatable 3D Gaussian consists of a set of skinned 3D Gaussians and a corresponding canonical skeleton. Each skinned 3D Gaussian contains center 𝑥 0 , rotation 𝑅, scale 𝑆, opacity 𝛼 0 , and skinning weights w. First, we sample spherical harmonic coefficients 𝑆𝐻 , vertex displacement 𝛿 𝑥 , and ambient occlusion 𝑎𝑜 from the multi-head hash-encoded parameter field according to the center 𝑥 0 , where the multilayer perceptron for 𝑎𝑜 requires an additional frequency encoded time 𝛾 (𝑡) as input. Next, we concatenate the sampled parameters, the original parameters, and a shifted center 𝑥 ′ 0 in canonical space. Finally, we deform 3D Gaussians to the posed space according to the input pose 𝑆 𝑡 ,𝑇 𝑡 and render them to the image using 3D Gaussian rasterization [28] .We define each frame within the video sequence as a posed space with a timestamp 𝑡 and deform points from canonical space to posed space via linear blend skinning. We use a skinning weight field to model articulation. The skinning weight field [33] is defined as:EQUATIONwhere 𝑛 𝑏 is the count of bones and 𝑥 𝑐 is a point in canonical space. Target bone transformations B 𝑡 = {𝐵 𝑡 1 , ..., 𝐵 𝑡 𝑛 𝑏 } in frame 𝑡 can be calculated from the input poses and the corresponding skeleton as follows:EQUATIONwhere S 𝑡 = {𝜔 𝑡 1 , ..., 𝜔 𝑡 𝑛 𝑏 } refers to the rotation Euler Angle of each joint in frame 𝑡 (world rotation for 𝜔 𝑡 1 and local rotation for the rest), 𝑇 𝑡 is the world translation in frame 𝑡, and J = {𝐽 1 , ..., 𝐽 𝑛 𝑏 } is the local position of each joint in canonical space. Please refer to the supplementary materials for the detailed calculation process.Then we can deform points 𝑥 𝑐 in canonical space to the posed space via linear blend skinning:EQUATIONwhere 𝑥 𝑡 refers to a point in frame 𝑡. 
method: 
 Given a video sequence {𝐼 𝑡 } 𝑇 𝑡 =1 with one or more moving humans and their poses S 𝑡 ,𝑇 𝑡 , we reconstruct an animatable 3D Gaussian representation for each person in seconds. For multi-view video sequences, we reconstruct time-dependent ambient occlusion to achieve high-quality novel view synthesis.In this section, we first introduce our animatable 3D Gaussian representation in Sec. 4.1. Then we deform 3D Gaussians from canonical space to posed space, as introduced in Sec. 3.2 and Sec. 4.2. Finally, we run 3D Gaussian rasterization (Sec. 3.1) to render an image for specific camera parameters. Moreover, we propose to build time-dependent ambient occlusion (Sec. 4.3) for multiview, multi-person, and wide-range motion tasks, which enables our algorithm to fit dynamic shadows caused by occlusion. The pipeline of the proposed method is illustrated in Fig. 2 .3D Gaussians with Skeleton. We bind 3D Gaussians to a corresponding skeleton to enable the linear skinning algorithm in Eq. ( 8). The proposed animatable 3D Gaussian representation consists of a set of skinned 3D Gaussians and a corresponding skeleton J mentioned in Eq. (7) . The skinned 3D Gaussian 𝑃 𝑠𝑘𝑖𝑛 is adapted from the static 3D Gaussian 𝑃 in Eq. ( 1) and defined as follows:EQUATION)where w is the skinning weights, and 𝛿 𝑥 is the vertex displacement. Directly optimizing the skinning weights from random initialization is a significant challenge. Instead, we treat the skinning weights as a strong prior for a generic human body model unrelated to any specific human shape. In practice, we initialize the skinning weights and positions of Gaussian points using a standard skinned model as discussed in Sec. 4.4. During optimization, the skinning weights w are kept fixed, while the vertex displacement 𝛿 𝑥 and skeleton J are optimized to capture the shape and motion of an individual accurately. We shift the Gaussian center 𝑥 0 before implementing pose-based deformation in Eq. ( 8):EQUATIONMulti-Head Hash-encoder. We note that using per-vertex colors performs poorly in deformable dynamic scenes [28] . Since the deformation of Gaussians from canonical space to posed space is initially uncertain, it needs more samples and iterations to reach convergence. Moreover, rendering based on 3D Gaussian rasterization can only backpropagate the gradient to a finite number of Gaussians in a single iteration, which leads to a slow or even divergent optimization process for dynamic scenes. To address this issue, we suggest sampling spherical harmonic coefficients 𝑆𝐻 for each vertex from a continuous parameter field, which is able to affect all neighboring Gaussians in a single optimization. Similarly, using unconstrained per-vertex displacement can easily cause the optimization process to diverge in dynamic scenes. Therefore, we also model a parameter field for vertex displacement. For the remaining parameters in Eq. ( 9), we store them in each point to preserve the ability of the 3D Gaussian to fit different shapes. We define the parameter fields as follows:EQUATIONIn order to acquire pose-dependent deformation, we add pose 𝑆 𝑡 as an additional input for vertex displacement:EQUATION)Since our animatable 3D Gaussian representation is initialized by a standard human body model, the centers of 3D Gaussians are uniformly distributed near the human surface. We only need to sample at fixed positions near the surface of the human body in the parameter fields. This allows for significant compression of the hash table for the hash encoding [36] . Thus, we choose the hash encoding to model our parameter field to reduce the time and storage consumption.Optionally, we provide UV-encoded spherical harmonic coefficients, allowing fast processing of custom human models with UV coordinate mappings. UV encoding potentially achieves higher reconstruction quality compared to hash encoding. The UV mapping for spherical harmonic coefficients 𝑆𝐻 is defined as follows:EQUATIONwhere 𝑢, 𝑣 is the UV coordinate of a 3D Gaussian.We use poses to guide the deformation of 3D Gaussians illustrated in Fig. 3 . We introduce pose-based deformation which transforms the point position from the canonical space to the posed space in Sec. 3.2. For 3D Gaussians, we also need to deform their orientation to achieve a consistent anisotropic Gaussian distribution at different poses. Using the same orientation in different poses causes the 3D Gaussian to degenerate into Gaussian spheres. Same as Eq. ( 8), we apply the linear blend skinning to the rotation 𝑅 of 3D GaussiansCanonical Space Posed Space = =1 = ( =1 ) -1Figure 3 : 3D Gaussian Deformation. The rotation 𝑅 𝑐 of 3D Gaussian in canonical space is deformed into the posed space using Eq. ( 14), while view direction 𝑑 𝑡 is implemented the inverse transformation in Eq. (15) .as follows:EQUATIONwhere 𝑅 𝑐 is the rotation in canonical space and 𝑅 𝑡 is the rotation in posed space in frame 𝑡.The view direction for the computation of color based on spherical harmonics should be deformed to canonical space, in order to achieve consistent anisotropic colors. Hence, we apply the inverse transformation of the linear blend skinning to the view direction:EQUATIONwhere 𝑑 𝑐 is the direction in canonical space and 𝑑 𝑡 is the direction in posed space in frame 𝑡.We implement the mentioned transformations by extending the 3D Gaussian rasterizer [28] and explicitly derive the gradients for all parameters. This makes the time overhead for the pose-based deformation of 3D Gaussians almost negligible.We propose a time-dependent ambient occlusion module to address the issue of dynamic shadows in specific scenes. We additionally model an ambient occlusion factor 𝑎𝑜 ∈ [0, 1] on top of the skinned 3D Gaussian in Eq. ( 9) as follows:EQUATIONWe calculate the color after considering the ambient occlusion for each individual Gaussian as follows:EQUATIONwhere Y refers to the spherical harmonics. As discussed in Sec. 4.1, we also employ hash encoding for the ambient occlusion 𝑎𝑜, since shadows should be continuously modeled in space. Furthermore, we introduce an additional input of the positional encoding [35] of time 𝑡 in the MLP module of the hashing encoding [36] in order to capture time-dependent ambient occlusion. The parameter field for ambient occlusion is defined as:EQUATIONwhere function 𝛾 (•) refers to the positional encoding used in NeRF [35] .At the beginning of training, we use a fixed ambient occlusion to allow the model to learn time-independent spherical harmonic coefficients. We start optimizing the ambient occlusion after the color stabilizes.Animatable 3D Gaussian Initialization. The initialization of the 3D Gaussian has a significant impact on the quality of optimization results. Improper initialization can even lead to divergence in the optimization process. For static 3D Gaussians [28] , the Structurefrom-Motion (SFM [41] ) method is used to obtain the initial 3D Gaussian point cloud, which provides a very good and dense initialization.However, for dynamic scenes, obtaining an initial point cloud using SFM is a huge challenge. Therefore, we use a standard skinned model to initialize our deformable 3D Gaussian representation. This standard skinned model should include a set of positions and skinning weights w, and a skeleton J corresponding to the input poses. We recommend upsampling the vertices of the input model to around 100,000 to achieve high reconstruction quality. Specifically, for the SMPL model [33] , we randomly sample K additional points (we set K=20 based on experimental results) within the neighborhood of each vertex and directly copy their skinning parameters to get a model of around 140,000 points.Loss Function. The proposed animatable 3D Gaussian representation is capable of accurately fitting dynamic scenes containing moving humans, thus eliminating the need for additional regularization losses. We directly employ a combination of L 1 and D-SSIM term:EQUATION)where we use 𝜆 = 0.2 following the best practices of the static 3D Gaussian [28] . 
experiments: 
 We evaluate the speed and quality (PSNR, SSIM [46] , and LPIPS [51] ) of our method on monocular scenes (Sec. 5.1) and multi-view scenes (Sec. 5.2). Comparative experiments with state-of-the-art methods [5, 26] show the superiority of our method. In addition, we provide extensive ablation studies of our methods. In all experiments, we evaluate our approach on a single RTX 3090 and do not use any predicted body parameters but a generic template.We use the PeopleSnapshot [1] and ZJU-MoCap [37] dataset to evaluate the performance of our method in single-human, monocular scenes.PeopleSnapshot. Following the previous approach [26] , we downsample the images to 540×540 resolution and use the SMPL [33] model for initialization. Since this dataset does not contain timestamps and drastic shading variations, we do not use the timedependent ambient occlusion module.We provide quantitative comparisons in Tab. 1 and visual comparisons in Fig. 4 . Compared to InstantAvatar [26] and Anim-NeRF [5] , our method achieves higher reconstruction quality in a shorter training time (5s and 30s). As shown in Fig. 4 , our method solves the problem of artifacts that have occurred in the previous methods. ZJU-MoCap. We pick 5 sequences (377, 386, 387, 392, 394) in the ZJU-MoCap dataset to evaluate our method and select camera 1 as the train set and the remaining 22 cameras as the test set. In keeping with baselines [26, 37, 50] , we downsample the images to 512 × 512 resolution.InstantAvatar Ours InstantAvatar OursTab. 2 shows that our method is superior to other baseline methods on SSIM, and also achieves best or second best on PSNR and LPIPS. In comparison with NeuralBody [37] and MonoHuman [50] , our approach achieves a 240× faster speed in training time and a 60× faster speed in rendering speed. In comparison to the main baseline InstantAvatar [26] , our approach achieved a 2× faster speed in training time and a 15× faster speed in rendering speed.Ablation Study. We performed ablation experiments on the Peo-pleSnapshot dataset. Tab. 3 quantitatively illustrates that our proposed hash-encoded spherical harmonic coefficients, hash-encoded vertex displacement, and joint optimization, can improve the quality of the reconstruction in a short period of training time (30s). We also visualize the point cloud optimization results in Fig. 5 to show the effect of hash-encoded vertex displacement.There are few pose and shadow changes in the PeopleSnapshot [1] dataset. In order to demonstrate the reconstruction performance of the proposed method in scenes containing complex motions and dynamic shadows, we create a dataset called GalaBasketball, which is synthesized from several player models with different shapes and appearances. The GalaBasketball dataset consists of four single-human and three multi-human scenes, providing six uniformly surrounded cameras as a training set and one camera as a test set. For the single-human scenes, we provide an additional set of actions to evaluate the novel pose synthesis capability. Moreover, We provide a standard skinned model corresponding to the Gal-aBasketball dataset for initialization, which does not resemble any of the players in the dataset in terms of geometry and appearance.Novel View Synthesis. As shown in Tab. 4 and Fig. 6 , we evaluate the ability of our method and InstantAvatar [26] to synthesize novel views on the single-human scenes of GalaBasketball. Compared with InstantAvatar [26] , our approach under any setting achieves a higher-quality synthesis of novel views in a shorter training time and significantly eliminates artifacts. This proves that our animatable 3D Gaussian representation can be trained under complex motion variations and obtain high-quality human avatars. In contrast, InstantAvatar suffers from artifacts and achieves low synthesis quality, because skin weights fail to converge to the ground truth under complex motion variations.Ablation Study. We also provide the result of raw 3D-GS [28] and our result without ambient occlusion in Tab. 4. The comparison results between 3D-GS and our full methods demonstrate the validity of our proposed Animatable 3D Gaussian representation. The ablation experiment of ambient occlusion shows that the timedependent ambient occlusion helps our method achieve higher Table 4 : Comparison on Single-Human Scenes of GalaBasketball Dataset. We provide our novel view synthesis results under different settings, including without time-dependent ambient occlusion (w/o ao) and our full method, all trained for the 10 epochs (around 1 minute). We compare them with InstantAvatar [26] and raw 3D-GS [28] . novel view synthesis quality, which fits time-dependent shadow changes as shown in Fig. 7 .Novel Pose Synthesis. We render the images using the other set of actions different from the training set and compare our novel pose synthesis results with InstantAvatar in Fig. 8 . Since the novel pose synthesis task does not have a timestamp, we do not use the time-dependent ambient occlusion. The result shows that our reconstructed human avatar achieves high synthesis quality in novel poses, while InstantAvatar still suffers from artifacts.Multi-Human Scenes. We extend our method to multi-human scenes by rendering multiple human avatars simultaneously on a single RTX 3090, while the existing methods [5, 26] can not be Our approach reconstructs avatars for each person in the 5v5 scene. Each avatar can synthesize new poses individually. Fig. 9 illustrates the novel pose synthesis results using the ten human avatars reconstructed from the 5v5 scene. Please refer to the supplementary materials for video results. 
conclusion: 
 We propose a method for high-quality human reconstruction in seconds. The reconstructed human avatar can be used for real-time novel view synthesis and novel pose synthesis. Our method consists of the Animatable 3D Gaussian representation in canonical space and pose-based 3D Gaussian deformation, which extends 3D Gaussian [28] to deformable humans. Our proposed time-dependent ambient occlusion achieves high-quality reconstruction results in scenes containing complex motions and dynamic shadows. Compared to the state-of-the-art methods [5, 26] , our method takes less training time, renders faster, and yields better reconstruction results. Moreover, our approach can be easily extended to multi-human scenes.Limitations and Future Work. To implement optional modules, our approach uses multiple MLPs. However, in specific tasks, multiple MLPS can be merged into one to further reduce training consumption. Due to the complexity of the dynamic scenes, we do not add or remove Gaussian points during the training process, which leads to the quality of the reconstruction greatly affected by the number of initialized points, and reduces the reconstruction quality in the high-frequency region. Considering the training speed, our method does not optimize pose which fails in scenes with inaccurate poses. 
 
 Appendix:  


Decision: All 3 reviewers suggested acceptance for this paper. The AC concurs that this is a quality and timely paper addressing human reconstruction and rendering with animatable 3D Gaussians. Congratulations and pls revise the paper per the reviewer's comments and incorporate the results in the rebuttal file.

Reviews: review: paper_summary: paper_summary: The paper presents a new method called Animatable 3D Gaussian for reconstructing high-quality drivable human avatars. Traditional neural radiance fields can create good reconstructions but are computationally expensive and not well-suited for scenes with multiple humans and complex shadows. The proposed method overcomes these limitations by learning from input images and poses. The method outperforms a previous methods in terms of reconstruction quality, while also requiring significantly less training time, using less GPU memory and offering faster rendering speed.
strength_weakness: strength_weakness: The authors introduce a new neural representation called Animatable 3D Gaussian, which is more efficient in terms of memory and time compared to existing methods. They also propose a module for time-dependent ambient occlusion that helps recreate dynamic shadows, enhancing the realism of avatars in scenes with multiple moving humans and varying lighting conditions.
limitations: limitations: 1. **Inadequate contribution claim** The authors claim they method that  "enables 3D Gaussians to reconstruct human avatars from scratch without any preprocessing such as SFM". However, their method still requires preprocessing steps to acquire per-frame skeleton poses. It's inadequate to claim that no preprocessing is required.
2. **(Possible) Unfair comparisons.** 
Table 1 should also show converged results from various methods for fairness. Additionally, the rendering speeds in Table 2 (8 fps) do not match those reported by InstantAvatar(15fps). In figure 6 and 8, the performance of InstantAvatar is quite poor compared to what's reported in their original paper, where it's shown to handle a range of novel poses. If this discrepancy is due to the InstantAvatar not fully converging in the comparison, please ensure to compare after it has converged for accurate results.
3. **Lack of real image comparisons for novel view and novel poses.** The paper lacks a comparison of novel poses and novel views on real data; please provide the relevant results on PeopleSnapshot for fair comparison with previous methods.
paper_summary: paper_summary: This paper introduces a novel framework for dynamic and animatable human representation, leveraging skinned 3D Gaussians and pose-dependent deformation strategy on 3D Gaussians. This approach benefits from the efficiency inherent to 3D Gaussian models. To overcome the sparsity of 3D Gaussian optimization and accelerate training, the authors also incorporate a multi-head hash encoder for pose-dependent shape and appearance, as well as a time-dependent ambient occlusion. This ambient occlusion is specifically designed to effectively model scenes with complex motions.
strength_weakness: strength_weakness: I concur with the authors' key idea that 3D Gaussians are an effective representation for dynamic humans due to their learning and rendering efficiency. To utilize 3D Gaussians for human representation, the authors have:

1. Introduced animatable 3D Gaussians, which integrate 3D Gaussians with a skeletal structure to facilitate the use of linear skinning algorithms.
2. Enhanced traditional articulated neural body representations like InstantAvatar by considering vertex displacement in addition to pose-guided deformation through linear blend skinning, which improves the accuracy of shape and motion capture.
3. Developed a time-dependent ambient occlusion that is particularly effective in scenes with complex motion and dynamic shadows.
4. Demonstrated the method's effectiveness and efficiency through experiments on monocular scenes, multiview settings, and innovative multi-human scenes.
5. Conducted a well-structured ablation study that validates the impact of each component within the proposed method.
limitations: limitations: 1. In my opinion, the initialization of the 3D Gaussian for human representation without Sfm, which leverages the SMPL mesh body model, is straightforward and should not be considered a unique contribution of this paper.

2. Regarding the experiments on the ZJU Mocap dataset, the authors chose to evaluate five sequences. However, in the original papers for the baseline methods NeuralBody [37] and MonoBody [50], all nine sequences in the dataset were used for evaluation. Therefore, it would be more comprehensive to present results for all sequences in the dataset unless there are specific reasons for the selection.
paper_summary: paper_summary: This paper introduces animatable 3D Gaussians for rapid and high-quality 3D human reconstruction.  The proposed multi-head hash encoding encodes the gaussian  parameters efficiently and the time-dependent ambient occlusion improves the rendering quality a lot on the newly proposed GalaBasketball dataset. The experiments demonstrate that the proposed method requires significantly less training time and GPU memory, leading to faster rendering speeds and improved performance.
strength_weakness: strength_weakness: - The pipeline of the method is straight-forward and effective and the method needs significantly less training time compared with previous work.
- The method not only reconstructs a single human avatar but can also adapt to multi-human situations, which expands its applications.
- The proposed time-dependent ambient occlusion handles dynamic shadows well and synthesizes high-quality novel view shadows.
limitations: limitations: - The paper did not compare its method with other 3D Gaussian-based reconstruction methods. There are other 3D Gaussian methods for human reconstruction that have been accepted by CVPR and have been open-sourced. They also need less training time and memory and achieve fast rendering speed. It seems that these benefits come from 3D Gaussians, not your method.
- The paper only compares its method with InstantAvatar qualitatively. Other nerf-based methods such as instantNVR (Learning Neural Volumetric Representations of Dynamic Humans in Minutes, CVPR2023) should also be compared qualitatively and quantitatively.
- I would like to know how your model performs on videos in the wild, as opposed to those from datasets. How robust is it, and what are the results like for videos with loose clothing?