Mesh-Centric Gaussian Splatting for Human Avatar Modelling with Real-time Dynamic Mesh Reconstruction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-15 15:05:25

Abstract: Real-time mesh reconstruction is highly demanded for integrating human avatar in modern computer graphics applications. Current methods typically use coordinate-based MLP to represent 3D scene as Signed Distance Field (SDF) and optimize it through volumetric rendering, relying on Marching Cubes for mesh extraction. However, volumetric rendering lacks training and rendering efficiency, and the dependence on Marching Cubes significantly impacts mesh extraction efficiency. This study introduces a novel approach, Mesh-Centric Gaussian Splatting (MCGS), which introduces a unique representation Mesh-Centric SDF and optimizes it using high-efficiency Gaussian Splatting. The primary innovation introduces Mesh-Centric SDF, a thin layer of SDF enveloping the underlying mesh, and could be efficiently derived from mesh. This derivation of SDF from mesh allows for mesh optimization through SDF, providing mesh as 0 iso-surface, and eliminating the need for slow Marching Cubes. The secondary innovation focuses on optimizing Mesh-Centric SDF with high-efficiency Gaussian Splatting. By dispersing the underlying mesh of Mesh-Centric SDF into multiple layers and generating Mesh-Constrained Gaussians on them, we create Multi-Layer Gaussians. These Mesh-Constrained Gaussians confine Gaussians within a 2D surface space defined by mesh, ensuring an accurate correspondence between Gaussian rendering and mesh geometry. The Multi-Layer Gaussians serve as sampling layers of Mesh-Centric SDF and can be optimized with Gaussian Splatting, which would further optimize Mesh-Centric SDF and its underlying mesh. As a result, our method can directly optimize the underlying mesh through Gaussian Splatting, providing fast training and rendering speeds derived from Gaussian Splatting, as well as precise surface learning of SDF. Experiments demonstrate that our method achieves dynamic mesh reconstruction at over 30 FPS. In contrast, SDF-based methods using Marching Cubes achieve less than 1 FPS, and concurrent 3D Gaussian Splatting-based methods cannot extract reasonable mesh.

Main paper: 
 introduction: 
 Human avatars play a pivotal role in a multitude of human-centric applications, ranging from movie production to telepresence and computer games. Modeling human figures, in contrast to static objects, presents distinct challenges due to their non-rigid dynamics, intricate poses, and diverse clothing variations. Intrigued by the potential applications of human avatars, numerous systems have been proposed to tackle these challenges [4, 9, [11] [12] [13] . However, many of these methods require the use of sophisticated equipment, such as dense camera rigs [9, 12, 13] , or controlled lighting setups [4, 11] for capturing human models. Subsequently, skilled artists are often tasked with manually designing a skeleton for the human model and meticulously adjusting skinning weights to achieve realistic animations [16] . Consequently, the substantial investment in equipment and human labor confines the applicability of these methods to research laboratories or larger corporations.Recent advancements in neural implicit representations have significantly reduced cost of creating human avatars by leveraging multi-view videos [8, 17, 30, 30, 45, 46, 52] . These methods typically involve learning a deformation field [8, 17, 30, 30, 45] to deform a static implicit model [25, 43] or directly modeling a dynamic implicit model [46, 52] . Some methods [31, 44] further integrate Signed Distance Fields (SDF) as implicit model to leverage its surface constraint, demonstrating promising mesh reconstruction performance. Despite their capabilities, these approaches lag in efficiency, primarily due to the inherent limitations of volumetric rendering in implicit models. Volumetric rendering requires querying dense sample points along a camera ray to simulate density distribution, leading to a high number of queries for the entire scene [23] . To address this efficiency issue, some methods have incorporated specialized data structures like Plenoctrees [49] , K-Planes [6] , or Hash Encodings [38] into human avatar creation [8] . However, these methods are still tethered to volumetric rendering and suffer from excessive querying.In parallel with the evolution of implicit representations, explicit representations remain prevalent in the industry, encompassing Mesh [15] , Voxels [20, 40] , Point Clouds [1, 5] , and Multiplane Images (MPI) [24, 53] . Notably, a Point Cloud-based technique called 3D Gaussian Splatting (3DGS) [14] has gained popularity for capturing 3D scenes and rendering them from various viewpoints. This method represents a scene using numerous small 3D Gaussians with optimized positions, orientations, appearances (represented as spherical harmonics). Rendering involves splatting a limited number of optimized Gaussians onto the image plane, offering a faster alternative to implicit representations. While 3DGS enables realistic renderings, leveraging this technique for human avatar creation poses challenges, particularly in extracting mesh from the Gaussians due to their unordered structure and lack of correspondence with the scene's actual surface. Failure in mesh extraction hinders its utilization in numerous applications, encompassing texture editing, sculpting, animation, and Physically Based Rendering.In this paper, we present a method achieving real-time mesh reconstruction for human avatars by optimizing Mesh-Centric Signed Distance Field (SDF) with high-efficiency Gaussian Splatting. The first innovation introduces Mesh-Centric SDF, which is a thin layer of SDF enveloping the underlying mesh, where Signed Distance for a specific point is its distance from the underlying mesh. Mesh-Centric SDF could be efficiently derived from mesh, allowing for mesh optimization through SDF optimization. It naturally provides mesh as 0 iso-surface, eliminating the need for slow Marching Cubes. The secondary innovation focuses on optimizing Mesh-Centric SDF with high-efficiency Gaussian Splatting. By dispersing the underlying mesh of Mesh-Centric SDF into multiple layers and generating Mesh-Constrained Gaussians on them, we create Multi-Layer Gaussians. Mesh-Constrained Gaussians involves attaching a set of flat Gaussians to triangle faces of mesh, with rotations and scaling constrained within the triangle plane. This flat representation ensures an even distribution over mesh surface, effectively aligning rendering of Gaussians with geometry of mesh. The Multi-Layer Gaussians serve as sampling layers of Mesh-Centric SDF and can be optimized with Gaussian Splatting, which would further optimize Mesh-Centric SDF and its underlying mesh. As a result, the proposed method, Mesh-Centric Gaussian Splatting (MCGS), can directly optimize the underlying mesh through efficient Gaussian Splatting, combining fast training and rendering speed of Gaussian Splatting with precise surface learning of SDF. Experimental results demonstrate that our method achieves dynamic mesh retrieval at over 30 FPS. In comparison, SDF-based methods using Marching Cubes achieve less than 1 FPS, and concurrent 3D Gaussian Splatting methods cannot extract reasonable mesh. The real-time mesh reconstruction capability will further enable seamless integration with established computer graphics pipelines for editing, sculpting, animation, and Physically Based Rendering, thereby enhancing industry engagement.In summary, our contributions include:• Proposing Mesh-Centric Gaussian Splatting (MCGS) for human avatar modelling with real-time mesh reconstruction, combining fast training and rendering of Gaussian Splatting with precise surface learning of SDF. • Introducing Mesh-Centric SDF as a novel surface representation, which is a thin layer of SDF wrapped around mesh and naturally provides mesh as 0 iso-surface, eliminating reliance on slow Marching Cubes to extract 0 iso-surface. • Optimizing Mesh-Centric SDF with high-efficiency Gaussian Splatting, achieved by generating Multi-Layer Gaussians from underlying mesh of Mesh-Centric SDF and optimizing them with Gaussian Splatting. 
related works 2.1 implicit human avatar modelling methods: 
 Starting from Neural Radiance Fields (NeRF) [23] , implicit representations have sparked a revolution in traditional reconstruction methods [7, 18, 23, 25, 43, 47, 48, 50] . These methods depict a scene as continuous functions, offering advantages in memory efficiency and high resolution. For instance, NeRF [25] directly maps a continuous 5D coordinate (including spatial coordinate and viewing direction) to volume density and view-dependent emitted radiance, serving as a successful example of representing a scene as a neural radiance field. NeRF can portray a continuous scene in arbitrary resolution and effectively learn from multi-view images using a differentiable volumetric renderer.Recent studies have integrated implicit representation into dynamic human modeling by combining a static neural implicit representation with dynamic deformation fields [19, 27, 28, 30, 32, 34, 39, 45] . These methods typically utilize SMPL [21] as a prior for skeleton-based motion, with deformation fields mainly representing non-linear deformations. Another approach involves directly learning dynamic implicit representation that conditions the neural implicit representations on human pose [46, 52] . These methods demonstrate a robust representation of pose-dependent dynamics. Some methods [31, 44] further integrate Signed Distance Fields (SDF) as implicit model to leverage its surface constraint, demonstrating promising mesh reconstruction performance.However, implicit representations have certain limitations. Firstly, they rely on volumetric rendering, which defines neural fields throughout the entire 3D space, even if the 3D object only occupies a small fraction of that space, leading to memory redundancy. Secondly, volumetric rendering requires dense sample points along a ray to accumulate as pixel color, resulting in significant computation. In contrast, explicit representations represent scenes as explicit components, providing the potential to directly project components into images with real-time efficiency. The proposed method in this work is based on an explicit representation called 3D Gaussian Splatting, which enhances real-time rendering efficiency.Explicit representations, such as mesh [15] , voxels [20, 40] , point clouds [1, 5] , and multiplane images (MPI) [24, 53] , continue to be the mainstream in production due to their high efficiency and ease of manipulation, allowing for immediate interactions like texture editing. However, developing explicit models is often costly, involving manual design that requires significant labor. To mitigate these costs, parametric models [2, 21, 29] are commonly used to fit parametric body models directly to skinned human scans. Nevertheless, these models are not suitable for modeling clothed humans, which are much more complex and require expensive 3D scans.Recently, 3D Gaussian Splatting (3DGS) [14] has emerged as an efficient alternative to implicit representation for fast inference and training with point-based rendering. This approach models the rendering process as splatting a set of 3D Gaussians onto the image plane via alpha blending, achieving state-of-the-art rendering quality with real-time inference speed and rapid training when provided with multi-view inputs. However, 3DGS is not currently compatible with existing computer graphics pipelines, and the process of extracting a mesh from millions of tiny Gaussians after optimization remains unclear.In this study, we introduce a method that achieves high-fidelity rendering and real-time mesh reconstruction using 3D Gaussian Splatting (3DGS). This innovative approach paves the way for integrating 3DGS into modern computer graphics pipelines. While some concurrent works [10, 51] also explore extracting meshes from 3DGS, they exhibit certain gaps compared to our method. These works typically involve jointly learning an implicit Signed Distance Field (SDF) and explicit Gaussians, and utilize isolated methods such as Marching Cube or Poisson Reconstruction to extract meshes from SDF or point clouds. However, this strategy relies on Marching Cube or Poisson Reconstruction, and the process of mesh extraction consistently operates at a speed slower than 1 FPS. In contrast, our method directly optimizes the underlying mesh by optimizing Gaussians, achieving high reconstruction accuracy and demonstrating promising real-time efficiency. 
proposed method: 
 In line with previous works targeting surface reconstruction for human avatars [31, 44] , we utilize a training set of 𝑇 -frame multi-view videos of a human performer captured by a sparse set of 𝐾 synchronized and calibrated cameras: I = {𝐼 𝑘 𝑡 } (𝑡 = 1 . . . 𝑇 , 𝑘 = 1 . . . 𝐾). We aim at human avatar modelling with real-time mesh reconstruction and the ability to perform actions not present in training frames I. We employ off-the-shelf methods to derive segmentation masks and fit parametric body models (SMPL [21] in this work). The pipeline of proposed method is illustrated in Figure 2 . We begin with T-pose Canonical Mesh, which is initialized as SMPL mesh and deformed using non-linear deformation 𝑓 𝑛𝑜𝑛-𝑙𝑖𝑛𝑒𝑎𝑟 and Linear Blending Skinning (LBS) to generate target-pose Deformed Mesh, as explained in Section 3.2. The Deformed Mesh serves as dynamic mesh reconstruction based on pose estimation of SMPL. Subsequently, we construct Mesh-Centric SDF around Deformed Mesh and disperse Deformed Mesh into Multi-Layer Mesh along its vertex normal vectors, serving as sampling layers of Mesh-Centric SDF, as depicted in Section 3.3. The Mesh-Centric SDF, a volume representation, offers spatial coverage of Real Surface and facilitates surface learning similar to previous SDF-based methods. To efficiently train it using Gaussian Splatting, we store Primitives P on triangle faces of T-pose Canonical Mesh and deform them with mesh deformation. These Primitives remain fixed during mesh deformation and are subsequently transformed into Multi-Layer Gaussians, as elaborated in Section 3.4. Finally, we employ Gaussian Splatting described in Section 3.1 to optimize Multi-Layer Gaussians, which further optimizes non-linear deformation 𝑓 𝑛𝑜𝑛-𝑙𝑖𝑛𝑒𝑎𝑟 , LBS, T-pose Canonical Mesh, and Primitives attached to it.3D Gaussian Splatting (3DGS) [14] captures a 3D scene using an ensemble of 3D Gaussians {G}, each characterized by position x, covariance Σ, opacity 𝛼, and color c represented by spherical harmonics (SH). To ensure positive semi-definiteness, the covariance matrix Σ is represented by scaling matrix S and rotation matrix R:EQUATIONPractically, 3DGS stores diagonal vector s ∈ R 3 of the scaling matrix and a quaternion vector q ∈ R 4 to represent rotation matrix, which can be converted to a valid covariance matrix.To optimize from image observation, 3DGS builds a differentiable Gaussian Splatting process to render 3D Gaussians into image. Specifically, 3D Gaussians are projected onto image plane, and alpha blending is employed to compute pixel color 𝐶 by blending N ordered Gaussians that overlap the pixel:EQUATIONwhere 𝛼 ′ 𝑖 denotes the learned opacity 𝛼 𝑖 weighted by the probability density of the 𝑖-th projected Gaussian at the target pixel location.We build an efficient method to obtain pose-dependent mesh reconstruction given pose estimation of SMPL. Illustrated in Figure 2 , the T-pose Canonical Mesh is initialized as SMPL mesh. For vertex v 𝑐 on Canonical Mesh, we deform it with pose-dependent non-linear deformation to obtain v 𝑛 . The non-linear deformation is estimated by a hash-encoding MLP [38] 𝑓 𝑛𝑜𝑛-𝑙𝑖𝑛𝑒𝑎𝑟 : where 𝜽 is pose parameters of SMPL. Then v 𝑛 is transformed to target-pose Deformed Mesh using Linear Blending Skinning (LBS):EQUATIONEQUATION)where 𝜽 is pose parameters of SMPL, 𝜷 is shape parameters of SMPL, and a skinning MLP 𝑓 𝑠𝑘𝑖𝑛𝑛𝑖𝑛𝑔 is learned to predict skinning weight of v 𝑛 . During inference, we could efficiently obtain posedependent mesh reconstruction by deforming vertices with nonlinear deformation 𝑓 𝑛𝑜𝑛-𝑙𝑖𝑛𝑒𝑎𝑟 and LBS.As we possess target-pose Deformed Mesh as mesh reconstruction result, we could conduct mesh smoothness loss on Deformed Mesh to regularize mesh deformation. Specifically, we incorporate regularization losses from Pytorch3d [36] concerning edge length, normal consistency, and Laplacian smoothing. Our full loss function consists of a RGB loss 𝐿 𝑅𝐺𝐵 , a mask loss 𝐿 𝑚𝑎𝑠𝑘 , a skinning weight regularization loss 𝐿 𝑠𝑘𝑖𝑛 and a mesh smoothness loss 𝐿 𝑚𝑒𝑠ℎ . More details are shown in Supplementary Material.In this section, we describe the definition of Mesh-Centric SDF and how to generate Multi-Layer Mesh using Mesh Dispersion. As depicted in Figure 3 (a), we take Deformed Mesh from Section 3.2 as underlying mesh and build Mesh-Centric Signed Distance Field (SDF) as a thin layer of SDF enveloping the underlying mesh, with the mesh itself serving as 0 iso-surface. Naturally, the Signed Distance of a point corresponds to its distance from the underlying mesh. Since Mesh-Centric SDF is derived from the underlying mesh, we can directly optimize the underlying mesh through SDF optimization, eliminating the need for time-consuming Marching Cube algorithm [22] typically used to extract 0 iso-surface in previous SDF-based methods [31, 43, 44] .Traditional SDF-based methods typically represent SDF as a Coordinate-based MLP and utilize volumetric rendering for optimization, which involves extensive querying of the MLP and significantly hinders efficiency. The proposed Mesh-Centric SDF naturally incorporates the underlying mesh as 0 iso-surface, allowing for optimization through a more efficient Mesh Dispersion strategy. Details of Mesh Dispersion are illustrated in Figure 3 (a) . Naive Gaussian Splatting in Section 3.1 achieves high efficiency and high-quality rendering. However, it exhibits poor alignment between appearance and geometry, as highlighted in previous studies [10, 42] . To mitigate this issue, we confine Gaussians to a 2D surface space defined by an underlying mesh, ensuring a consistent correspondence between appearance and geometry. This methodology enables us to achieve precise surface reconstruction by optimizing appearance through alpha blending in equation ( 2). In Figure 2 , we store Primitives P on triangle faces of mesh and convert them into Gaussians G. We start with a basic scenario where we generate Gaussian G from Primitive P attached to triangle face 𝑉 . Subsequently, we elaborate on the process of generating Multi-Layer Gaussians, as depicted in Figure 3 (b). We then describe converting Primitive P = (w, 𝛼, 𝑠, z) on triangle face 𝑉 = (v 1 , v 2 , v 3 ) to Mesh-Constrained Gaussian G = (x, 𝛼, R, S, z). For each Primitive, w = (𝑤 1 , 𝑤 2 , 𝑤 3 ) is barycentric coordinates and 𝑤 1 + 𝑤 2 + 𝑤 3 = 1, 𝛼 is opacity, 𝑠 is scale relative to triangle face, and z is multi-dimension color feature. To capture appearance dynamics like shadows, we follow a similar approach to 3DGS-Avatar [35] by storing per-Primitive color feature vectors z and employing a small pose-dependent Color MLP 𝑓 𝑐𝑜𝑙𝑜𝑟 to predict Gaussian colors c: c = 𝑓 𝑐𝑜𝑙𝑜𝑟 (z, 𝜽 ), where 𝜽 denotes pose parameters of SMPL. Given the underlying mesh, we possess knowledge of vertex order and define rotation and scaling matrices relative to triangle face. We define rotation matrix R in equation 1 as orthonormal vectors: R = [r 1 , r 2 , r 3 ]. As shown in Figure 4 , the first vector r 1 is defined as the normal vector:EQUATIONThe second vector r 2 is defined as the vector from triangle center x 𝑐 to vertex v 1 :EQUATIONwhere x 𝑐 = mean(v 1 + v 2 + v 3 ) is triangle center. Finally, the third vector r 3 is obtained through orthonormalizing the vector with respect to the existing two vectors (a single step in the Gram-Schmidt process [3] ):EQUATIONwhere orth(x; r 1 , r 2 ) = xproj(x; r 1 )proj(x; r 2 ),and proj(v, u) = ⟨v, u⟩ ⟨u, u⟩ u.For scaling parameters S, we use:EQUATIONwhere 𝑠 1 = 𝜖, 𝑠 2 = ||x 𝑐v 1 ||, and 𝑠 3 = ⟨v 2 , r 3 ⟩. 𝑠 1 corresponds with the normal vector and is fixed to be a constant small value 𝜖.Consequently, the covariance Σ of mesh-constrained Gaussian is obtained through equation ( 1) and corresponds with the shape of triangle 𝑉 . To represent more details, as shown in Figure 4 (b), we store several Primitives P = (w, 𝛼, 𝑠, z) on a same triangle face and control their position with learnable barycentric coordinate w by:EQUATIONand control their size with a learnable scale parameters 𝑠:EQUATIONSubsequently, we describe the process of generating Multi-Layer Gaussians on Multi-Layer Mesh, where their opacities are adjusted based on respective Signed Distances. As illustrated in Figure 3 (b), the opacity for G 𝑖 on layer 𝑖 is initially inherited from Primitive P and then further modified by Signed Distance 𝑙 𝑖 between layer 𝑖 and the underlying mesh. To be specific, we convert Signed Distance 𝑙 𝑖 into density 𝜎 𝑖 in accordance with [43] :EQUATION)where 𝛽 is an optimizable parameter controlling the degree of density concentration when converting SDF to density field. Then we adjust opacity 𝛼 with 𝜎 𝑖 :EQUATION)Remark. There have been several methods that involve attaching flat Gaussians onto meshes [10, 42] ; however, these methods rely on pre-captured meshes and struggle to learn accurate geometry. The proposed Mesh-Centric SDF is pivotal in learning precise geometry and distinguishes our approach from these existing techniques. In Figure 3 (c), a 2D example illustrates the learning process of Mesh-Centric SDF. Multiple positions are optimized by ray r 1 from camera 1, encompassing point 𝐴 near Real Surface and point 𝐵 farther away. When optimized with rays from camera 2, point 𝐴 near Real Surface is assigned a higher blending weight, which enhances 𝛼 𝑖 and 𝜎 𝑖 in equation (14) . Consequently, the Signed Distance 𝑙 𝑖 is reduced, bringing the underlying mesh closer to Real Surface. In contrast, the naive Mesh-Constrained Gaussians and previous methods that involve attaching flat Gaussians onto meshes [10, 42] are depicted in Figure 3 (d) . These methods only consider a single layer of Gaussians and optimize a single position from ray r 1 , resulting in a lack of spatial coverage of Real Surface. Consequently, the optimization process falls short in accurately learning geometry by failing to enhance the alpha blending weight at the intersection point of ray r 1 and Real Surface. . Following [31] , we choose 4 uniformly distributed cameras for training and the remaining cameras for testing.On ZJU-MoCap dataset, we evaluate image rendering performance on seen poses and unseen poses with PSNR and LPIPS metrics.SyntheticHuman [31] is a synthetic dataset that contains 7 animated 3D characters from RenderPeople [37] and Mixamo [26] , which provides 3D ground truth. Similarly, we choose 4 cameras for training and the remaining cameras for testing. We use this dataset to evaluate the performance of 3D reconstruction and adopt Chamfer Distance (CD) and Point-To-Surface Euclidean distance (P2S) as metrics. Units for CD and P2S are in cm. 
compared methods: 
 The compared methods include NeuralBody [33] , AniSDF [31] , and concurrent 3DGS-Avatar [35] , which are SOTA methods based on Neural Radiance Field (NeRF), Signed Distance Fields (SDF), and 3D Gaussian Splatting (3DGS), respectively. It is worth noting that 3DGS-Avatar, which is based on 3DGS, struggles to generate reasonable meshes. Therefore, our comparative evaluation with 3DGS-Avatar is focused on image rendering performance, as elaborated in Section 4.5. For completeness, we include the mesh reconstruction results of 3DGS-Avatar in the Supplementary Material.The primary strength of our method lies in its exceptional efficiency in mesh reconstruction. During inference, we leverage 𝑓 non-linear and LBS in equations ( 3) and ( 4) to rapidly generate pose-dependent meshes from Canonical Mesh, enabling real-time mesh reconstruction. Table 1 clearly demonstrates that compared methods significantly lag in mesh rendering efficiency. This performance gap arises from their approach of voxelizing the density field and using Marching Cubes [22] for mesh extraction. For 3DGS-Avatar, we apply mesh extraction methods proposed in DreamGaussian [41] . However, we observed that it fails to generate reasonable meshes when applied to human avatars, leading us to present its mesh reconstruction results in the Supplementary Material.Certain concurrent works [10, 51] concentrate on training SDF and Gaussian Splatting together for static scenes, employing voxelization and Marching Cubes for mesh extraction, which encounter similar speed limitations and have yet to explore in human avatar modelling scenarios. In contrast, our method directly optimizes Another notable strength of our approach is its efficiency in both training and image rendering, a merit derived from leveraging 3D Gaussian Splatting. While 3DGS-Avatar also leverages 3D Gaussian Splatting and demonstrates efficiency in training and rendering, it grapples with the challenge of mesh extraction from 3D Gaussians, hindering its mesh generation capabilities.In addition to significantly enhanced mesh reconstruction efficiency, quantitative results presented in Table 2 demonstrate that our method outperforms compared approaches by a substantial margin in terms of accuracy. Through a qualitative assessment depicted in Figure 5 , we can observe distinct characteristics of compared methods. NeuralBody generates noisy meshes due to the absence of surface constraints, whereas AniSDF yields smoother meshes by leveraging surface constraint of SDF. However, AniSDF tends to introduce noise in certain body parts, as evidenced by the arm of the left subject in the second row. This issue can be attributed to the utilization of SDF as an intermediary for surface representation and further adopts Marching Cube to extract mesh. In contrast, our approach learns mesh directly without relying on Marching Cube, resulting in a higher consistency with Ground Truth.Benefiting from representation capabilities of 3D Gaussian Splatting, our method demonstrates outstanding image rendering quality. We assess image rendering performance on both seen and unseen poses using ZJU-MoCap [33] dataset. As highlighted by Nerfies [28] , PSNR metric tends to favor blurry images; therefore, we emphasize the LPIPS metric in our evaluation. Quantitative analysis in Table 3 demonstrates that 3DGS-based methods, including 3DGS-Avatar and our method, achieve significantly better LPIPS metrics compared to Neural Body and AniSDF, showcasing the enhanced representation abilities of 3D Gaussian Splatting. Furthermore, our method exhibits superior performance compared to 3DGS-Avatar due to the additional surface constraints introduced by constraining Gaussians on the mesh. In terms of the PSNR metric, the differences are relatively minor, with our proposed method performing competitively with the compared approaches. Qualitative comparison in Fig. 6 showcases the remarkable visual quality of our MCGS (Mesh-Centric Gaussian Splatting) method, particularly in capturing clean and pose-dependent non-linear details. This improvement aligns with the enhancements in LPIPS metric and reinforces the notion that LPIPS is a more representative measure of visual quality compared to PSNR. Neural Body and AniSDF exhibit twisted and oversmoothed renderings, especially for unseen poses. For instance, the renderings in the third row for unseen poses illustrate that Neural Body and AniSDF miss most details on the back. Conversely, the 3DGS-based methods (3DGS-Avatar and ours) generate realistic wrinkles. Our method, in comparison to 3DGS-Avatar, captures more details, as evident in the second row of seen poses, where 3DGS-Avatar produces overly smoothed results while our method represents realistic wrinkles.We perform a comprehensive ablation study on subject 387 of ZJU-MoCap dataset, which includes investigating the significance of Mesh-Centric SDF, the importance of learning mesh and appearance dynamics, and the impact of subdividing initialized SMPL mesh. As ZJU-MoCap dataset lacks ground truth mesh, we train the proposed method using all cameras for a single frame to generate a reference mesh, as depicted in Figure 7 . The high accuracy and consistency of the reference mesh with the reference image highlight a promising direction for applying our method to static scene reconstruction. i) Mesh-Centric SDF. We introduce Mesh-Centric SDF in Section 3.3 as a novel surface representation, where the underlying mesh serves as 0 iso-surface and can be optimized using Gaussian Splatting through the generation of Multi-Layer Gaussians from the underlying mesh. This strategy is crucial for capturing precise geometry as shown in Figure 3 . To evaluate this, we exclude Mesh-Centric SDF and directly train Mesh-Constrained Gaussians without adjusting opacity 𝛼 using Equation ( 14). This variant of the model is denoted as 'w/o Mesh-Centric SDF' in Table 4 and Figure 7 , illustrating its inability to learn accurate geometry and only generating a skinned model. This outcome underscores the critical importance of Mesh-Centric SDF in learning precise geometry. ii) Learning mesh and appearance dynamics. We model mesh and appearance dynamics using two MLP networks, denoted as 𝑓 𝑛𝑜𝑛-𝑙𝑖𝑛𝑒𝑎𝑟 and 𝑓 𝑐𝑜𝑙𝑜𝑟 . By excluding these networks, we introduce entries 'Fixed Mesh' and 'Fixed Appearance' in Table 4 and Figure 7 correspondingly. The comparison in Figure 7 reveals that these two variants are unable to capture accurate geometry like the hood. Furthermore, Table 4 demonstrates that they exhibit significantly lower PSNR and LPIPS metrics, underscoring the efficacy of our approach in learning mesh and appearance dynamics. iii) Subdivision of SMPL mesh. We subdivide SMPL mesh once to enhance the level of detail in geometry. To evaluate the impact of this strategy, we create two variations termed 'w/o Subdivision' and 'Double Subdivision', where one omits subdivision and the other undergoes subdivision twice. As depicted in Table 4 and Figure 7 , increased subdivision results in improved performance both quantitatively and qualitatively. However, excessive subdivision leads to an abundance of faces and Gaussians, as each face is assigned at least one Gaussian. Consequently, the rendering speed drops below real-time (30 FPS). Hence, we opt to perform subdivision only once on SMPL mesh, which already yields superior mesh reconstruction, as evidenced in Table 2 , while maintaining real-time efficiency. 
limitations and future work: 
 The Mesh-Centric Gaussian Splatting (MCGS) approach proposed in this work has shown promising results in realistic image rendering and real-time mesh reconstruction. However, there exist certain limitations. Firstly, the reliance on a coarse mesh (such as SMPL in this context) for aligning information across video frames and as an initialization may not be feasible in scenarios involving loose cloth or attachments. One potential avenue for improvement could involve combining a point cloud reconstructed from MVS (similar to the initialization in 3DGS [14] ) and constructing an initialized mesh from the point cloud. Secondly, the optimization process of the underlying mesh does not concern adaptive subdivision or undivision and may encounter challenges when dealing with excessively loose cloth. Exploring the incorporation of adaptive subdivision or undivision algorithm during optimization presents an intriguing direction, which could also extend the proposed method to scene reconstruction and enable direct mesh training from multi-view images. These issues are left for future investigation. 
conclusion: 
 In this paper, we introduce Mesh-Centric Gaussian Splatting (MCGS), a novel method for human avatar modelling with real-time mesh reconstruction from multi-view video. Firstly, we present Mesh-Centric SDF as a unique surface representation, consisting of a thin layer of SDF enveloping an underlying mesh, where the mesh itself serves as 0 iso-surface. This approach eliminates the need for Marching Cubes to extract 0 iso-surface and enables real-time mesh reconstruction. Secondly, we construct Multi-Layer Mesh from the underlying mesh of Mesh-Centric SDF, which acts as sampling layers, and generate Mesh-Constrained Gaussians on these layers. This process facilitates training Mesh-Centric SDF using efficient Gaussian Splatting. Experimental results demonstrate that our method achieves real-time mesh reconstruction and SOTA performance in both image rendering and mesh reconstruction. This research establishes a pathway for direct learning meshes from multi-view videos, offering substantial potential for merging 3D reconstruction research with computer graphics in practical applications.  
 
 Appendix:  


Decision: This paper focuses on human avatar modeling with real-time dynamic mesh reconstruction via the proposed mesh-centric Gaussian Splatting. It can extract mesh from Mesh-Centric SDF at over 30FPS, which is much faster than existing methods.

This paper accept three final borderline accepts and one initial Borderline reject. The reviewer who rates as Borderline reject didn't update the final recommendation. I read the author's rebuttal, and found that the rebuttal well address the third reviewer's concern. Therefore the paper can be accepted for publication at ACM MM.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel technique to optimize human body meshes obtained using off-the-shelf pose estimators and semantic segmentation models from multiple views. A pose-dependent deformation model deforms the template mesh in canonical pose, and then the deformed template is skinned in the estimated pose using LBS. The paper introduces multi-layer meshes constructed by deforming vertices along their normals. Gaussians are attached to these multi-layer meshes, making it renderable. The deformation model and the gaussians are optimized to reduce photo-consistency and mask-based losses with regularization.
strength_weakness: strength_weakness: The paper is well written and easy to follow.
Conceptually if an SDF can be derived using a skeleton or a coarse mesh then the method can be used to get high quality output.  
The approach seems to produce high quality human meshes compared to other approaches. 
LPIP values for the generated mesh renders are better than comparable methods.
limitations: limitations: The work relies on SMPL or some kind of a prior mesh for any of it to work. I feel this should be clearly shown in the Fig 1, so as to not over emphasizing the methods capabilities. 

Since the approach needs a coarse mesh to begin with the usefulness of this approach compared to 3DGS+MC or 3DGS combined with a meshing technique that doesn't require a reference/coarse initial mesh is greatly reduced. 

Given that the Gaussians here represent flat surfaces and the fact that the surface being optimized is an explicitly defined mesh, why did the authors pick gaussians as the preferred primitive? Would it have been more straightforward to use differentiable rasterization in this case?

Comparisons on novel view synthesis with basic 3DGS, say, are missing. It would be informative to see if optimizing for a mesh also improves output image quality.

The paper claims the deformation model as one of the contributions, but it does not seem to make too big of a difference. Related, SMPL includes its own pose-dependent shape correction term (pose blend-shapes) to deal with skinning artifacts. Did the authors include this term in the ablation without the deformation model ("Fixed mesh" if I understand correctly)?

Typos:
Table 1 and Figure 1 -- Incorrectly bolded numbers
line 316 -- Prelininary -> Preliminary
paper_summary: paper_summary: The paper introduces MCGS for real-time mesh reconstruction of human avatars from multi-view video. This method uses a Mesh-Centric Signed Distance Field (SDF) representation, which envelops the underlying mesh, and optimizes it through Gaussian Splatting. By dispersing the underlying mesh into multiple layers and generating Mesh-Constrained Gaussians, MCGS effectively aligns Gaussian rendering with mesh geometry, allowing for precise surface learning. Experimental results demonstrate that MCGS achieves over 30 FPS in dynamic mesh reconstruction.
strength_weakness: strength_weakness: - The proposed method achieves dynamic mesh reconstruction at over 30 FPS and outperforms traditional SDF methods that rely on Marching Cubes.

- The introduced Mesh-Centric Signed Distance Field (SDF) is a novel surface representation

- The proposed method has notable better quality than baselines
limitations: limitations: - It would be better if the authors can show some video results.
- From Table 4., It seems that without subdivision, the results are better than the final model, why the authors are still using that?
paper_summary: paper_summary: This study introduces a novel approach, Mesh-Centric Gaussian Splatting (MCGS), which introduces a unique representation Mesh-Centric SDF and optimizes it using high-efficiency Gaussian Splatting。This method can directly optimize the underlying mesh through Gaussian Splatting, providing fast training and rendering speeds derived from Gaussian Splatting, as well as precise surface learning of SDF.
strength_weakness: strength_weakness: This study presents an innovative approach called Mesh-Centric Gaussian Splatting (MCGS), which introduces a novel Mesh-Centric Signed Distance Function (SDF) representation. This new representation envelops the underlying mesh with a thin layer of SDF, which can be efficiently derived from the mesh itself. This method enables mesh optimization via SDF, using the mesh as the zero iso-surface and eliminating the need for the slower Marching Cubes algorithm.
The primary contribution of this study is the introduction of the Mesh-Centric SDF. By deriving SDF from the mesh, the process allows for direct optimization of the mesh through SDF representation. The secondary contribution is the optimization of this Mesh-Centric SDF using high-efficiency Gaussian Splatting. 
In detail, the approach involves dispersing the underlying mesh of the Mesh-Centric SDF into multiple layers, then generating Mesh-Constrained Gaussians within these layers to create Multi-Layer Gaussians. These Mesh-Constrained Gaussians are confined within a 2D surface space defined by the mesh, ensuring precise alignment between Gaussian rendering and the mesh geometry.
limitations: limitations: 1. The article only uses two datasets. It is recommended to add another dataset for validation and analysis of the model. Besides analyzing the advantages of your model, please also discuss its disadvantages.
2. It is suggested to describe the technical details of your implementation more thoroughly, so interested researchers can replicate your work.
3. The Mesh-Centric Gaussian Splatting (MCGS) method presented in the article shows significant advantages in real-time mesh reconstruction and image rendering. However, the comparative experiments should more thoroughly discuss the comparisons with other methods, including specific performance metrics and experimental results.
4. The description of the Mesh-Centric SDF and Gaussian Splatting optimization process is unclear and lacks sufficient detail and explanation. This may prevent readers from accurately understanding the implementation steps of the method.
5. When introducing the new method, the article does not sufficiently discuss its distinctions and innovations compared to existing research. The lack of comparative analysis may make it difficult for readers to understand the advantages of this method over existing techniques.
paper_summary: paper_summary: This work focuses on human avatar modeling with real-time dynamic mesh reconstruction. It proposes Mesh-Centric Gaussian Splatting (MCGS), which is trained to produce image rendering and mesh from a multi-view video. 

MCGS consists of two key parts: Mesh-Centric SDF and Multi-Layer Gaussians. Mesh-Centric SDF can be seen as a counterpart of mesh. Multi-Layer Gaussians are used to optimize Mesh-Centric SDF. Rendering Multi-Layer Gaussians as images, MCGS can optimize Mesh-Centric SDF with image observations. After optimizing, meshes can be extracted from Mesh-Centric SDF at over 30 FPS, which is much faster than existing methods.
strength_weakness: strength_weakness: 1. The experiments are sufficient.
2. The mesh reconstruction is quite fast to meet real-time requirements.
limitations: limitations: 1. The method section's subtitles are confusing at first glance. I cannot recognize the connections between the method details in the introduction and these subtitles.
2. The details of the method are not clear. I have some questions below:
   1. Why construct multi-layer mesh surfaces from a single SDF layer with random normal vectors? I cannot find the reason to replace a single and accurate mesh surface with multi-layer mesh surfaces.
   2. Are these normal vectors learnable? If not, a set of completely random normals introduces uncertainty to the reconstructed mesh surfaces.
   3. The same questions to the multi-layer design in Gaussian Spaltting.