Title: MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Avatar Creation

Digital Humans

Gaussian Splatting

Motion Modeling
Abstract: While there has been significant progress in the field of 3D avatar creation from visual observations, modeling physically plausible dynamics of humans with loose garments remains a challenging problem. Although a few existing works address this problem by leveraging physical simulation, they suffer from limited accuracy or robustness to novel animation inputs. In this work, we present MPMAvatar, a framework for creating 3D human avatars from multi-view videos that supports highly realistic, robust animation, as well as photorealistic rendering from free viewpoints. For accurate and robust dynamics modeling, our key idea is to use a Material Point Method-based simulator, which we carefully tailor to model garments with complex deformations and contact with the underlying body by incorporating an anisotropic constitutive model and a novel collision handling algorithm. We combine this dynamics modeling scheme with our canonical avatar that can be rendered using 3D Gaussian Splatting with quasi-shadowing, enabling high-fidelity rendering for physically realistic animations. In our experiments, we demonstrate that MPMAvatar significantly outperforms the existing state-of-the-art physics-based avatar in terms of (1) dynamics modeling accuracy, (2) rendering accuracy, and (3) robustness and efficiency. Additionally, we present a novel application in which our avatar generalizes to unseen interactions in a zero-shot manner—which was not achievable with previous learning-based methods due to their limited simulation generalizability. Our code will be publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MPMAvatar, a method for learning physically accurate and robust 3D garment dynamics from multi-view video using the Material Point Method (MPM). The approach simulates only the garments using an anisotropic elastoplastic constitutive model, while treating the human body and external scene objects as colliders. A small set of physical parameters—including Young’s modulus, density, and rest shape scaling—are optimized via finite-difference methods to match tracked garment meshes. For appearance, the method leverages 3D Gaussian Splatting (3DGS) with a shared canonical representation and quasi-shadowing to produce photorealistic rendering across views. Compared to PhysAvatar, MPMAvatar demonstrates superior accuracy, simulation robustness, generalization to novel interactions, and rendering quality. It also runs orders of magnitude faster than an IPC-based baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. By using MPM with a novel mesh-based collision handler, the method avoids simulation crashes common in PhysAvatar and achieves a 100% success rate across sequences.\n\n2. The method supports physically plausible cloth behavior (e.g., folds, collisions) with \\~1.1s/frame simulation time—substantially faster than PhysAvatar (\\~170s/frame).\n\n3. MPMAvatar handles unseen pose sequences and novel scene-object interactions (e.g., garments contacting cushions or sand) without retraining, attributed to the physically grounded formulation.\n\n4. The use of 3D Gaussian Splatting, combined with quasi-shadowing, results in visually superior renderings compared to mesh-based avatars.\n\n5.  The supplementary video and ablation studies isolate the contributions of anisotropic modeling, rest geometry correction, and quasi-shadowing.\n\nWeaknesses (please see the Questions section for details):\n1. Although there are some high-level descriptions in the Supp. Mat., the learning process is unclear,\n\n2. The experimental setup also has some clarity issue, which makes the effectiveness of using MPM questionable.\n\n3. Fig.2, especially the collision handling part, is confusing. While the B-spline part is explained sufficiently in Sec. 4.2.2 (main paper) and Alg. 1 (Supp. Mat.), the bottom row needs more clarification.\n\n4. Some quantities, such as edge vectors, are brought up without context and need more clarification.\n\n5. The Quasi-shadow modeling gives only marginal improvement.\n\nQUESTIONS:\n1. The method first constructs a tracked mesh from multi-view videos. Then, it performs MPM material optimization, followed by appearance optimization. Is this a stage-wise optimization, i.e., MPM and appearance are optimized separately? Also, will either of these 2 optimizations modify the underlying tracked-mesh?\n\n2. Regarding the Geometry metric reported in Table 1, are the inputs to PhysAvatar and MPMAvatar exactly the same? Does rendering have any effect on Geometry metrics in this experiment? I ask this because to confirm the effectiveness of MPM, we should rule out any other factors (e.g., 3DGS rendering) that could benefit Geometry metrics.\n\n3. In the bottom row of the right subfigure of Fig. 2, it is unclear what \"Grid Velocity\", \"Projected Velocity\", and \"Final Velocity\" are.\n\n5. The term \"edge vector\" (line 264) is introduced abruptly without a formal definition, which can hinder clarity. Further, it is unclear why this edge vector decomposition is crucial for learning the scale parameter, as the paper does not introduce anywhere how edge vectors are involved in an anisotropic constitutive model. Simply referring to Jiang et al. is not enough; there should at least be a section that builds connections between garment mesh topologies and the actual MPM function. Currently, only Eq. (1) in the Supp. Mat. treats MPM as a function of current vertices and physical parameters, but provides no more details. In this context, it is impossible to judge why the \"edge vector decomposition\" technique is helpful. Also, if MPM takes only the vertices of the garment mesh as input, how do \"edge vectors\" affect MPM at all?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MPMAvatar, a hybrid framework for learning physically plausible 3D human avatars from multi-view videos. It integrates a Material Point Method (MPM)-based cloth simulator with 3D Gaussian Splatting for high-fidelity rendering. Compared to prior works (e.g., PhysAvatar), the proposed system achieves more accurate garment dynamics, improved visual quality, and greater robustness under challenging animation inputs, including zero-shot scene interactions.\n\nKey contributions include:\n\nIntroducing a tailored MPM pipeline with an anisotropic constitutive model to model cloth dynamics.\n\nProposing a mesh-based collision handling algorithm for robust simulation against SMPL-X driven body inputs.\n\nOverall, I like the paper even I still have several concerns and the score will change according to the discussion and other reviews.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nTechnically sound and novel: Tailoring MPM with anisotropic elasticity and mesh-aware collisions enables robust garment simulation that surpasses C-IPC-based avatars in both quality and efficiency.\n\nEfficient simulation: The proposed MPM implementation reduces simulation time alot compared to PhysAvatar due to the use of MPM.\n\n**Weaknesses:**\n\nSingle-material assumption: The method assumes the entire garment shares a single set of material parameters (e.g., a global Young’s modulus E). This limits its applicability to heterogeneous clothing composed of materials with different physical properties (e.g., jeans + silk top). No support is provided for segment-wise or per-region physical parameter estimation.\n\nDiscussion of simulation parameters: MPM’s accuracy and stability are highly sensitive to parameter initializaton, particle numbers, grid resolution and time step, yet the paper provides few discussion or empirical justification for their selection. These are critical hyperparameters in practical deployment.\n\nLack of failure case analysis: The paper primarily focuses on showcasing positive results but does not include any failure cases or challenging scenarios where the method might break down. For instance: (1) Are there cases where the simulation fails due to numerical instabilities or incorrect body-garment collisions? (2) Are certain garment types (e.g., multilayered skirts, long coats) poorly handled?\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nLimited discussion of limitations: While the authors briefly mention in the conclusion that they do not support relighting or strand-based simulation, a more thorough analysis of the method's limitations is missing.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MPMAvatar, a novel framework that constructs 3D clothed human avatars from multi-view videos with robust animation and high-fidelity rendering. The method integrates a tailored Material Point Method (MPM) simulator that includes an anisotropic constitutive model for garment dynamics and a mesh-based collision handling algorithm for efficient and robust simulation. The method demonstrates strong performance over baselines and enables zero-shot generalization to novel interactions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tTailoring of MPM for effective garment dynamics modeling based on anisotropic constitutive model and mesh-based collider handling is novel.\n2.\tBoth quantitative and qualitative results show clear improvements in dynamics accuracy, rendering fidelity, and simulation robustness and efficiency.\n3.\tThe method successfully handles unseen physical interactions, demonstrating its significant zero-shot generalizable ability.\n\nWeaknesses:\n1.\tUnlike PhysAvatar, MPMAvatar does not support relighting, which could be critical for some applications.\n2.\tSome technical descriptions could benefit from clearer exposition or additional illustrations, as certain implementation details may be difficult to follow for readers unfamiliar with MPM or anisotropic constitutive models.\n3.\tAs shown in the supplemental video at 16s, minor holes are visible in the clothing.\n4.\tThe method relies on multi-view video input, which may limit its practicality in real-world applications.\n5.\tThe dynamics of non-garment regions are modeled using linear blend skinning, which may not sufficiently capture realistic motion for complex structures such as hair.\n\nQUESTIONS:\n1.\tIt is unclear how the method distinguishes between the garment and the underlying body regions in the avatar mesh. Could the authors clarify the process used to segment or identify these regions?\n2.\tIn my understanding, the rightmost image of Fig. 2 depicts the body grid with blue arrows, the garment grid with green arrows, and the collision regions between them with red arrows. However, I am not entirely certain this interpretation is correct. It would be helpful if the authors could provide a more detailed explanation to clarify the figure and improve overall readability.\n3.\tWhy is the formula for computing the final velocity in the rightmost image of Fig. 2 defined in that specific way? Is there a theoretical foundation behind this formulation?\n4.\tAre the physical parameter learning and appearance learning processes conducted separately or jointly?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents MPM-Avatar, a method for learning 3D avatars with physically grounded deformation behavior using   Material Point Method (MPM) simulations. The approach leverages sparse 3D Gaussian representations to model both geometry and appearance, while also incorporating MPM-based losses to ensure plausible physical responses to external forces. The system jointly optimizes visual fidelity and dynamic realism from monocular RGB videos and physical annotations. Key contributions include the integration of physics priors into Gaussian-based avatar learning, a differentiable MPM framework for supervision, and experiments demonstrating improved physical plausibility over prior purely visual methods.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes a framework that integrates Material Point Method (MPM) simulations with 3D Gaussian splatting for learning avatars that exhibit physically grounded deformations from monocular video. Experiments show quantitative and  qualitative improvements compared to PhysAvatar.\n\nRegarding related works, there is a lack of discussion and citation of several foundational works in differentiable cloth simulation that are directly relevant to the avatar reconstruction problem. In particular, Differentiable Cloth Simulation for Inverse Problems (Liang et al., NeurIPS 2019), DiffCloth (Li et al., SIGGRAPH 2022), and DiffAvatar (Li et al., CVPR 2024) present physically grounded, differentiable simulation frameworks that have been applied to cloth and digital avatars and addressing key aspects such as collisions.  These works should be cited and contrasted to clarify the novelty and scope of the current method. \n\n In addition, while the paper introduces a contact-aware physics loss via MPM simulation, it lacks a dedicated quantitative evaluation of the contact handling. There is no ablation or metric provided to demonstrate how well the method models contact dynamics compared to baselines or ground truth, which limits the ability to assess this component’s contribution.\n\nQUESTIONS:\n1. Cite relevant prior work and clarify distinctions: Please cite key missing works in differentiable cloth and avatar simulation, including Differentiable Cloth Simulation for Inverse Problems (Liang et al., NeurIPS 2019), DiffCloth (Li et al., SIGGRAPH 2022), and DiffAvatar (Li et al., CVPR 2024). These works are directly relevant to your goals and share conceptual ground in learning physically realistic avatars. In your revision, please clearly articulate how your approach differs from and improves upon these methods, especially in terms of simulation modeling (e.g., MPM vs. mass-spring/cloth) and supervision (e.g., RGB + physics vs. mesh-based optimization).\n\n2. Evaluation of contact handling: The paper introduces contact-aware physics supervision, but there is no targeted quantitative evaluation of this component. Could the authors include an ablation or metric-based analysis (e.g., contact force accuracy, penetration depth, collision fidelity) to isolate and verify the effectiveness of the contact modeling? This would help clarify how much contact dynamics contribute to the observed improvements.\n\n3. Quantitative evaluation of physical plausibility: While qualitative examples are strong, can you provide more quantitative metrics to assess physical realism? This would help substantiate the claim of improved physical behavior.\n\n4. Scalability and computational cost: What is the training and inference cost of incorporating MPM simulation-based gradients, especially using finite differences? Could you provide a breakdown of runtimes on the over learning process and each iteration (forward pass, MPM loss computation, Gaussian updates)? This is important for judging the method’s practicality and applicability in larger-scale setups.\n\nLIMITATIONS:\nThe authors do discuss limitations, particularly regarding physical supervision and force annotations. However, one important limitation not mentioned is the scalability of the framework with respect to the number of physical parameters. The current optimization procedure resembles that of PhysAvatar, which used a non-differentiable simulator and required two simulations per parameter, resulting in linear runtime scaling. If the same approach is used here, the method may become computationally infeasible for identifying a large number of physical parameters. Clarifying this limitation and discussing possible mitigation strategies (e.g., using differentiable simulators) would strengthen the discussion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}