GaussianProperty: Integrating Physical Properties to 3D Gaussians with
  LMMs
Authors: name: Xinli Xu; name: Wenhang Ge; name: Dicong Qiu; name: ZhiFei Chen; name: Dongyu Yan; name: Zhuoyun Liu; name: Haoyu Zhao; name: Hanfeng Zhao; name: Shunsi Zhang; name: Junwei Liang; name: Ying-Cong Chen
Published: 2024-12-15 17:44:10

Abstract: Estimating physical properties for visual data is a crucial task in computer
vision, graphics, and robotics, underpinning applications such as augmented
reality, physical simulation, and robotic grasping. However, this area remains
under-explored due to the inherent ambiguities in physical property estimation.
To address these challenges, we introduce GaussianProperty, a training-free
framework that assigns physical properties of materials to 3D Gaussians.
Specifically, we integrate the segmentation capability of SAM with the
recognition capability of GPT-4V(ision) to formulate a global-local physical
property reasoning module for 2D images. Then we project the physical
properties from multi-view 2D images to 3D Gaussians using a voting strategy.
We demonstrate that 3D Gaussians with physical property annotations enable
applications in physics-based dynamic simulation and robotic grasping. For
physics-based dynamic simulation, we leverage the Material Point Method (MPM)
for realistic dynamic simulation. For robot grasping, we develop a grasping
force prediction strategy that estimates a safe force range required for object
grasping based on the estimated physical properties. Extensive experiments on
material segmentation, physics-based dynamic simulation, and robotic grasping
validate the effectiveness of our proposed method, highlighting its crucial
role in understanding physical properties from visual data. Online demo, code,
more cases and annotated datasets are available on
https://Gaussian-Property.github.io{this https URL}.