MaterialFusion: Enhancing Inverse Rendering with Material Diffusion
  Priors
Authors: name: Yehonathan Litman; name: Or Patashnik; name: Kangle Deng; name: Aviral Agrawal; name: Rushikesh Zawar; name: Fernando De la Torre; name: Shubham Tulsiani
Published: 2024-09-23 17:59:06

Abstract: Recent works in inverse rendering have shown promise in using multi-view
images of an object to recover shape, albedo, and materials. However, the
recovered components often fail to render accurately under new lighting
conditions due to the intrinsic challenge of disentangling albedo and material
properties from input images. To address this challenge, we introduce
MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that
incorporates a 2D prior on texture and material properties. We present
StableMaterial, a 2D diffusion model prior that refines multi-lit data to
estimate the most likely albedo and material from given input appearances. This
model is trained on albedo, material, and relit image data derived from a
curated dataset of approximately ~12K artist-designed synthetic Blender objects
called BlenderVault. we incorporate this diffusion prior with an inverse
rendering framework where we use score distillation sampling (SDS) to guide the
optimization of the albedo and materials, improving relighting performance in
comparison with previous work. We validate MaterialFusion's relighting
performance on 4 datasets of synthetic and real objects under diverse
illumination conditions, showing our diffusion-aided approach significantly
improves the appearance of reconstructed objects under novel lighting
conditions. We intend to publicly release our BlenderVault dataset to support
further research in this field.