Alchemist: Parametric Control of Material Properties with Diffusion
  Models
Authors: name: Prafull Sharma; name: Varun Jampani; name: Yuanzhen Li; name: Xuhui Jia; name: Dmitry Lagun; name: Fredo Durand; name: William T. Freeman; name: Mark Matthews
Published: 2023-12-05 18:58:26

Abstract: We propose a method to control material attributes of objects like roughness,
metallic, albedo, and transparency in real images. Our method capitalizes on
the generative prior of text-to-image models known for photorealism, employing
a scalar value and instructions to alter low-level material properties.
Addressing the lack of datasets with controlled material attributes, we
generated an object-centric synthetic dataset with physically-based materials.
Fine-tuning a modified pre-trained text-to-image model on this synthetic
dataset enables us to edit material properties in real-world images while
preserving all other attributes. We show the potential application of our model
to material edited NeRFs.