ControlMat: A Controlled Generative Approach to Material Capture
Authors: name: Giuseppe Vecchio; name: Rosalie Martin; name: Arthur Roullier; name: Adrien Kaiser; name: Romain Rouffet; name: Valentin Deschaintre; name: Tamy Boubekeur
Published: 2023-09-04 16:18:49

Abstract: Material reconstruction from a photograph is a key component of 3D content
creation democratization. We propose to formulate this ill-posed problem as a
controlled synthesis one, leveraging the recent progress in generative deep
networks. We present ControlMat, a method which, given a single photograph with
uncontrolled illumination as input, conditions a diffusion model to generate
plausible, tileable, high-resolution physically-based digital materials. We
carefully analyze the behavior of diffusion models for multi-channel outputs,
adapt the sampling process to fuse multi-scale information and introduce rolled
diffusion to enable both tileability and patched diffusion for high-resolution
outputs. Our generative approach further permits exploration of a variety of
materials which could correspond to the input image, mitigating the unknown
lighting conditions. We show that our approach outperforms recent inference and
latent-space-optimization methods, and carefully validate our diffusion process
design choices. Supplemental materials and additional details are available at:
https://gvecchio.com/controlmat/.