Jointly Generating Multi-view Consistent PBR Textures using
  Collaborative Control
Authors: name: Shimon Vainer; name: Konstantin Kutsy; name: Dante De Nigris; name: Ciara Rowles; name: Slava Elizarov; name: Simon Donné
Published: 2024-10-09 15:21:46

Abstract: Multi-view consistency remains a challenge for image diffusion models. Even
within the Text-to-Texture problem, where perfect geometric correspondences are
known a priori, many methods fail to yield aligned predictions across views,
necessitating non-trivial fusion methods to incorporate the results onto the
original mesh. We explore this issue for a Collaborative Control workflow
specifically in PBR Text-to-Texture. Collaborative Control directly models PBR
image probability distributions, including normal bump maps; to our knowledge,
the only diffusion model to directly output full PBR stacks. We discuss the
design decisions involved in making this model multi-view consistent, and
demonstrate the effectiveness of our approach in ablation studies, as well as
practical applications.