FreeControl: Training-Free Spatial Control of Any Text-to-Image
  Diffusion Model with Any Condition
Authors: name: Sicheng Mo; name: Fangzhou Mu; name: Kuan Heng Lin; name: Yanli Liu; name: Bochen Guan; name: Yin Li; name: Bolei Zhou
Published: 2023-12-12 18:59:14

Abstract: Recent approaches such as ControlNet offer users fine-grained spatial control
over text-to-image (T2I) diffusion models. However, auxiliary modules have to
be trained for each type of spatial condition, model architecture, and
checkpoint, putting them at odds with the diverse intents and preferences a
human designer would like to convey to the AI models during the content
creation process. In this work, we present FreeControl, a training-free
approach for controllable T2I generation that supports multiple conditions,
architectures, and checkpoints simultaneously. FreeControl designs structure
guidance to facilitate the structure alignment with a guidance image, and
appearance guidance to enable the appearance sharing between images generated
using the same seed. Extensive qualitative and quantitative experiments
demonstrate the superior performance of FreeControl across a variety of
pre-trained T2I models. In particular, FreeControl facilitates convenient
training-free control over many different architectures and checkpoints, allows
the challenging input conditions on which most of the existing training-free
methods fail, and achieves competitive synthesis quality with training-based
approaches.