FineControlNet: Fine-level Text Control for Image Generation with
  Spatially Aligned Text Control Injection
Authors: name: Hongsuk Choi; name: Isaac Kasahara; name: Selim Engin; name: Moritz Graule; name: Nikhil Chavan-Dafle; name: Volkan Isler
Published: 2023-12-14 18:59:43

Abstract: Recently introduced ControlNet has the ability to steer the text-driven image
generation process with geometric input such as human 2D pose, or edge
features. While ControlNet provides control over the geometric form of the
instances in the generated image, it lacks the capability to dictate the visual
appearance of each instance. We present FineControlNet to provide fine control
over each instance's appearance while maintaining the precise pose control
capability. Specifically, we develop and demonstrate FineControlNet with
geometric control via human pose images and appearance control via
instance-level text prompts. The spatial alignment of instance-specific text
prompts and 2D poses in latent space enables the fine control capabilities of
FineControlNet. We evaluate the performance of FineControlNet with rigorous
comparison against state-of-the-art pose-conditioned text-to-image diffusion
models. FineControlNet achieves superior performance in generating images that
follow the user-provided instance-specific text prompts and poses compared with
existing methods. Project webpage:
https://samsunglabs.github.io/FineControlNet-project-page