ControlNeXt: Powerful and Efficient Control for Image and Video
  Generation
Authors: name: Bohao Peng; name: Jian Wang; name: Yuechen Zhang; name: Wenbo Li; name: Ming-Chang Yang; name: Jiaya Jia
Published: 2024-08-12 11:41:18

Abstract: Diffusion models have demonstrated remarkable and robust abilities in both
image and video generation. To achieve greater control over generated results,
researchers introduce additional architectures, such as ControlNet, Adapters
and ReferenceNet, to integrate conditioning controls. However, current
controllable generation methods often require substantial additional
computational resources, especially for video generation, and face challenges
in training or exhibit weak control. In this paper, we propose ControlNeXt: a
powerful and efficient method for controllable image and video generation. We
first design a more straightforward and efficient architecture, replacing heavy
additional branches with minimal additional cost compared to the base model.
Such a concise structure also allows our method to seamlessly integrate with
other LoRA weights, enabling style alteration without the need for additional
training. As for training, we reduce up to 90% of learnable parameters compared
to the alternatives. Furthermore, we propose another method called Cross
Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and
stable training convergence. We have conducted various experiments with
different base models across images and videos, demonstrating the robustness of
our method.