Generate Anything Anywhere in Any Scene
Authors: name: Yuheng Li; name: Haotian Liu; name: Yangming Wen; name: Yong Jae Lee
Published: 2023-06-29 17:55:14

Abstract: Text-to-image diffusion models have attracted considerable interest due to
their wide applicability across diverse fields. However, challenges persist in
creating controllable models for personalized object generation. In this paper,
we first identify the entanglement issues in existing personalized generative
models, and then propose a straightforward and efficient data augmentation
training strategy that guides the diffusion model to focus solely on object
identity. By inserting the plug-and-play adapter layers from a pre-trained
controllable diffusion model, our model obtains the ability to control the
location and size of each generated personalized object. During inference, we
propose a regionally-guided sampling technique to maintain the quality and
fidelity of the generated images. Our method achieves comparable or superior
fidelity for personalized objects, yielding a robust, versatile, and
controllable text-to-image diffusion model that is capable of generating
realistic and personalized images. Our approach demonstrates significant
potential for various applications, such as those in art, entertainment, and
advertising design.