Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
Authors: name: Lirui Zhao; name: Tianshuo Yang; name: Wenqi Shao; name: Yuxin Zhang; name: Yu Qiao; name: Ping Luo; name: Kaipeng Zhang; name: Rongrong Ji
Published: 2024-07-24 03:58:58

Abstract: This paper addresses an important problem of object addition for images with
only text guidance. It is challenging because the new object must be integrated
seamlessly into the image with consistent visual context, such as lighting,
texture, and spatial location. While existing text-guided image inpainting
methods can add objects, they either fail to preserve the background
consistency or involve cumbersome human intervention in specifying bounding
boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree,
a Text-to-Image (T2I) model that facilitates text-guided object addition with
only text control. To this end, we curate OABench, an exquisite synthetic
dataset by removing objects with advanced image inpainting techniques. OABench
comprises 74K real-world tuples of an original image, an inpainted image with
the object removed, an object mask, and object descriptions. Trained on OABench
using the Stable Diffusion model with an additional mask prediction module,
Diffree uniquely predicts the position of the new object and achieves object
addition with guidance from only text. Extensive experiments demonstrate that
Diffree excels in adding new objects with a high success rate while maintaining
background consistency, spatial appropriateness, and object relevance and
quality.