Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
Authors: name: Jixuan He; name: Wanhua Li; name: Ye Liu; name: Junsik Kim; name: Donglai Wei; name: Hanspeter Pfister
Published: 2024-12-19 02:23:13

Abstract: As a common image editing operation, image composition involves integrating
foreground objects into background scenes. In this paper, we expand the
application of the concept of Affordance from human-centered image composition
tasks to a more general object-scene composition framework, addressing the
complex interplay between foreground objects and background scenes. Following
the principle of Affordance, we define the affordance-aware object insertion
task, which aims to seamlessly insert any object into any scene with various
position prompts. To address the limited data issue and incorporate this task,
we constructed the SAM-FB dataset, which contains over 3 million examples
across more than 3,000 object categories. Furthermore, we propose the
Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream
architecture to simultaneously denoise the RGB image and the insertion mask. By
explicitly modeling the insertion mask in the diffusion process, MADD
effectively facilitates the notion of affordance. Extensive experimental
results show that our method outperforms the state-of-the-art methods and
exhibits strong generalization performance on in-the-wild images. Please refer
to our code on https://github.com/KaKituken/affordance-aware-any.