Title: ControlSwap: Controllable Personalized Subject Swapping
Primary area: generative models
Keywords: diffusion models, image editing, personalized subject swapping

Abstract: Personalized subject swapping aims to replace a source concept in a source image with a user-specified target concept while preserving the rest of the scene. Despite recent advances, two key controllability issues persist: (1) object-agnostic foreground processing impedes identity transfer under pose and context changes; and (2) background regions that should remain unchanged are unintentionally altered. This lack of controllability profoundly curtails expressivity and narrows real-world editing applicability. We introduce ControlSwap, an SDXL-based framework that redesigns the refiner into an identity-preserving Identifiner. In contrast to prior approaches that apply a uniform, object-agnostic loss across the entire image, ControlLoss decomposes optimization into object-specific and nonobject-specific objectives, enabling fine-grained, region-aware control. Experiments demonstrate improved controllability and background preservation across diverse personalization scenarios. Code and models will be released to support reproducibility and future research.

TL;DR: To address existing limitations of personalized subject swapping, ControlSwap achieves controllability—defined as foreground personalization and background preservation—via region-aware ControlLoss and an SDXL-based identity-preserving Identifiner.