Concept Lancet: Image Editing with Compositional Representation
  Transplant
Authors: name: Jinqi Luo, name: Tianjiao Ding, name: Kwan Ho Ryan Chan, name: Hancheng Min, name: Chris Callison-Burch, name: René Vidal

Diffusion models are widely used for image editing tasks. Existing editing
methods often design a representation manipulation procedure by curating an
edit direction in the text embedding or score space. However, such a procedure
faces a key challenge: overestimating the edit strength harms visual
consistency while underestimating it fails the editing task. Notably, each
source image may require a different editing strength, and it is costly to
search for an appropriate strength via trial-and-error. To address this
challenge, we propose Concept Lancet (CoLan), a zero-shot plug-and-play
framework for principled representation manipulation in diffusion-based image
editing. At inference time, we decompose the source input in the latent (text
embedding or diffusion score) space as a sparse linear combination of the
representations of the collected visual concepts. This allows us to accurately
estimate the presence of concepts in each image, which informs the edit. Based
on the editing task (replace/add/remove), we perform a customized concept
transplant process to impose the corresponding editing direction. To
sufficiently model the concept space, we curate a conceptual representation
dataset, CoLan-150K, which contains diverse descriptions and scenarios of
visual terms and phrases for the latent dictionary. Experiments on multiple
diffusion-based image editing baselines show that methods equipped with CoLan
achieve state-of-the-art performance in editing effectiveness and consistency
preservation.