UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit
  Consistency
Authors: name: Enis Simsar; name: Alessio Tonioni; name: Yongqin Xian; name: Thomas Hofmann; name: Federico Tombari
Published: 2024-12-19 18:59:58

Abstract: We propose an unsupervised model for instruction-based image editing that
eliminates the need for ground-truth edited images during training. Existing
supervised methods depend on datasets containing triplets of input image,
edited image, and edit instruction. These are generated by either existing
editing methods or human-annotations, which introduce biases and limit their
generalization ability. Our method addresses these challenges by introducing a
novel editing mechanism called Cycle Edit Consistency (CEC), which applies
forward and backward edits in one training step and enforces consistency in
image and attention spaces. This allows us to bypass the need for ground-truth
edited images and unlock training for the first time on datasets comprising
either real image-caption pairs or image-caption-edit triplets. We empirically
show that our unsupervised technique performs better across a broader range of
edits with high fidelity and precision. By eliminating the need for
pre-existing datasets of triplets, reducing biases associated with supervised
methods, and proposing CEC, our work represents a significant advancement in
unblocking scaling of instruction-based image editing.