Title: VINCIE: Unlocking In-context Image Editing from Video
Primary area: generative models
Keywords: Image Editing, Video Generation, Diffusion Model

Abstract: In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. 
To effectively learn from this data, we design three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. 
Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area.
Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.

TL;DR: We learn in-context image editing directly from videos using a block-wise causal diffusion transformer, achieving strong multi-turn image editing abilities.