VINCIE: Unlocking In-context Image Editing from Video

## Abstract

Abstract In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area.
Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.

## 1 Introduction

Recent research has devoted significant effort to the task of image editing, which enables users to generate images that closely follow editing instructions provided in text prompts. The performance of image editing models largely depends on the high-quality training data, typically composed of three elements: an input image, a text prompt describing the desired modification, and the corresponding edited image brooks2023instructpix2pix; shi2024seededit; xiao2024omnigen; wei2024omniedit; han2024ace; xia2024dreamomni; liu2025step1x. To collect such paired image data at scale, various methods have been proposed, including generating image grids wu2025less, leveraging diffusion denoising processes brooks2023instructpix2pix, and developing specialized models or tools to extract before-and-after image pairs from the web hertz2022prompt; zhuang2024task; boesel2024improving.

Very recently, the problem of *in-context image editing* openai2025gpt4o has garnered growing interest in the research community. In this setting, a target image is generated based on a contextual sequence of text prompts and previously generated images. Unlike single-turn image editing, in-context image editing supports multi-turn interactions, enabling users to iteratively refine images while maintaining visual consistency throughout the editing process.
A key challenge lies in acquiring contextualized training data that includes coherent sequences of text and images, Existing approaches to mine single-turn image editing brooks2023instructpix2pix; wu2025less; hertz2022prompt; zhuang2024task; boesel2024improving struggle to construct meaningful long-form content that is capable of capturing the dependencies and evolving intent that emerge over multiple editing steps. The lack of contextualized, quality training data remains a significant barrier to progress in this area of research.

In this paper, we approach in-context image editing from a different perspective and investigate the following research question: *Can a meaningful in-context image editing model be learned solely from videos, without using any standalone images?* Our intuition is that videos, as a rich source of multimodal information, inherently contain a long duration of visual dynamics that might facilitate the learning of multi-turn interactions. For instance, changes within a scene, such as objects entering or exiting the frame, shifts in camera focus, or character actions, provide implicit cues for learning operations like addition, removal, and modification in image editing.

To this end, we propose an approach that natively learns transitions from video data, named Video-driven IN-Context Image Editing (VINCIE). Unlike conventional image editing methods that rely on separately collected pairs of pre- and post-editing images for training, we choose not to alter the video, *i.e.*, we train on native video data, but instead provide the model with detailed annotations that describe the transitions or actions occurring within the scene. Since our method eliminates the need for paired data collection and relies solely on video, it can be trivially scaled using the vast amount of video data readily available on the web.

Specifically, we first sample a few coherent frames from a video scene, annotate the visual transitions, and identify Regions of Interest for editing (RoEs) using a pretrained Vision-Language Model (VLM). Additionally, we employ Grounding-DINO liu2024grounding and SAM2 ravi2024sam to generate RoE segmentation masks based on textual descriptions of the transitions. This process establishes our training samples, which capture context and form an interleaved multimodal sequence. Next, we design Block-Causal attention within a diffusion transformer peebles2023scalable, which applies bidirectional attention within each frame, text, and segmentation mask, and causal attention across them.

Finally, to enhance the model’s learning of contextual dependencies, we design three proxy tasks: (1) next-image prediction, which serves as the primary task in training; (2) current segmentation prediction, which enables the model to understand which regions have changed; and (3) next segmentation prediction, which prepares the model to anticipate where changes are likely to occur.

Extensive experiments show that our model, trained solely on video data, demonstrates strong in-context image editing capabilities and outperforms existing baselines on the multi-turn image editing tasks.
Scaling up the model and training data leads to substantial performance gains—for example, the success rate at the challenging 5-turn editing increases from 5% to 22% when scaling the training data from 0.25M to 10M sessions—demonstrating the scalability of our approach enabled by native video data.
Notably, to the best of our knowledge, this is the first work to demonstrate the feasibility of learning an in-context image editing model solely from video data, while also showcasing the scalability benefits of this approach.

We find that our model can learn disentangled representations of visual changes (*e.g.*, object appearance/disappearance, posture shifts, and orientation changes) purely from patterns inherent in video data. It also demonstrates reasonable generalization to scenarios that are less common in natural video, such as background changes, attribute modifications, and multi-concept compositions. As an additional benefit, our model can be used for generating consistent frames for storytelling through in-context editing.

## 2 Related Work

Image Editing.
Building on advances in foundational image generation models huang2025diffusion; ramesh2022hierarchical; saharia2022photorealistic; esser2024scaling, image editing has achieved remarkable progress. Techniques now enable a wide range of edits, including zero-shot editing li2024zone; huang2023region; wu2023latent; han2024proxedit; chen2023fec, changing object classes kim2022diffusionclip; xu2023cyclenet; ackermann2022high; yang2023object; tsaban2023ledits; gholami2023diffusion; brack2024ledits++; nie2023blessing and faces ding2023diffusionrig, free-form text-based modifications brooks2023instructpix2pix; hertz2022prompt; lin2023regeneration; dong2023prompt; zhang2023forgedit; kawar2023imagic; guo2024focus; zhang2024text; sheynin2024emu; wei2024omniedit; shi2024seededit; wang2023instructedit; li2023blip; mirzaei2024watch; miyake2025negative, mask-based edits wang2023imagen; xie2023smartbrush; couairon2022diffedit; zou2024towards; mao2024mag, point dragging mou2023dragondiffusion; shin2024instantdrag; liu2024drag; lu2024regiondrag; choi2025dragtext, and reference image-guided transformations song2023objectstitch; goel2023pair; yang2023paint. A series of recent works yang2023uni; wu2023visual; xiao2024omnigen; najdenkoska2024context; sun2024x enables edits conditioned on multiple text and images. Our work focuses on in-context image editing openai2025gpt4o, where edits are conditioned on a contextual sequence of text and previously generated images. Moreover, we explore learning from native video data, unlike existing methods that use hand-crafted synthesized data.

Data Construction Methods for Image Editing.
Constructing image editing datasets requires first designing clear and diverse editing instructions that articulate the intended visual modifications. Based on these instructions, paired image examples are then created, consisting of original images and their corresponding edited versions that reflect the specified transformations.
Single-turn image editing methods hertz2022prompt; brooks2023instructpix2pix; sheynin2024emu; shi2024seededit; zhao2024ultraedit; wei2024omniedit; hui2024hq; yang2024editworld; jin2024reasonpix2pix use pre-trained off-the-shelf models ramesh2022hierarchical; rombach2022high; brown2020language; sauer2024adversarial to construct paired data for image editing.
For example, InstructPix2Pix brooks2023instructpix2pix leverages GPT-3 brown2020language for generating editing instructions and Stable Diffusion v1.5 rombach2022high for paired image data generation. UltraEdit creates editing instructions using LLMs and combines grounding models kirillov2023segment; liu2024grounding with SDXL-Turbo sauer2024adversarial to produce region-based editing samples. Our approach relies on learning transitions from videos without manual-crafted paired data construction pipelines, bringing scalability in data preparation.

Learning from Video for Image Generation.
Video Frames naturally exhibit consistency across characters, objects, and scenes, which has inspired recent efforts to construct source and target images from sampled video frames. Leveraging such frame-based data has proven beneficial for enhancing consistency in image generation tasks, such as instructive image editing chen2024unireal; krojer2024learning, interactive image editing zhang2025framepainter; shi2024dragdiffusion, and object-level image customization chen2024anydoor.
The most recent work, *e.g.*, RealGeneral lin2025realgeneral and UES chen2024omnicreator, explored the temporal in-context consistency within video foundation models yang2024cogvideox for universal image generation and editing.
Despite notable progress, existing methods typically rely on only two frames per video, overlooking richer, long-range contextual information. Furthermore, they often depend on task-specific data construction pipelines chen2024unireal; zhang2025framepainter; chen2024anydoor, limiting their universality and scalability.
In this work, we propose constructing session-wise data with long, interleaved image-text context from native videos, and leverage it for pre-training or mid-training to learn the inherent consistency and transformations in abundant multimodal sequences.

## 3 Methodology

### 3.1 Interleaved Multimodal Sequence Construction

Figure: Figure 2: Our session data construction pipeline. We use a VLM to annotate the visual transitions. We then use the generated textual descriptions to prompt GroundingDINO+SAM2, extracting segmentation masks for the edited regions.
Refer to caption: /html/2506.10941/assets/x2.png

Figure [2](#S3.F2) shows an overview of our data construction pipeline. Starting with a video, we sparsely sample $K$ frames $(I_{0},\dots,I_{K})$ and use a vision-language model (VLM) to generate textual visual transitions $T_{i}$ describing the change from frame $I_{i}$ to $I_{i+1}$. To better capture the Regions-of-interest for editing (RoEs), we additionally annotate segmentation masks $M_{i}$ and $M_{i+1}$, which identify the changing objects in $I_{i}$ and $I_{i+1}$, respectively. Combining these elements, we construct the multimodal sequence $(I_{0},T_{0},T_{m0},M_{00},T_{m1},M_{01},I_{1},\dots,I_{K})$. $T_{m0}$ and $T_{m1}$ are predefined prompts such as “generate the mask of changing areas in the source image” and “generate the mask of changing areas in the target image”.

Frame Sampling.
We use a hybrid sampling strategy: 1) Equal-interval sampling, which selects frames at fixed time intervals (*e.g.*3 sec), and 2) Fixed-frame sampling, which uniformly samples a fixed number (*e.g.*$2\leq n\leq 6$) of frames regardless of video duration. This approach is used to capture both subtle object-level changes and significant scene-level transitions.

Visual Transition Annotation.
To describe visual transitions between frames, we
use chain-of-thought (CoT) prompting wei2022chain to instruct a VLM to perform visual transition annotation: 1) generate detailed and coherent descriptions of each frame from multiple aspects (*e.g.*, characters, objects, attributes, interactions, scenes, and environments); 2) identify semantic and visual differences between the two frames from the above aspects; 3) and summarize all the differences into a concise, instruction-style statement $T_{i}$ suitable for guiding editing.
Unlike existing interleaved datasets zhu2023multimodal; laurenccon2023obelics; chen2024comm derived from web documents and retrieval tools, our dataset is built from native videos, ensuring stronger textual and visual coherence.

Segmentation Annotation and Encoding
We explicitly annotate Regions-of-Editing (RoEs) in both adjacent frames $I_{i}$ and $I_{i+1}$. Specifically, we leverage region-level descriptions (*i.e.*, characters and objects) in the visual transition annotation as input to GroundingDINO liu2024grounding and SAM 2 ravi2024sam for extracting segmentation maps. Based on the region-level difference annotations, we determine which regions undergo visual transitions, *i.e.*, RoEs, and construct corresponding global maps by fusing local maps from the current and next session images.

### 3.2 Model Architecture

Fig. [3](#S3.F3) illustrates the overall framework. We represent the interleaved input sequence as $S=(I_{0},T_{0},\dots,T_{M-1},I_{M})$, where $T_{i}$ denotes the textual editing instruction at turn-$i$, and $I_{i}$ represents either an image or a segmentation mask.

Figure: Figure 3: Model architecture. We apply a diffusion transformer framework with block-wise causal attention (*i.e.*, the current frame/mask can not see future ones) to learn from the multimodal interleaved context, through three tasks (CSP, NSP, and NIP). Losses are only computed on noised tokens.
Refer to caption: /html/2506.10941/assets/x3.png

As our focus is on the in-context image editing task, we optimize the model by maximizing the likelihood of the next image prediction:

$$ $\log p(S)=\sum_{i=1}^{M}\log p(I_{i}\mid I_{0},\dots,T_{i-1},I_{i-1})$ (1) $$

where the conditional probability is modeled using flow-matching in the latent space, an objective commonly used in diffusion model for text-to-image rombach2022high; esser2024scaling; flux2024; podell2023sdxl and text-to-video singer2022make; wan2025wan; hong2022cogvideo; seawead2025seaweed generation tasks.
Each text instruction ($T_{i}$) and image ($I_{i}$) is encoded into latent tokens using a text encoder (*e.g.*, T5) and an image encoder (*e.g.*, VAE), respectively. The details about the text encoder and VAE are provided in the supplementary material.

Learnable <TURN> Tokens. We separate the interleaved input sequence $S$ by modality into two groups: $S=(I_{0},T_{0},\dots,T_{M-1},I_{M})\to T=(T_{0},T_{1},...,T_{M-1});I=(I_{0},...,I_{M})$. Their latent tokens are concatenated together. Since the number of text tokens at each turn may vary, we introduce $M$ special learnable tokens $\texttt{<TURN>}_{i},i=1,...,M$ to mark the turn boundary, where $\texttt{<TURN>}_{i}$ is inserted before the latent tokens of $T_{i}$.

Separate Text and Image Position Embedding. We apply 1D RoPE su2024roformer to text tokens and 3D RoPE to image tokens. The starting positions are 0 for all dimensions. This separate RoPE design aligns with our pretrained MM-DiT model, where text and image tokens are positioned continuously. Position collisions are avoided as MM-DiT employs distinct weights for each modality, and the bias terms in the linear layers effectively act as modality-specific embeddings.

Block-Causal Attention. We implement causal attention across blocks (e.g., text or image) and bidirectional attention within each block. This ensures efficient information flow among tokens while preserving causality, as past text and images cannot attend to future ones, maintaining consistency between training and inference.

Condition on Clean Context.
We model the probability of all images except the first using diffusion loss. Since diffusion loss requires noisy images as input, the latent states of the conditioning images also become noisy. To mitigate this, we input both clean and noisy tokens of each image to the model, applying an attention mask to ensure that each noisy image conditions only on the clean versions of preceding images. Further details are provided in the Appendix.

### 3.3 Context Composition Learning

We augment Eqn. [1](#S3.E1) by adding a random dropout operation $Rd$ on the context, as shown in equation:

$$ $\log p(S)=\sum_{i=1}^{M}\log p(F_{i}\mid Rd(I_{0},T_{1}),Rd(T_{m0},M_{00}),Rd(T_{m1},M_{01})\dots)$ (2) $$

where $F_{i}$ can be either the target image, RoE mask of source image, RoE mask of target image. We ensure that the image or mask required to generate the target is always retained, while only the contextual images and texts are randomly dropped. The model is jointly learning three tasks:

- •
Next Image Prediction (NIP). NIP is our primary in-context image editing task.
- •
Current Segmentation Prediction (CSP). CSP enhances the model’s grounding ability, enabling it to identify regions requiring edits while preserving consistency in other areas. This is particularly useful for local editing tasks such as removal, attribute changes, and replacements.
- •
Next Segmentation Prediction (NSP). NSP improves the model’s controllable generation by incorporating the current segmentation map into the context, aiding in dynamic layout adjustments for scenarios like shape changes and movements.

By randomly combining different contexts and tasks, the model learns essential abilities such as grounding, controllable generation, and multi-concept composition, enabling versatile in-context image editing.

Figure: Figure 4: Two examples from MSE-Bench. Compared to existing benchmarks, it covers a broader range of categories, such as posture, expression, and interaction, and emphasizes coherence and aesthetics across editing turns.
Refer to caption: /html/2506.10941/assets/x4.png

## 4 Experiments

### 4.1 Implementation Details

Data.
Through the proposed scalable data construction pipeline, we collect and annotate about 10M session instances, with the number of images in each session from 2 to 20.
For each session data, we consider RoE map with a probability of 80%. We apply a context drop rate with 20%, 70%, and 70%, to the current frame, current RoE map, and next RoE map, respectively.
During inference, the sampling step is set to 50, the classifier-free guidance scale is set to 10.
Using the proposed data construction pipeline, we collect and annotate about 10M session instances, each containing 2 to 20 images. During training, a RoE map is included with an 80% probability for each session. We apply context dropout rates of 20%, 70%, and 70% to the current frame, current RoE map, and next RoE map, respectively, with dropout applied independently at each turn. We use 50 sampling steps and set the classifier-free guidance scale to 10.

Model. We initialize our model with the weights of our in-house MM-DiT (3B and 7B), pre-trained on text-to-video tasks and architecturally similar to seawead2025seaweed; kong2024hunyuanvideo.
The 3B and 7B variants are optimized on session data for 15k and 40k steps, respectively, consuming approximately 30 and 150 hours on 256 H100 GPUs. The learning rate is set to 1e-4.

### 4.2 Multi-Turn Session Image Editing Benchmark

Existing benchmarks zhang2023magicbrush; basu2023editval; sheynin2024emu, such as MagicBrush zhang2023magicbrush, are constrained to basic editing operations, such as addition, replacement, removal, attribute modification, and background changes, and thus fall short of meeting practical user needs. Moreover, MagicBrush supports only up to three editing turns per session, with each turn treated in isolation, further diverging from real-world editing workflows. To address these limitations,

Figure: Figure 5: Category distribution of MSE-Bench. “others” includes expression, orientation, position, global, and action change.
Refer to caption: /html/2506.10941/assets/x5.png

we propose MSE-Bench (Multi-turn Session image Editing Benchmark), which comprises 100 test instances, each featuring a coherent five-turn editing session. MSE-Bench expands the range of editing categories to include more complex and realistic scenarios such as posture adjustment, object interaction, and camera view changes, as shown in Fig. [5](#S4.F5). To better reflect user intent and practical applications, we also incorporate aesthetic considerations into the construction of each editing instruction, encouraging progressive visual enhancement across turns. We show two examples in Fig. [4](#S3.F4).

For each editing instruction, multiple generated images may satisfy the user’s request. Consequently, our benchmark does not provide ground-truth images. Instead, we use GPT-4o to evaluate whether the generated image successfully follows the instructions and remains consistent with the input image. The final score for each turn is computed by averaging the corresponding success rates across all samples.

### 4.3 Comparison with State-of-the-Arts

We evaluate our model on two multi-turn image editing benchmarks: MagicBrush zhang2023magicbrush and our proposed MSE-Bench.

MagicBrush. Tab. [1](#S4.T1) reports quantitative results across three standard evaluation metrics: DINO, CLIP-I, and CLIP-T. First, our model, trained solely on interleaved video data, achieves performance comparable to SOTA methods UltraEdit and OmniGen, which rely on pairwise editing data, highlighting video data as a natural and effective source for image editing tasks. Second, with supervised fine-tuning on pairwise data, our method outperforms nearly all metrics, demonstrating that interleaved video data complements existing data creation approaches. Lastly, our model’s advantages become increasingly evident with more edit turns, *e.g.*, the DINO score improves by +0.004, +0.07, and +0.16 from turn-1 to turn-3, showcasing the benefits of learning from contextual video data.

MSE-Bench. Tab. [2](#S4.T2) presents the multi-turn editing success rates as evaluated by GPT-4o. In this setup, the generated image at turn-$i$ serves as the input for editing at turn-$i+1$. Consequently, failure at any turn propagates to subsequent turns. Existing academic methods perform poorly, with a success rate of < 2% at turn-5. In contrast, our method achieves a 25% success rate at turn-5, demonstrating the advantages of our model and the use of native video data. However, our approach still falls short compared to proprietary models like GPT-4o, which benefit from significantly larger training datasets and model sizes. Even so, GPT-4o achieves only a 62.7% success rate, highlighting the long-term value of our proposed benchmark for advancing multi-turn editing.

**Table 1: Performance comparison on MagicBrush zhang2023magicbrush (multi-turn). SFT means we carry out supervised fine-tuning on the pairwise training dataset wei2024omniedit. * indicates no use of context, *i.e.*, only the result from the previous turn is used as input. Entries by gray denote proprietary models.**
| Method | Turn-1 | Turn-2 | Trun-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DINO | CLIP-I | CLIP-T | DINO | CLIP-I | CLIP-T | DINO | CLIP-I | CLIP-T |  |
| HQEdit* hui2024hq | 0.522 | 0.696 | 0.259 | 0.441 | 0.659 | 0.248 | 0.397 | 0.637 | 0.238 |
| UltraEdit* zhao2024ultraedit | 0.755 | 0.852 | 0.289 | 0.706 | 0.827 | 0.278 | 0.683 | 0.810 | 0.266 |
| OmniGen* xiao2024omnigen | 0.874 | 0.924 | 0.273 | 0.718 | 0.851 | 0.264 | 0.586 | 0.786 | 0.261 |
| GPT-4o | 0.805 | 0.875 | 0.293 | 0.708 | 0.820 | 0.300 | 0.666 | 0.789 | 0.292 |
| Ours (3B) | 0.804 | 0.891 | 0.275 | 0.704 | 0.845 | 0.274 | 0.673 | 0.828 | 0.269 |
| Ours (7B) | 0.838 | 0.906 | 0.272 | 0.721 | 0.848 | 0.272 | 0.645 | 0.804 | 0.271 |
| Ours (7B) + SFT | 0.878 | 0.931 | 0.280 | 0.788 | 0.886 | 0.282 | 0.739 | 0.854 | 0.278 |

Figure: Figure 6: Editing success rates in 5 turns at various data scales.
Refer to caption: /html/2506.10941/assets/x6.png

### 4.4 In-depth Analysis

In-Context Editing Mitigates Artifact Accumulation. Artifact accumulation, where artifacts become more pronounced with increasing editing turns, is a common issue in multi-turn editing sheynin2024emu. We observe this phenomenon as well (upper part of Fig. [7](#S4.F7)) when using our model as a single-turn editing method, i.e., without incorporating context from previous turns. However, when all contexts are included as input, no artifacts are observed (lower part of Fig. [7](#S4.F7)).

Training on Native Video Data Introduces Addressable Subject Position-Shift.
A key challenge when training on video data is the potential for subject position shifts across editing turns, as illustrated in the upper part of Fig.[8](#S4.F8). This issue arises from the natural movement of subjects over time in videos. However, incorporating segmentation mask prediction—where the model first predicts a mask before generating the target image—substantially mitigates this drifting effect (see lower part of Fig.[8](#S4.F8)). The segmentation mask enforces consistency in unedited regions, thereby reducing positional drift.

Figure: Figure 7: In-context editing mitigates artifact accumulation issue in sequential single-turn editing.
Refer to caption: /html/2506.10941/assets/x7.png

in multi-turn image editing.
In Turn-1, where no prior context exists, adding a dummy context—comprising the original image and an instruction,
"generate the same image," prepended before Turn-1—significantly improves performance. The L1 and L2 distances are nearly halved, indicating greater consistency between the generated image and the original image in unchanged areas, as these distances are measured pixel-wise.
Impact of Context. Table [3](#S4.T3) highlights the impact of context In Turn-2 and Turn-3, where editing instructions and ground-truth images from previous turns are provided as context, adding a dummy context results in minimal improvements. This is expected, as the existing context already provides sufficient information. These findings underscore the critical role of context in multi-turn image editing tasks.

**Table 3: Impact of context on multi-turn image editing with MagicBrush. The "Dummy-Context" consists of the original image and the instruction, "generate the same image." "History" refers to providing previous turns’ ground-truth images as context. Results show that performance significantly improves when a reasonable context is included, emphasizing the importance of context in multi-turn image editing.**
| Method | L1$\downarrow$ | L2$\downarrow$ | DINO$\uparrow$ | CLIP-I$\uparrow$ | CLIP-T$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Turn-1 |  |  |  |  |  |
| w/o Context | 0.155 | 0.063 | 0.814 | 0.894 | 0.277 |
| Dummy-Context | 0.086 | 0.031 | 0.850 | 0.913 | 0.277 |
| Turn-2 |  |  |  |  |  |
| w/o Context | 0.159 | 0.067 | 0.834 | 0.902 | 0.279 |
| History | 0.099 | 0.038 | 0.845 | 0.909 | 0.278 |
| Dummy-Context | 0.087 | 0.033 | 0.869 | 0.922 | 0.280 |
| Turn-3 |  |  |  |  |  |
| w/o Context | 0.164 | 0.071 | 0.851 | 0.904 | 0.273 |
| History | 0.088 | 0.034 | 0.878 | 0.923 | 0.273 |
| Dummy-Context | 0.088 | 0.034 | 0.895 | 0.929 | 0.272 |

Impact of Segmentation Mask Prediction.
As shown in Tab. [4](#S4.T4), training with segmentation masks improves the editing success rate by 4% and 5% for Turn-1 and Turn-2, respectively, but shows no improvement for subsequent turns. However, applying the chain-of-edit strategy—predicting the RoE segmentation mask first and then the target image at each turn—yields a significant performance boost ($\sim$7%) from Turn-2 to Turn-5.

**Table 4: Success rates (GPT-4o evaluated) with and without segmentation mask prediction on MSE-Bench. The first two rows compare models trained with and without segmentation masks. The best performance is achieved when the model first predicts the RoI segmentation mask for the input image and then generates the target image. This ablation study was conducted using an intermediate checkpoint, so the reported numbers may not be directly comparable to those in other tables.**
| Method | Turn-1 | Turn-2 | Turn-3 | Turn-4 | Turn-5 |
| --- | --- | --- | --- | --- | --- |
| Train w/o Seg. | 0.8467 | 0.4733 | 0.3367 | 0.1767 | 0.1133 |
| Train w/ Seg. | 0.8867 | 0.5200 | 0.3267 | 0.1833 | 0.1033 |
| + Inf. $\rightarrow$ Cur Seg. $\rightarrow$ Image | 0.8733 | 0.5900 | 0.4067 | 0.2600 | 0.1733 |
| + Inf. $\rightarrow$ Next Seg. $\rightarrow$ Image | 0.8367 | 0.4867 | 0.3233 | 0.1967 | 0.1167 |
| + Inf. $\rightarrow$ Cur Seg. $\rightarrow$ Next Seg. $\rightarrow$ Image | 0.8667 | 0.5233 | 0.3667 | 0.1900 | 0.1100 |

Scalability. Fig. [6](#S4.F6) illustrates the editing success rate as a function of training data size. While the success rate at Turn-1 begins to saturate at 2.5M training samples, the success rate at later turns (*e.g.*, Turn-4 and Turn-5) exhibits a nearly log-linear increase with more training data. These results demonstrate the scalability of both our model and data construction pipeline.

**Table 5: Ablation study on MSE-Bench (GPT-4o evaluated success rate), to assess the impact of our video sequence data.**
| Training Data | Turn-1 | Turn-2 | Turn-3 | Turn-4 | Turn-5 |
| --- | --- | --- | --- | --- | --- |
| pairwise | 0.723 | 0.263 | 0.123 | 0.033 | 0.010 |
| sequence | 0.887 | 0.597 | 0.417 | 0.280 | 0.220 |
| sequence $\to$ pairwise | 0.880 | 0.647 | 0.483 | 0.370 | 0.250 |

Effectiveness of Our Video Sequence Data.
Table [5](#S4.T5) demonstrates the impact of incorporating our video sequence data. Using the same pretrained model, training with our video sequence data increases success rates by 16.4% and 21.0% on Turn-1 and Turn-5, respectively, compared to training solely on specialized pairwise image editing data wei2024omniedit. The highest performance is achieved by first pretraining on our video sequence data, followed by supervised fine-tuning (SFT) on pairwise data, underscoring the effectiveness of our data for continual pretraining.

### 4.5 Applications

Fig. [1](#S0.F1) showcases several emerging capabilities that arise when training our model exclusively on video data. Notably, these abilities seem to develop implicitly, as they differ from the model’s explicit training objectives:

- •
Controllable Editing: By including the segmentation mask of the region of interest in the context, users can achieve controllable editing by modifying the segmentation mask.
- •
Multi-Concept Composition: The model demonstrates the ability to compose multiple concepts together, even without explicit composition training data—a surprising emergent capability.
- •
Story Generation: Leveraging the consistent and extended context in video data, the model can generate coherent frames for storytelling through in-context editing.
- •
Chain-of-Editing: Each multi-turn editing session functions as a multimodal chain of thought, where the model interprets editing instructions, identifies regions of interest, generates RoI masks, produces target images, and iterates the process. Our model reveals the potential of video data in modeling multimodal chains of thought.

## 5 Conclusion

In this work, we explore the research question: "Can an in-context image editing model be learned solely from videos?" To address this, we propose a learning framework that enables context-aware image generation directly from native videos. We introduce a scalable data construction pipeline that transforms videos into contextual multimodal sequences, comprising sparsely sampled frames, textual visual transition descriptions, and segmentation masks of regions of interest. To model this multimodal sequence, we design block-causal attention within a DiT and train it using three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Experimental results demonstrate that our model, trained exclusively on videos, exhibits strong in-context image editing capabilities and achieves state-of-the-art performance on multiple multi-turn image editing benchmarks. Additionally, our model showcases emerging abilities such as controllable editing, multi-concept composition, story generation, and multimodal chain-of-thought, highlighting the untapped potential of video data and the effectiveness of our proposed framework.

Limitations. First, we use T5 to encode text, which restricts the model’s ability to comprehend complex instructions and generate nuanced textual outputs. Integrating a vision-language model (VLM) into the framework could significantly improve this capability.
Second, while our framework demonstrates preliminary but promising emerging abilities, these can be further enhanced through supervised fine-tuning (SFT) on high-quality, application-specific datasets.
Lastly, due to the high cost of querying VLM, we annotated only 10M training samples. Expanding both the model size and the dataset scale presents an exciting avenue for future research.

Broader Impact. Our work on scalable, context-aware image editing has the potential to democratize creative tools, enhance accessibility, streamline media production, and advance intuitive human-AI collaboration. However, it also raises important concerns, including the risk of misuse for misinformation or manipulation, privacy issues from large-scale video data, potential biases in generated content, job displacement in creative industries, and increased environmental impact due to computational demands. Addressing these challenges will require careful dataset curation, privacy safeguards, bias mitigation, responsible deployment practices, and ongoing engagement with diverse stakeholders.

## References

- (1)
Johannes Ackermann and Minjun Li.
High-resolution image editing via multi-stage blended diffusion.
arXiv preprint arXiv:2210.12965, 2022.
- (2)
Samyadeep Basu, Mehrdad Saberi, Shweta Bhardwaj, Atoosa Malemir Chegini,
Daniela Massiceti, Maziar Sanjabi, Shell Xu Hu, and Soheil Feizi.
Editval: Benchmarking diffusion based text-guided image editing
methods.
arXiv preprint arXiv:2310.02426, 2023.
- (3)
Frederic Boesel and Robin Rombach.
Improving image editing models with generative data refinement.
In The Second Tiny Papers Track at ICLR 2024, 2024.
- (4)
Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick
Schramowski, Kristian Kersting, and Apolinário Passos.
Ledits++: Limitless image editing using text-to-image models.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 8861–8870, 2024.
- (5)
Tim Brooks, Aleksander Holynski, and Alexei A Efros.
Instructpix2pix: Learning to follow image editing instructions.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 18392–18402, 2023.
- (6)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:1877–1901, 2020.
- (7)
Haodong Chen, Lan Wang, Harry Yang, and Ser-Nam Lim.
Omnicreator: Self-supervised unified generation with universal
editing.
arXiv preprint arXiv:2412.02114, 2024.
- (8)
Songyan Chen and Jiancheng Huang.
Fec: Three finetuning-free methods to enhance consistency for real
image editing.
In 2023 International Conference on Image Processing, Computer
Vision and Machine Learning (ICICML), pages 76–87. IEEE, 2023.
- (9)
Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingting Gao, Yu Wu, and Long
Chen.
Comm: A coherent interleaved image-text dataset for multimodal
understanding and generation.
arXiv preprint arXiv:2406.10462, 2024.
- (10)
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen.
Re-imagen: Retrieval-augmented text-to-image generator.
arXiv preprint arXiv:2209.14491, 2022.
- (11)
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.
Anydoor: Zero-shot object-level image customization.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 6593–6602, 2024.
- (12)
Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li,
Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al.
Unireal: Universal image generation and editing via learning
real-world dynamics.
arXiv preprint arXiv:2412.07774, 2024.
- (13)
Gayoon Choi, Taejin Jeong, Sujung Hong, and Seong Jae Hwang.
Dragtext: Rethinking text embedding in point-based image editing.
In 2025 IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV), pages 441–450. IEEE, 2025.
- (14)
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.
Diffedit: Diffusion-based semantic image editing with mask guidance.
arXiv preprint arXiv:2210.11427, 2022.
- (15)
Zheng Ding, Xuaner Zhang, Zhihao Xia, Lars Jebe, Zhuowen Tu, and Xiuming Zhang.
Diffusionrig: Learning personalized priors for facial appearance
editing.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 12736–12746, 2023.
- (16)
Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han.
Prompt tuning inversion for text-driven image editing using diffusion
models.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 7430–7440, 2023.
- (17)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas
Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic
Boesel, et al.
Scaling rectified flow transformers for high-resolution image
synthesis.
In Forty-first international conference on machine learning,
2024.
- (18)
Peyman Gholami and Robert Xiao.
Diffusion brush: A latent diffusion model-based editing tool for
ai-generated images.
arXiv preprint arXiv:2306.00219, 2023.
- (19)
Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell,
Zhangyang Wang, and Humphrey Shi.
Pair-diffusion: Object-level image editing with
structure-and-appearance paired diffusion models.
CoRR, 2023.
- (20)
Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang
Liu, Wei Liu, Wei Lu, Yichun Shi, et al.
Seedream 2.0: A native chinese-english bilingual image generation
foundation model.
arXiv preprint arXiv:2503.07703, 2025.
- (21)
Qin Guo and Tianwei Lin.
Focus on your instruction: Fine-grained and multi-instruction image
editing by attention modulation.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 6986–6996, 2024.
- (22)
Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua
Lin, and Lu Jiang.
Long context tuning for video generation.
arXiv preprint arXiv:2503.10589, 2025.
- (23)
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren,
Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al.
Proxedit: Improving tuning-free real image editing with proximal
guidance.
In Proceedings of the IEEE/CVF Winter Conference on Applications
of Computer Vision, pages 4291–4301, 2024.
- (24)
Zhen Han, Zeyinzi Jiang, Yulin Pan, Jingfeng Zhang, Chaojie Mao, Chenwei Xie,
Yu Liu, and Jingren Zhou.
Ace: All-round creator and editor following instructions via
diffusion transformer.
arXiv preprint arXiv:2410.00086, 2024.
- (25)
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel
Cohen-Or.
Prompt-to-prompt image editing with cross attention control.
arXiv preprint arXiv:2208.01626, 2022.
- (26)
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.
Cogvideo: Large-scale pretraining for text-to-video generation via
transformers.
arXiv preprint arXiv:2205.15868, 2022.
- (27)
Nisha Huang, Fan Tang, Weiming Dong, Tong-Yee Lee, and Changsheng Xu.
Region-aware diffusion for zero-shot text-driven image editing.
arXiv preprint arXiv:2302.11797, 2023.
- (28)
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei
Xiong, He Zhang, Liangliang Cao, and Shifeng Chen.
Diffusion model-based image editing: A survey.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
2025.
- (29)
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin
Zhou, and Cihang Xie.
Hq-edit: A high-quality dataset for instruction-based image editing.
arXiv preprint arXiv:2404.09990, 2024.
- (30)
Ying Jin, Pengyang Ling, Xiaoyi Dong, Pan Zhang, Jiaqi Wang, and Dahua Lin.
Reasonpix2pix: instruction reasoning dataset for advanced image
editing.
arXiv preprint arXiv:2405.11190, 2024.
- (31)
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar
Mosseri, and Michal Irani.
Imagic: Text-based real image editing with diffusion models.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 6007–6017, 2023.
- (32)
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.
Diffusionclip: Text-guided diffusion models for robust image
manipulation.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 2426–2435, 2022.
- (33)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura
Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.
Segment anything.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 4015–4026, 2023.
- (34)
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng
Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al.
Hunyuanvideo: A systematic framework for large video generative
models.
arXiv preprint arXiv:2412.03603, 2024.
- (35)
Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jampani, Eva Portelance,
Chris Pal, and Siva Reddy.
Learning action and reasoning-centric image editing from videos and
simulation.
Advances in Neural Information Processing Systems,
37:38035–38078, 2024.
- (36)
Black Forest Labs.
Flux.
[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux), 2024.
- (37)
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet
Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush,
Douwe Kiela, et al.
Obelics: An open web-scale filtered dataset of interleaved image-text
documents.
Advances in Neural Information Processing Systems,
36:71683–71702, 2023.
- (38)
Dongxu Li, Junnan Li, and Steven Hoi.
Blip-diffusion: Pre-trained subject representation for controllable
text-to-image generation and editing.
Advances in Neural Information Processing Systems,
36:30146–30166, 2023.
- (39)
Shanglin Li, Bohan Zeng, Yutang Feng, Sicheng Gao, Xiuhui Liu, Jiaming Liu, Lin
Li, Xu Tang, Yao Hu, Jianzhuang Liu, et al.
Zone: Zero-shot instruction-guided local editing.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 6254–6263, 2024.
- (40)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In Computer Vision–ECCV 2014: 13th European Conference, Zurich,
Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755.
Springer, 2014.
- (41)
Yijing Lin, Mengqi Huang, Shuhan Zhuang, and Zhendong Mao.
Realgeneral: Unifying visual generation via temporal in-context
learning with video models.
arXiv preprint arXiv:2503.10406, 2025.
- (42)
Yupei Lin, Sen Zhang, Xiaojun Yang, Xiao Wang, and Yukai Shi.
Regeneration learning of diffusion models with rich prompts for
zero-shot image translation.
arXiv preprint arXiv:2305.04651, 2023.
- (43)
Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, and Shengfeng He.
Drag your noise: Interactive point-based editing via diffusion
semantic propagation.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 6743–6752, 2024.
- (44)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing
Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al.
Grounding dino: Marrying dino with grounded pre-training for open-set
object detection.
In European Conference on Computer Vision, pages 38–55.
Springer, 2024.
- (45)
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao,
Yingming Wang, Honghao Fu, Chunrui Han, et al.
Step1x-edit: A practical framework for general image editing.
arXiv preprint arXiv:2504.17761, 2025.
- (46)
Jingyi Lu, Xinghui Li, and Kai Han.
Regiondrag: Fast region-based image editing with diffusion models.
In European Conference on Computer Vision, pages 231–246.
Springer, 2024.
- (47)
Qi Mao, Lan Chen, Yuchao Gu, Zhen Fang, and Mike Zheng Shou.
Mag-edit: Localized image editing in complex scenarios via mask-based
attention-adjusted guidance.
In Proceedings of the 32nd ACM International Conference on
Multimedia, pages 6842–6850, 2024.
- (48)
Ashkan Mirzaei, Tristan Aumentado-Armstrong, Marcus A Brubaker, Jonathan Kelly,
Alex Levinshtein, Konstantinos G Derpanis, and Igor Gilitschenski.
Watch your steps: Local image and scene editing by text instructions.
In European Conference on Computer Vision, pages 111–129.
Springer, 2024.
- (49)
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka.
Negative-prompt inversion: Fast image inversion for editing with
text-guided diffusion models.
In 2025 IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV), pages 2063–2072. IEEE, 2025.
- (50)
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang.
Dragondiffusion: Enabling drag-style manipulation on diffusion
models.
arXiv preprint arXiv:2307.02421, 2023.
- (51)
Ivona Najdenkoska, Animesh Sinha, Abhimanyu Dubey, Dhruv Mahajan, Vignesh
Ramanathan, and Filip Radenovic.
Context diffusion: In-context aware image generation.
In European Conference on Computer Vision, pages 375–391.
Springer, 2024.
- (52)
Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan
Li.
The blessing of randomness: Sde beats ode in general diffusion-based
image editing.
arXiv preprint arXiv:2311.01410, 2023.
- (53)
OpenAI.
Addendum to gpt-4o system card: Native image generation.
openai, 2025.
- (54)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 4195–4205, 2023.
- (55)
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas
Müller, Joe Penna, and Robin Rombach.
Sdxl: Improving latent diffusion models for high-resolution image
synthesis.
arXiv preprint arXiv:2307.01952, 2023.
- (56)
Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, and
Tat-Seng Chua.
Tiger: Unifying text-to-image generation and retrieval with large
multimodal models.
In The Thirteenth International Conference on Learning
Representations, 2025.
- (57)
Leigang Qu, Haochuan Li, Wenjie Wang, Xiang Liu, Juncheng Li, Liqiang Nie, and
Tat-Seng Chua.
Silmm: Self-improving large multimodal models for compositional
text-to-image generation.
In Proceedings of the Computer Vision and Pattern Recognition
Conference, pages 18497–18508, 2025.
- (58)
Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie.
Dynamic modality interaction modeling for image-text retrieval.
In Proceedings of the 44th International ACM SIGIR Conference on
Research and Development in Information Retrieval, pages 1104–1113, 2021.
- (59)
Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, and Tat-Seng
Chua.
Discriminative probing and tuning for text-to-image generation.
arXiv preprint arXiv:2403.04321, 2024.
- (60)
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua.
Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image
generation.
In Proceedings of the 31st ACM International Conference on
Multimedia, pages 643–654, 2023.
- (61)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv preprint arXiv:2204.06125, 2022.
- (62)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali,
Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson,
et al.
Sam 2: Segment anything in images and videos.
arXiv preprint arXiv:2408.00714, 2024.
- (63)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
Ommer.
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 10684–10695, 2022.
- (64)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L
Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim
Salimans, et al.
Photorealistic text-to-image diffusion models with deep language
understanding.
Advances in Neural Information Processing Systems,
35:36479–36494, 2022.
- (65)
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.
Adversarial diffusion distillation.
In European Conference on Computer Vision, pages 87–103.
Springer, 2024.
- (66)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation
image-text models.
Advances in Neural Information Processing Systems,
35:25278–25294, 2022.
- (67)
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma,
Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al.
Seaweed-7b: Cost-effective training of video generation foundation
model.
arXiv preprint arXiv:2504.08685, 2025.
- (68)
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron
Ashual, Devi Parikh, and Yaniv Taigman.
Emu edit: Precise image editing via recognition and generation tasks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8871–8879, 2024.
- (69)
Yichun Shi, Peng Wang, and Weilin Huang.
Seededit: Align image re-generation to image editing.
arXiv preprint arXiv:2411.06686, 2024.
- (70)
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang,
Vincent YF Tan, and Song Bai.
Dragdiffusion: Harnessing diffusion models for interactive
point-based image editing.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8839–8849, 2024.
- (71)
Joonghyuk Shin, Daehyeon Choi, and Jaesik Park.
Instantdrag: Improving interactivity in drag-based image editing.
In SIGGRAPH Asia 2024 Conference Papers, pages 1–10, 2024.
- (72)
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan
Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.
Make-a-video: Text-to-video generation without text-video data.
arXiv preprint arXiv:2209.14792, 2022.
- (73)
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang,
Soo Ye Kim, and Daniel Aliaga.
Objectstitch: Object compositing with diffusion model.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 18310–18319, 2023.
- (74)
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
Neurocomputing, 568:127063, 2024.
- (75)
Zeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuanjun
Xiong, Dahua Lin, and Jiaqi Wang.
X-prompt: Towards universal in-context image generation in
auto-regressive vision language foundation models.
arXiv preprint arXiv:2412.01824, 2024.
- (76)
Linoy Tsaban and Apolinário Passos.
Ledits: Real image editing with ddpm inversion and semantic guidance.
arXiv preprint arXiv:2307.00522, 2023.
- (77)
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen,
Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al.
Wan: Open and advanced large-scale video generative models.
arXiv preprint arXiv:2503.20314, 2025.
- (78)
Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka.
Instructedit: Improving automatic masks for diffusion-based image
editing with user instructions.
arXiv preprint arXiv:2305.18047, 2023.
- (79)
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy,
Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut,
et al.
Imagen editor and editbench: Advancing and evaluating text-guided
image inpainting.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 18359–18369, 2023.
- (80)
Cong Wei, Zheyang Xiong, Weiming Ren, Xeron Du, Ge Zhang, and Wenhu Chen.
Omniedit: Building image editing generalist models through specialist
supervision.
In The Thirteenth International Conference on Learning
Representations, 2024.
- (81)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language
models.
Advances in neural information processing systems,
35:24824–24837, 2022.
- (82)
Chen Henry Wu and Fernando De la Torre.
A latent space of stochastic diffusion models for zero-shot image
editing and guidance.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 7378–7387, 2023.
- (83)
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan
Duan.
Visual chatgpt: Talking, drawing and editing with visual foundation
models.
arXiv preprint arXiv:2303.04671, 2023.
- (84)
Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He.
Less-to-more generalization: Unlocking more controllability by
in-context generation.
arXiv preprint arXiv:2504.02160, 2025.
- (85)
Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu,
Bei Yu, and Jiaya Jia.
Dreamomni: Unified image generation and editing.
arXiv preprint arXiv:2412.17098, 2024.
- (86)
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan,
Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu.
Omnigen: Unified image generation.
arXiv preprint arXiv:2409.11340, 2024.
- (87)
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang.
Smartbrush: Text and shape guided object inpainting with diffusion
model.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 22428–22437, 2023.
- (88)
Sihan Xu, Ziqiao Ma, Yidong Huang, Honglak Lee, and Joyce Chai.
Cyclenet: Rethinking cycle consistency in text-guided diffusion for
image manipulation.
Advances in Neural Information Processing Systems,
36:10359–10384, 2023.
- (89)
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong
Chen, and Fang Wen.
Paint by example: Exemplar-based image editing with diffusion models.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 18381–18391, 2023.
- (90)
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui.
Mastering text-to-image diffusion: Recaptioning, planning, and
generating with multimodal llms.
arXiv preprint arXiv:2401.11708, 2024.
- (91)
Ling Yang, Bohan Zeng, Jiaming Liu, Hong Li, Minghao Xu, Wentao Zhang, and
Shuicheng Yan.
Editworld: Simulating world dynamics for instruction-following image
editing.
arXiv preprint arXiv:2405.14785, 2024.
- (92)
Shiyuan Yang, Xiaodong Chen, and Jing Liao.
Uni-paint: A unified framework for multimodal image inpainting with
pretrained diffusion model.
In Proceedings of the 31st ACM International Conference on
Multimedia, pages 3190–3199, 2023.
- (93)
Zhen Yang, Ganggui Ding, Wen Wang, Hao Chen, Bohan Zhuang, and Chunhua Shen.
Object-aware inversion and reassembly for image editing.
arXiv preprint arXiv:2310.12149, 2023.
- (94)
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu,
Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al.
Cogvideox: Text-to-video diffusion models with an expert transformer.
arXiv preprint arXiv:2408.06072, 2024.
- (95)
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn,
David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al.
Language model beats diffusion–tokenizer is key to visual
generation.
arXiv preprint arXiv:2310.05737, 2023.
- (96)
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su.
Magicbrush: A manually annotated dataset for instruction-guided image
editing.
Advances in Neural Information Processing Systems,
36:31428–31449, 2023.
- (97)
Shiwen Zhang, Shuai Xiao, and Weilin Huang.
Forgedit: Text guided image editing via learning and forgetting.
arXiv preprint arXiv:2309.10556, 2023.
- (98)
Yabo Zhang, Xinpeng Zhou, Yihan Zeng, Hang Xu, Hui Li, and Wangmeng Zuo.
Framepainter: Endowing interactive image editing with video diffusion
priors.
arXiv preprint arXiv:2501.08225, 2025.
- (99)
Zhongping Zhang, Jian Zheng, Zhiyuan Fang, and Bryan A Plummer.
Text-to-image editing by image information removal.
In Proceedings of the IEEE/CVF winter conference on applications
of computer vision, pages 5232–5241, 2024.
- (100)
Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An,
Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang.
Ultraedit: Instruction-based fine-grained image editing at scale.
Advances in Neural Information Processing Systems,
37:3058–3093, 2024.
- (101)
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex
Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.
Multimodal c4: An open, billion-scale corpus of images interleaved
with text.
Advances in Neural Information Processing Systems,
36:8958–8974, 2023.
- (102)
Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen.
A task is worth one word: Learning with task prompts for high-quality
versatile image inpainting.
In European Conference on Computer Vision, pages 195–211.
Springer, 2024.
- (103)
Siyu Zou, Jiji Tang, Yiyi Zhou, Jing He, Chaoyi Zhao, Rongsheng Zhang, Zhipeng
Hu, and Xiaoshuai Sun.
Towards efficient diffusion-based image editing with instant
attention masks.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 38, pages 7864–7872, 2024.

## Appendix A Implementation Details

### A.1 Data Details

The training videos are sourced from a wide spectrum of domains, including stock footage, films, documentaries, etc. We split the raw videos into both single-shot clips and multi-shot scene videos.
We also pre-process the raw videos by using different filtering strategies to keep high-quality videos, including logo detection, black border detection, and aesthetic estimation.

Figure: Figure 9: Two ways of frame sampling: (a) equal-interval sampling and (b) fixed-frame sampling.
Refer to caption: /html/2506.10941/assets/x9.png

As described in Sec.[3.1](#S3.SS1), we adopt two frame sampling strategies: equal-interval sampling and fixed-frame sampling. As illustrated in Fig.[9](#A1.F9), these approaches jointly ensure both the diversity and temporal stability of visual dynamics—two key factors for effective training of in-context image editing models.

### A.2 Visual Transition Annotation

Figure: Figure 10: Examples (1/2) of visual transition annotation performed by our in-house large multimodal model.
Refer to caption: /html/2506.10941/assets/x10.png

Figure: Figure 11: Examples (2/2) of visual transition annotation performed by our in-house large multimodal model.
Refer to caption: /html/2506.10941/assets/x11.png

To bridge the semantic gap between two sampled frames, we use our in-house LMM to annotate visual transitions, as introduced in Sec.[3.1](#S3.SS1). The instruction used during annotation is shown above, and Fig. [11](#A1.F11) presents example annotations to illustrate their quality.

### A.3 Segmentation Mask Annotation and RoE Construction

The proposed visual transition annotation framework leverages an LMM to generate multi-level annotations, ranging from local concepts to global scene descriptions. As illustrated in Fig.[2](#S3.F2), we first use character and object descriptions from the source and target frames as query inputs to GroundingDINO[44] to obtain object detection results. These detections are then passed to SAM 2 [62] to extract segmentation masks for the corresponding local concepts. Guided by the annotated local changes, we identify and fuse the objects or characters undergoing transitions to construct the final RoEs.

### A.4 Model Architecture

Variational Autoencoder.
Following prior work [95], we adopt the encoder in a pretrained VAE to embed each image into the latent space separately for efficient computation. Specifically, it compress raw pixels with shape $(H,W,3)$ into a $(h,w,c)$-shape latent representation, with downsampling ratios as $d_{h}=\frac{H}{h}$ and $d_{w}=\frac{W}{w}$ for height and width, respectively, and the latent channel $c$. The decoder in VAE aims to transform latent representations generated by the DiT back into the pixel space during inference.

Text Encoder.
We employ the pretrained Flan-T5 as the text encoder to separately encode the prompt in each turn, and then concatenate all the embedding with inserting turn embeddings in between. Specifically, to make the model better discriminate different turns, we define a special turn token <TURN>i for the $i$-th turn, and introduce a learnable turn embedding for each one, which is inserted before the prompt embedding in the $i$-th turn.

### A.5 Details of MSE-Bench

Figure: Figure 12: Multi-turn image editing examples of MSE-Bench.
Refer to caption: /html/2506.10941/assets/x12.png

The source images for our constructed multi-turn image editing benchmark, MSE-Bench, are sampled from MS-COCO [40] and LAION-Aesthetics [66]. Specifically, we randomly sample 6,000 images from each dataset and employ GPT-4o to perform prompt imagination, guided by criteria such as editing reasonability, aesthetics, consistency, and coherence.
To facilitate this, we define a set of editing operations (e.g., add, remove, replace) and design a series of rules to instruct GPT-4o to simulate realistic and coherent multi-turn editing prompts from real users’ perspectives. The instruction used in this process is illustrated above.
Following prompt generation, we conduct careful human filtering to remove low-quality cases, resulting in a final set of 100 high-quality, category-balanced examples that constitute MSE-Bench. Additional examples are shown in Fig.[12](#A1.F12).

## Appendix B Additional Experimental Results

### B.1 Human Evaluation on Multi-turn Image Editing

**Table 6: Human evaluation on MSE-Bench based on editing success rate. * indicates no use of context. Entries by gray denote proprietary models.**
| Method | Human Evaluation |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Turn-1 | Turn-2 | Turn-3 | Turn-4 | Turn-5 |  |
| HQEdit* [29] | 0.170 | 0.073 | 0.020 | 0.003 | 0.000 |
| UltraEdit* [100] | 0.310 | 0.062 | 0.015 | 0.002 | 0.000 |
| OmniGen* [86] | 0.333 | 0.035 | 0.002 | 0.000 | 0.000 |
| GPT-4o | 0.872 | 0.783 | 0.755 | 0.642 | 0.491 |
| Ours | 0.661 | 0.500 | 0.323 | 0.209 | 0.070 |

To further verify the effectiveness and superiority of the proposed method for multi-turn image editing, we conduct human evaluations to assess editing success rates. The results are reported in Tab. [6](#A2.T6).
These findings validate the benefits of training on native video data, combined with supervised fine-tuning on pairwise editing examples, in enhancing multi-turn editing performance.

### B.2 Correlation Between GPT-4o and Human Evaluation

**Table 7: Correlation between automatic metrics and human evaluation**
| Metric | GPT-4o vs Human | CLIP-T vs Human | CLIP-I vs Human |
| --- | --- | --- | --- |
| Pearson $r$ | 0.4858 ($p=0.0000$) | 0.0817 ($p=0.4191$) | -0.0549 ($p=0.5875$) |
| Spearman $\rho$ | 0.4644 ($p=0.0000$) | 0.0692 ($p=0.4941$) | -0.0217 ($p=0.8303$) |
| Kendall $\tau$ | 0.4154 ($p=0.0000$) | 0.0502 ($p=0.4963$) | -0.0195 ($p=0.7921$) |

In our experiments (Sec.[4](#S4)), we primarily report GPT-4o evaluated success rates to assess multi-turn image editing performance. To validate the reliability of GPT-4o-based evaluation, we compute the correlation between GPT-4o scores and human judgments. As shown in Tab.[7](#A2.T7), we also compare other metrics such as CLIP-T and CLIP-I. The results demonstrate that GPT-4o correlates well with human evaluation, supporting its use as a reliable proxy for scoring multi-turn image editing.

## Appendix C Additional Application Examples

### C.1 Multi-turn Image Editing

Figure: Figure 13: Qualitative comparison (1/X) between our method (w/ SFT on OmniEdit [80]) and recent baselines (HQ-Edit [29], UltraEdit [100], OmniGen [86], and GPT-4o [53]) on MSE-Bench.
Refer to caption: /html/2506.10941/assets/x13.png

Figure: Figure 14: Qualitative comparison (2/X) between our method (w/ SFT on OmniEdit [80]) and recent baselines (HQ-Edit [29], UltraEdit [100], OmniGen [86], and GPT-4o [53]) on MSE-Bench.
Refer to caption: /html/2506.10941/assets/x14.png

Figure: Figure 15: Qualitative comparison (3/X) between our method (w/ SFT on OmniEdit [80]) and recent baselines (HQ-Edit [29], UltraEdit [100], OmniGen [86], and GPT-4o [53]) on MSE-Bench.
Refer to caption: /html/2506.10941/assets/x15.png

Figure: Figure 16: Qualitative comparison (4/X) between our method (w/ SFT on OmniEdit [80]) and recent baselines (HQ-Edit [29], UltraEdit [100], OmniGen [86], and GPT-4o [53]) on MSE-Bench.
Refer to caption: /html/2506.10941/assets/x16.png

As shown in Fig. [13](#A3.F13), we compare our method with several baselines, including HQ-Edit [29], UltraEdit [100], OmniGen [86], and GPT-4o. The results reveal several key observations:
1) Most existing models suffer from error accumulation, leading to increasingly severe artifacts across editing turns.
2) These accumulated errors often degrade prompt-following performance, where the model fails to execute edits as instructed once artifacts dominate.
3) While GPT-4o—a strong proprietary model—achieves competitive results, it may exhibit inconsistencies in some cases compared to our method.
4) Overall, these comparisons highlight the effectiveness of training on native video data for achieving coherent and prompt-aligned multi-turn image editing.

Additional qualitative examples are provided in Fig. [22](#A3.F22), Fig. [23](#A3.F23), Fig. [24](#A3.F24), and Fig. [25](#A3.F25), further demonstrating the strong prompt-following and consistency of our approach across multiple editing turns.

### C.2 Multi-concept composition

Figure: Figure 17: More zero-shot qualitative results of multi-concept composition achieved by our method.
Refer to caption: /html/2506.10941/assets/x17.png

In Fig. [17](#A3.F17), we present qualitative results on multi-concept composition, which requires both composition and strong identity preservation. These examples demonstrate that training on video data can effectively unlock compositional capabilities, despite the rarity of such patterns in typical video content. This emergent behavior highlights the potential of video-based pre-training. Further scaling of model capacity, compute resources, and video data may enable the emergence of even more advanced capabilities.

### C.3 Story Generation

Figure: Figure 18: More qualitative results of story generation achieved by our method.
Refer to caption: /html/2506.10941/assets/x18.png

Since our method is trained on native video data, it inherently captures the underlying storylines present in the sequences. As illustrated in Fig.[18](#A3.F18), we formulate story generation as a multi-turn image editing task, guided by transition prompts between key frames during inference. These examples showcase the model’s ability to follow prompts while maintaining coherence and consistency across turns. When combined with existing long video generation methods[22], our approach has the potential to enhance top-down planning for generating coherent long-form story videos.

### C.4 Chain-of-Editing

Figure: Figure 19: More qualitative results of Chain-of-Editing.
Refer to caption: /html/2506.10941/assets/x19.png

Figure: Figure 20: Qualitative comparison between w/o Chain-of-Editing (CoE) and w/ CoE.
Refer to caption: /html/2506.10941/assets/x20.png

In Tab. [4](#S4.T4), we show the effectiveness of chain-of-editing, *i.e.*, predicting segmentation maps before performing image editing. The predicted segmentation maps could be viewed as a kind of “thoughts”. In Fig. [19](#A3.F19), we show more qualitative results for challenging cases to demonstrate the effectiveness of CoE.

### C.5 Drag-based Image Editing

Figure: Figure 21: Qualitative results of drag-based image editing.
Refer to caption: /html/2506.10941/assets/x21.png

The current and next segmentation prediction tasks introduced in Sec.[3.3](#S3.SS3) not only support progressive planning and generation, but also enable controllable editing for enhanced user interaction.
One such application is drag-based image editing for object displacement, scaling, and rotation, as illustrated in Fig.[21](#A3.F21). In this setting, users first provide an editing prompt to localize the RoE. Then, drag operations are applied to perform geometric transformations of the RoE. The transformed segmentation map driven by the transformation is incorporated into the context, allowing the model to generate a target image that adheres to the specified edits.

Figure: Figure 22: More qualitative results (1/4) of multi-turn image editing achieved by our method.
Refer to caption: /html/2506.10941/assets/x22.png

Figure: Figure 23: More qualitative results (2/4) of multi-turn image editing achieved by our method.
Refer to caption: /html/2506.10941/assets/x23.png

Figure: Figure 24: More qualitative results (3/4) of multi-turn image editing achieved by our method.
Refer to caption: /html/2506.10941/assets/x24.png

Figure: Figure 25: More qualitative results (4/4) of multi-turn image editing achieved by our method.
Refer to caption: /html/2506.10941/assets/x25.png

## Appendix D Future Work

In the future, we aim to solve more challenging image creation tasks [60, 90, 59] with complex and compositional prompts, by exploring multimodal chain-of-thought. Besides, post-training [57, 20] would stimulate more potential interesting abilities endowed by learning from videos. Finally, by introducing retrieved images [56, 10, 58] into context, our model could achieve knowledge-intensive visual creation scenarios via retrieval-augmented generation.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Natural videos excel at capturing state transitions like object addition/removal, pose changes, and motion. However, they rarely, contain examples of more "creative" or "unrealistic" edits that are common user requests. For example, drastic environmental changes (e.g., "change the weather from snow to summer"), complex style transfers (e.g., "make this look like a Picasso painting"), or material transformations (e.g., "make the person out of glass"). It is questionable whether a model trained only on natural video can generalize to these common editing tasks. I do agree that video data could be used in the pretraining, but using video data for image editing is not a novel concept. The paper's own related work section cites several recent methods that also "Learn from Video for Image Generation." The paper differentiates itself by using "long, interleaved image-text context" rather than just frame pairs, but this feels more like a strong incremental improvement than a paradigm shift.
soundness: 3
strengths: 1. The central idea of using native video data to learn in-context editing makes sense. It cleverly reframes the problem, identifying videos as a natural, abundant, and scalable source of "edit" data (i.e., state transitions) that intrinsically contains the visual consistency and temporal context missing from static image-pair datasets. 

2. The data construction pipeline is sound, combining a VLM for high-level semantic transition annotation with grounding models (GroundingDINO + SAM2) for precise, low-level spatial localization (RoEs). The three proxy tasks (NIP, CSP, NSP) are well-motivated; using segmentation prediction not only as a training signal (CSP) but also as a predictive target (NSP) is a smart way to improve grounding and controllable generation.

3. The paper is well-written, and the methodology is presented clearly. The figures are effective at illustrating the data pipeline and model architecture.
summary: This paper introduces VINCIE, a framework for in-context image editing using video data. The core premise is to learn this capability solely from native video data, bypassing the need for curated, paired image-editing datasets. The authors propose a scalable data pipeline that samples video frames and uses a Vision-Language Model to generate textual annotations of the visual transitions between them. It also generates segmentation masks for these "Regions of Editing" (RoEs). The model, a diffusion transformer, is trained using three proxy tasks: Next-Image Prediction (NIP), Current Segmentation Prediction (CSP), and Next-Segmentation Prediction (NSP). To evaluate their method, the authors introduce a new multi-turn benchmark, MSE-Bench, which consists of 5-turn editing sessions and is evaluated using GPT-4o. The results demonstrate that VINCIE achieves strong performance, outperforming many academic methods on MagicBrush and their new benchmark, and that the approach scales well with data.
weaknesses: 1. The paper defines in-context editing as modifying an image based on a "contextual sequence comprising text and previously generated images." This framing is functionally equivalent to multi-turn editing, where the primary role of the context is to serve as the input for the next step. This definition is quite narrow and overlooks a more common interpretation of "in-context" for image models: the ability to use one or more reference images in the context to provide new subjects, styles, or concepts for the current edit (e.g., "add the dog from the second image with the style from the 4th image into this one"). The paper's experiments all fit the multi-turn definition, and it's unclear if VINCIE can perform this more complex, composition-focused "reference-based" in-context editing.

2. The evaluation, while thorough on MagicBrush and the proposed MSE-Bench, feels incomplete. It would be much more convincing to see a comparison on more recent benchmarks, such as GEdit-Bench and also evaluate on the EMUEdit benchmark? 

3. The primary evaluation for the newly proposed MSE-Bench relies on a "success rate" computed by GPT-4o. This is a significant concern. The reliability of current VLMs for nuanced, multi-step comparative judgment is questionable. The appendix suggests multiple images are fed simultaneously, which could easily confuse the model. My own experience suggests GPT-4o can struggle to correctly identify subtle edits or preserve consistency even in simple source-vs-target comparisons, let alone a complex 5-turn sequence where errors propagate. This evaluation method needs much stronger validation.
Also, for automatic evaluation, it would be beneficial to also report scores like the PF (Prompt following) and SC (semantic consistency) from recent work like VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. These are designed for image editing and might offer a more reliable supplement to the GPT-4o-based success rate.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: see the Weaknesses.  Overall, I would not object to this paper being accepted.
soundness: 3
strengths: 1. Well-structured with clear figures (Figures 1-3 effectively convey main ideas)
2. Comprehensive appendix with implementation details
3. Three proxy tasks (NIP, CSP, NSP) provide complementary learning signals
4. Comprehensive ablation studies validate design decisions
summary: This paper proposes VINCIE, a method for learning in-context image editing directly from video data. The key innovation is constructing training data by: (1) sampling frames from videos, (2) using VLMs to annotate visual transitions between frames, (3) generating segmentation masks for regions of interest using GroundingDINO and SAM2, and (4) training a diffusion transformer with three proxy tasks (next-image prediction, current segmentation prediction, next segmentation prediction). The authors introduce MSE-Bench, a challenging 5-turn editing benchmark, and demonstrate competitive performance against existing methods while showing scalability with increasing data.
weaknesses: 1. The claim of learning from "native videos" is misleading—the method requires extensive preprocessing with VLMs, GroundingDINO, and SAM2. This is annotation-based learning, not purely native video learning
2. Evaluation relies heavily on GPT-4o as judge, which may introduce bias despite correlation analysis (Table 7)
3. Core technical novelty is limited: DiT architecture, segmentation prediction, and in-context learning are established techniques. The main contribution is the data construction pipeline
4. Heavy reliance on existing models (VLM, GroundingDINO, SAM2) reduces the "learning from video" novelty
5. Fair comparison at same dataset?

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: Q1) In Visual Transition Annotation, authors leveraged VLM for captioning. While it makes sense to me to use VLM for generating detailed and coherent descriptions of each frame from multiple aspects, I am concerned of using VLMs to identify semantic and visual differences
between the two frames when the two frames have minor differences. It is known that VLMs often fails in telling subtle difference between two images, as pointed out in The Dawn of LMMs and VIEScore. I question the robustness of this approach for fine-grained visual transition annotation, as critical details might be overlooked or misrepresented when using these models. Could the authors provide a case study or qualitative results focusing on more fine-grained edits (e.g. moving an object slightly to one direction, like only 1cm)?

Q2) Regarding Regions-of-Editing, GroundingDINO and SAM2 may propagate errors in mask generation or transition description, especially in complex or cluttered scenes. Any measures are conducted to minimize the error impact in this pipeline?
soundness: 3
strengths: S1) This paper proposed a new perspective in constructing session-wise data with long, interleaved image-text context from native videos while prior works that used video for editing were mainly for constructing pair-wise data. This highlights the originality.

S2) At first glance the designed approach is best suited for videos with a static viewpoint. When the camera moves, the region proposal and segmentation process can break down, leading to inaccurate edits or mask predictions. Large background or subject position shifts caused by camera movement are not explicitly addressed. But it is clever approach that predicting segmentation masks before editing could mitigates subject drift.

S3) The paper is well written, with good motivation, and communicated clearly. Overall I enjoyed reading this work.
summary: VINCIE introduces a novel approach to multi-turn image editing by directly leveraging video data. It treats sequential video frames and their natural transitions as in-context editing demonstrations, and designed three proxy tasks: next-image predic-
047 tion, current segmentation prediction, and next-segmentation prediction to tickle this problem. Using a diffusion transformer architecture guided by predicted segmentation masks, VINCIE learns to localize and apply edits step-by-step in a context-aware manner without needing manually curated editing datasets. The method shows strong performance on new and established multi-turn editing benchmarks.
weaknesses: W1) I find it surprising that the paper identifies MagicBrush (NeurIPS 2023) as the last major benchmark for multi-turn image editing, with no subsequent progress. Based on my recollection, ImgEdit-Bench (May 2025) also introduced evaluation protocols on multi-turn image editing. Including results on the ImgEdit-Bench benchmark would make the empirical comparison much more convincing.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: See weaknesses above
soundness: 3
strengths: 1. The proposed pipeline of training multi-turn image editing models from video data is novel and intuitively effective.
2. The proposed MSE-Bench can benefit the image editing community and more properly evaluate multi-turn image editing models.
3. VINCIE achieved strong multi-turn image editing performance compared to previous art.
summary: The paper proposes VINCIE, a video-driven approach to in-context image editing that learns multi-turn image editing directly from videos rather than paired images. In the method, videos are converted into interleaved multimodal sequences of sparsely sampled frames, VLM-written transition texts, and segmentation masks for regions of editing. The method leverages a DiT with block-wise causal attention trained on three proxy tasks: next-image prediction, current-segmentation prediction, and next-segmentation prediction. The authors also introduce MSE-Bench, a 5-turn editing benchmark, and report strong results on MSE-Bench and MagicBrush, with additional analyses on context, segmentation prediction, and data scaling.
weaknesses: 1. What are the training objectives for the segmentation prediction tasks NSP and CSP? Are they MSE or some other losses between the predicted and GT segmentation masks?
2. Can the author provide some intuition on why adding the next segmentation prediction task during inference benefits MagicBruch scores but damages MSE-Bench scores (compared to CS->I)?
3. From Table 4, it seems that the Dummy context is more effective than providing the editing history to the model. Is this the correct interpretation of this table? Does that mean the Dummy context itself is enough for multi-turn image editing?
4. Based on Section 4.4, the camera movement and subject position shift problems can be solved by incorporating the segmentation prediction task during inference. Can the author elaborate more on why this prediction task helps stabilize the movement of the subjects in the image? Is it because the predicted segmentation mask always tends to be the same mask as in the previous editing round?
5. The paper mentioned both full attention and block-wise causal attention in the method section. Which one is the method that is used in the final version of VINCIE? From Section 1, it seems that block-wise causal attention is actually used in VINCIE. If this is the case, what is the point of mentioning the full attention design? Or does the model support both attention mechanisms (across different editing turns) natively?
6. I would suggest adding some description on how the model is used during inference, especially on the different modes that the model can be used, since the model accepts a wide variety of input conditions and these inputs can be quite flexible. Right now it is a bit confusing on how the model operates during inference: for example, when adding the segmentation mask prediction task during inference, will the predicted segmentation mask be used as part of the input condition to the model, or will it only be predicted, but not used during the next edited image prediction?

Overall, I find the method and results to be promising, but I believe the current paper's presentation can be improved. I would be happy to raise my score if the concerns can be addressed.


Decision: Accept (Poster)