Title: Disentangling Instruction Influence in Diffusion Transformers for Parallel Multi-Instruction-Guided Image Editing
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Image Editing, Multi-Instruction Guided Image Editing

Abstract: Instruction-guided image editing enables users to specify modifications by natural language, offering more flexibility and control. Among existing frameworks, Diffusion Transformers (DiTs) outperform U-Net-based diffusion models in scalability and performance. However, while real-world scenarios often require concurrent execution of multiple instructions, step-by-step editing suffers from accumulated errors and degraded quality, and integrating various instructions with a single prompt usually results in incomplete edits. We propose an Instruction Influence Disentanglement (IID) framework that enables the parallel execution of multiple instructions within a single denoising process for DiT-based models. By analyzing self-attention mechanisms in DiTs, we identify distinct attention patterns in multi-instruction settings and derive instruction-specific masks to disentangle the influence of each instruction. These masks then guide the editing process to ensure localized modifications while preserving consistency in non-edited regions. Extensive experiments demonstrate that IID can enhance fidelity and instruction completion, while also reducing computation overhead compared to existing methods.

TL;DR: IID enables parallel multi-instruction image editing in Diffusion Transformers using attention masks to disentangle influences, ensuring better quality, consistency, and efficiency than existing methods