DreamOmni2: Multimodal Instruction-based Editing and Generation
Authors: name: Bin Xia, name: Bohao Peng, name: Yuechen Zhang, name: Junjia Huang, name: Jiyang Liu, name: Jingyao Li, name: Haoru Tan, name: Sitong Wu, name: Chengyao Wang, name: Yitong Wang, name: Xinglong Wu, name: Bei Yu, name: Jiaya Jia

Recent advancements in instruction-based image editing and subject-driven
generation have garnered significant attention, yet both tasks still face
limitations in meeting practical user needs. Instruction-based editing relies
solely on language instructions, which often fail to capture specific editing
details, making reference images necessary. Meanwhile, subject-driven
generation is limited to combining concrete objects or people, overlooking
broader, abstract concepts. To address these challenges, we propose two novel
tasks: multimodal instruction-based editing and generation. These tasks support
both text and image instructions and extend the scope to include both concrete
and abstract concepts, greatly enhancing their practical applications. We
introduce DreamOmni2, tackling two primary challenges: data creation and model
framework design. Our data synthesis pipeline consists of three steps: (1)
using a feature mixing method to create extraction data for both abstract and
concrete concepts, (2) generating multimodal instruction-based editing training
data using the editing and extraction models, and (3) further applying the
extraction model to create training data for multimodal instruction-based
editing. For the framework, to handle multi-image input, we propose an index
encoding and position encoding shift scheme, which helps the model distinguish
images and avoid pixel confusion. Additionally, we introduce joint training
with the VLM and our generation/editing model to better process complex
instructions. In addition, we have proposed comprehensive benchmarks for these
two new tasks to drive their development. Experiments show that DreamOmni2 has
achieved impressive results. Models and codes will be released.