Title: Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Instruction-based Image Edit, Multimodal Large Language Model, Reinforcement Learning

Abstract: Instruction-based image editing has garnered significant attention due to its direct interaction with users. However, real-world user instructions are immensely diverse, and existing methods often fail to generalize effectively to instructions outside their training domain, limiting their practical application. To address this, we propose Lego-Edit, which leverages the generalization capability of Multi-modal Large Language Model (MLLM) to organize a suite of model-level editing tools to tackle this challenge. Lego-Edit incorporates two key designs: (1) a model-level toolkit comprising diverse models efficiently trained on limited data and several image manipulation functions, enabling fine-grained composition of editing actions by the MLLM; and (2) a three-stage progressive reinforcement learning approach that uses feedback on unannotated, open-domain instructions to train the MLLM, equipping it with generalized reasoning capabilities for handling real-world instructions. Experiments demonstrate that Lego-Edit achieves state-of-the-art performance on GEdit-Bench and ImgBench. It exhibits robust reasoning capabilities for open-domain instructions and can utilize newly introduced editing tools without additional fine-tuning.

TL;DR: Our Lego-Edit leverages a Multimodal Large Language Model to organize a set of model-level editing tools and trains it via progressive reinforcement learning for generalized instruction-based image editing.