Guiding Instruction-based Image Editing via Multimodal Large Language Models
Authors: Tsu-Jui Fu
Wenze Hu
Xianzhi Du
William Yang Wang
Yinfei Yang
Zhe Gan
Venue: ICLR 2024 spotlight (2024)
Keywords: image editing
multimodal large language model
Primary area: generative models
TL;DR: MLLM-Guided image editing (MGIE) jointly learns the MLLM and editing model with visual-aware expressive instructions to provide explicit guidance for instruction-based image editing.

Abstract: Instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instructions and provides explicit guidance. The editing model jointly captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.

Published as a conference paper at ICLR 2024
GUIDING INSTRUCTION-BASED IMAGE EDITING VIA MULTIMODAL LARGE LANGUAGE MODELS
Tsu-Jui Fu', Wenze Hu², Xianzhi Du², William Yang Wang', Yinfei Yang", Zhe Gan? 1 UC Santa Barbara, AApple
Commonsense Reasoning
Photoshop Modification
remove the woman the background
make it more healthy
The pizza includes vegetable toppings, such as tomatoes and herbs
The focus of the Image would shift the man's expression
Global
add contrast simulate more light
Local Editing
let the laptop have green web page
Optimization
The light enhances the detail of the mother cheetah and her cubs rock hillside
The green web page is related project. The other scheme remains
Input Image
MGIE (Expressive Instruction)
Input Image
MGIE (Expressive Instruction)
Figure 1: We introduce MILLM-Guided Image Editing (MGIE) to improve instruction-basse image
editing for various editing aspects. The top is the input instruction, and the right the jointly derived expressive instruction by MGIE.
ABSTRACT
Instruction-basee image editing improves the controllability and flexibility of im- age manipulation via natural commands without claborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promis- ing capabilities in cross-modke understanding and visual-aware response genera- tion via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instruc- tions and provides explicit guidance. The editing model jodel captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instruc- tions crucial to instruction- based image editing, and our MGIE lead to notable improvement in automatic metrics and human evaluation while maintain- ing competitive inference efficiency.
INTRODUCTION
Visual design tools are widely adopted in various multimedia fields nowadays. Despite considerable demand, they require prior knowledge to operate. To enhance controllability and accessibility, text- guided image editing has obtained popularity in recent studies eLi 2020 Patashnik al. al,] [02T]] Crowson et al. 2022 Gal et al. 2022). With attractive ability model realistic images, diffusion models (Ho et al. 2020) are also adopted image editing (Kim et al. 2022). By swapping the latent cross-modal maps, models perform visual manipulation to reflect the alteration of the input-goal
Work done during internship Apple. Project wehsite: https: / /m1 1m- ie. github. i

---

Published as a conference paper at ICLR 2024
caption (Hertz et al., 2023 Mokady et al. 2022]] Kawar et al.,
2023
They can further edit specific
region by a guided mask (Nchhol et al. 2022; Avrahami et al. 2022
Instead of relying on elaborate
editing (El-Nouby et al. 2019 Li et al. 2020 Fu et al. 2020 allows human commands that directly express how and which aspect of an image to edit.
descriptions regional masks, instruction-
This dexxibiiity also benefits practicality such guidance is more aligned with human intuition.
Due to the data scarcity of the input-goallimstructinn triplet, InsPix2Pix (Brooks et al. 2023) collects curated IPr2Pr dataset. The instruction is generated by GPT-3 (Brown al. 2020p. and the input-
goal image pair is synthesized from Prompt-to- Proproppttttto (Hertz et al. (023). InsPix2Pix then applies a pre-trained CLIP text encoder (Radford et al. 2021 to lead the diftusion model along with the input image. Although having feasible results, CLIP is trained for static descriptions. which is challenging to capture the essential visual transformation in editing. Furthermore, the instruction is too brief but ambiguous and insufficient to guide toward the intended goal. The deficiency limits the effectiveness of InsPix2Pix in instruction- based image editing.
Large language models (LLMs) (Brown et al. 2020 Touvron et al. 2023) have shown significant ad- vancement diverse language tasks, including machine translation, text summarization, and ques- tion answering. Learning from large-scale corpora with diverse content, LLMs contain latent visual knowledge and creativity, which can assist various vision-and aane-and-lnngggge tasks (Wu et al., 2023 Feng et al. 2023 Chakrabarty et al. 2023). Upon LLMs, multimodal large language models (MLLMs) can treat images as input naturally and provide visual-aware responses to serve as multimodal assis-
tants Zhang al. 2023b
Liu
2023
Zhu et al. 2023 Koh al. 2023)
Inspired by MLLMs, we incorporate them to deal with the insufficient guidance issue of instructions and introduce MLLM-Guided Image Editing (MGIE). As demonstrated in Fig. MGIE consists of an MLLM and a diffusion model. The MLLM learns to derive concise expressive instructions and offers explicit visual-related guidance. The diffusion model is jointly updated and performs image editing with the latent imagination of the intended goal via end-to-end training. In this way, MGIE benefits from the inherent visual derivation and addresses ambiguous human commands achieve it is difficult to capture what "healthy" means without additional context. Our MGIE can precisely connect "vegetable toppings" with the pizza and lead to the related editing as human expectation.
reasonable editing. For the example in Fig. [
To learn instruction-based image editing, we adopt IPr2Pr as our pre-training dataset. We consider different editing aspects in EVR (Tan et al. 2019), GIER (Shi et al. 2020). MA5k (Shi et al. 2022). and MagicBrush (Zhang et al. (023a)). MGIE performs Photoshop-style modification, global photo optimization, and local object alteration. All should be guided by human instructions. Experimental results indicate that our MGIE significantly strengthens instruction-base image editing with reason- able expressive instructions in automatic metrics and human evaluation, and visual-aware guidance is crucial to this improvement. In summary, our contributions are three-fold:
We introduce MLLM-Guideds Image Editing (MGIE), which jointly learns the MLLM and editing model with visual-aware expressive instructions to provide explicit guidance.
We conduct comprehensive studies from various editing aspects, including Photoshop-style mod- ification, global photo optimization, and local editing, along with qualitative comparisons. Extensive experiments demonstrate that visual-award expressive instructions are crucial for image editing, and our MGIE effectively enhances editing performance.
RELATED WORK
Instruction-basse Image Editing. Text-guided image editing can significantly improve the con- trollability and accessibility of visual manipulation by following human commands. Previous works built upon the GAN frameworks (Goodfellow et al. [015] Reed et al. 2016) to alter images but are limited to unrealistic synthesis or specific domains (Nam et al. 2018 Li et al. 2020 Eo-No et al.
2019
Fu et al. 2020 2022). With promising large-scale training, diffusion models (Ho et al. 2020
Ramesh et al. 2022 Sahari et al. 2022 Rombach et al. 2022) can accomplish image transformation
via controlling the cross cross-modal attention maps for the global caption (Meng et al.) 2022 Hertz et al.
2023 Kawar et al., 2023 Gu al. 2023). Local image editing allows fine- tineggraiedd manipulation by
inpainting target regions with user- seer-provided (Nichol et al.) 2022) Avrahami et al., 2022; Wang et al.)
20236 or predicted masks (Bar-Tal et al. 2022 Couairon al. 2023) while preserving the remain-
2

---

Published as a conference paper at ICLR 2024
Expressive Instruction
The cabin in the desert is surrounded by sand dunes and some vegetation, such as cacti or small shrubs
The cabin in the is
Visual Token
Lins
LM Head
Summ
Multimodal Large Language Model (MLLM)
The sandy terrain and sparse vegetation would
Edit Head
Embed
Adapter
<BOS> The cabin the ... [IMG.] [IMG.]
stark contrast to lush forests... absence the surrounding
trees lage would
have them
affect the microclimate...
be in desert
Diffusion
MLLM*
Ledit
Input Instruction
Instruction
Image
Input Image
Goal Image
Figure 2: Overview of MLLM-Guided Image Editing (MGIE), which leverages MLLMs to enhance
instruction- basod image editing. MGIE learns to derive concise expressive instructions and provides explicit risual-related guidance for the intended goal. The diffusion model jointly trains and achieves image editing with the latent imagination through the edit head in an end-to-end manner. and show the module is trainable and frozer respectively.
ing areas. Different from them, instruction- image editing accepts straight commands, such as "add fireworks to the sky", which is not restricted elaborate descriptions regional masks. Recent methods learn from synthetic input-goal-instructin triples (Brooks et al. 2023) and with additional human feedback (Zhagg al. 2023c) follow editing instructions. However, the frozen CLIP text encoder is pre- pre-trained for static descriptions but not the crucial transformation in editing. Moreover, the instructions are sometimes ambiguous and imprecise for the editing goal. In this paper, we learn with multimodal large language models to perceive images along with given prompts for expressive instructions, which provides explicit yet detailed guidance, leading to superior editing performance.
Large Language Models for Vision. Large language models (LLMs) have demonstrated impres- sive capabilities for text generation and generalizability in various tasks (Brown et al. 2020 Chowd-
hery et al.
2022: Touvron et al. 2023). With robust text understanding, previous works adapt LLMs
for input prompts and reason downstream lisoonaand-language tasks (Zhang et al. 20247 Wu al.
2023
Lu et al.
2023 Yang et al. 2023; Chakrabarty et al. (2023). They further produce pseudocode
instructions executable programs by LLMs (Huang et al. 2022] Gupta & Kembhavil 2023 Surís
et al. 2023 Feng et al. 2023 Lian et al. 2023). Through visual feature alignment with instruction
tuning, multimodal large language models (MLLMs) can perceive images and provide adequate re-
sponses (Li et al.) 2023b Zhang et al., 20236 Liu et al. 2023 Zhu et al. 2023). Recently, studies
also adopt MI.LMss for generating chat-related images (Koh et al.) 2023 Sun et al. 2023). However, they can only produce images from scratch, which are distinct from inputs. Our proposed MGIE is
the first to leverage MLLMs and improve image editing with derived expressive instructions.
METHOD
3.1 BACKGROUND: MULTIMODAL LARGE LANGUAGE MODELS (MLLMs)
Large language models (LLMs) have shown impressive capabilities for natural language generation. Multimodal large language models (MLLMs) empower L.MMs to perceive images and provide rea- sonable responses. Initialized from pre-trained LLM, the MLLM contains visual encoder (e.g., CLIP-L (Radford et al., 2021 ) to extract the visual features f. and an adapter W to project f into the language modality. We follow the training of LLaVA (Liu et 2023). which summarized as:
= {Ii, I2, I2, ..., ..,, I1}.
Enovis (V).
(I)
2eg (ILLM(a1........ W()),
We adopt Flan-T5-XXI. (Chung al. 2022, which has been specifically fine-tuned for summarization, as summarization model for the original MLL.M (MLLM*).
3

---

Published as a conference paper at ICLR 2024
where is the length of the word token in C. C can be the image caption (Features Alignment) the data (Instruction Tuning). The MLLM follows the standard auto-
multimodal cettruction-folllwwee regressive training for the next token prediction and then can serve visual assistant for various tasks such as visual question answering and complex reasoning. Although the MLLM is capable of visual perception via the above training, its output is still limited text.
3.2 MELLEGGIIDDDD IMAGE EDITING (MGIE)
As illustrated in Fig. - we propose MLLL-GGidded Image Editing (MGIE) to edit an input image V into a goal image O, by given instruction X. To handle imprecise instructions, MGIE contains the MLLM and learns to derive explicit yet concise expressive instructions E. To bridge the language and visual modality, we add special [ [IMG] tokens after & and adopt the edit head T to transform them. They serve as the latent visual imagination from the MLLM and guide our diffusion model F to achieve the intended editing goal. MGIE is then able to comprehend ambiguous commands with visual-related perception for reasonable image editing.
Concise Expressive Instruction. From features alignment and instruction tuning, the MILMM can offer visual-related responses with its cross-modal perception. For image editing, we use this prompt as the language input with the image and derive
"what will this image be like if finstruction a detailed explanation of the editing command. However, those explanations are always too lengthy and involve redundant descriptions, which even mislead the intention. To obtain succinct narrations, apply pre- nee-traied summarizer and make the MLLM learn to generate the summarized outputs. We treat this explicit yet concise guidance as expressive instruction &
Summ(MLLM* ([ [prpot, X] W(J))
{wi, W2, .... w w/ MLLM (WE,, Wt w IW()),
Lins
CELoss(W,, we),
where we apply the cross-entropy loss (CELoss) to train the MLLM via teacher forcing. can pro- vide a more concrete idea than X' such as linking "desserr" with "sand dunes" and "cacti or small shrubs", which mitigates the comprehension gap for reasonable image editing. This strategy further enhances our efficiency. During inference, the trained MGIE straightforwardly derives concise & in- stead of rolling out lengthy narrations (22.7 vs. 64.5 tokens) and relying on external summarization. MGIE now can acquire a visual imagination of the editing intention but is confined to the language modality. To bridge the gap. we append N visual tokens [IMG] after E, where their word embed- dings are trainable, and the MLLM also learns to generate them through its language modeling (LM) head. Inspired by GILL (Koh et al. 2023), the visual tokens are treated as visual-related instruction understanding in & and establish a connection between the language and vision modalities.
Image Editing via Latent Imagination. We adopt the edit head to transform [TMG] into actual visual guidance. T is segpuence-t--seqqeee model, which maps the sequential visual tokens from the MLLM to the semantically meaningful latent U = {u1, U2 ..., } as the editing guidance:
Ut T([u,,... 4--13 3} | {e(zms) + h (1888))) }),
where is the word embedding and h is the hidden state (from the last layer of MILMM before the LM head) of [ [IMG] Specifically, the transformation over e can be treated as a general representation in the visual modality, and h is an instance-aware visual imagination for such editing intention. Our T is similar to GILL and BLIP-2 Li et al. 2023b a for extracting visual features.
To guide image editing with the visual imagination 24 we consider latent diffuuson model F (Rom- bach et al. 2022), which includes the variational autoencoder (VAE) and addresses denoising diffu- sion in the latent space. Our goal of F is to generate the latent goal EnovAE(O) from preserving the latent imput e = Encwar()) (V) and following the editing guidance {u}. J. The diffusion process keeps adding noises to o as 2t, where the noise level increasing over timesteps We then learn the UNet co to predict the added noise (Ho et al. 2020 As LDM, we inject the visual imagination into co via
the cross-attention layer Attention Q, K, V) softmax QK" V with
Vdim
Q w-(0) (2)(2,), K
w (w), w( [u],
(4)
4

---

Published as a conference paper at ICLR 2024
Method
EVR
GIER
MA5k
MagicBrush
L11
DINOf CVSt
L11
SSIM
CVSt
L11
SSIM
LPIPS!
Ltl
DINOT CVST CTS1
InsPix2Pix
0.189
67.82
81,38
0.144
57.51
86.63
0.176
58.92
0.359
0.101
71,46
85.22 29.34
LGIE
0.159
69.71
82.04
0.152
56.86
86.99
0.144
64.60
0.327
0.084
80.90
88.87
30.10
MGIE
0.163
71.49
81.73
0.135
59.24
88.59
0.133
66.25
0.298
0.082
82.22
91.14
30.40
Table
Zero-shot editing results. All models are only pre-trained on IPr2Pr (Brokks al. 2023
Method
EVR
GIER
MA5k
MagicBrush
L14 DINO CVSt
L.1
SSIM,
CVSt
L14
SSIM
LPIPS..
L14
DINO CVSt CTS1
InsPix2Pix
0.166
70.79
82.76
0.111
64.86
91.49
0.122
67.12
0.267
0.063
87.99 93.83 30,93
LGIE
0.147
74.71
85.06
0.104
65.30
90.61
0.094
71.47
0.246
0.058
93.57 31.33 88.09
MGIE
0.146
75.65
85.28
0.105
68.68
92.42
0.082
72.91
0.235
0.057
90.65 95.28 31.73
Table 2: Fine-tuned editing results. All models are further fine-tuned and adapted each dataset.
where yp is the flattened operation, w( W and W:( are learnable attention matrices. Following InsPix2Pix, we also concatenate v with 2,. In this way, our F can condition both y and to perform image editing. We take classifier-free guidance (Ho & Salimans 202T). and the score estimation S8 is extrapolated to keep away from the unconditional Ø, where the editing loss Cedin is calculated as:
s0 v, {u}) = s0 R)
ary (so v, Ø) s0(21, Ø, Ø)) +ax (50(2), U, {u}) - so sz v, Ø)),
(5)
Dedi = E,,,, (u) (0,.1), [||c - (2 t, {u))
where ay and ax are the weights of the guidance scale for the image and the instruction. Similar to InsPix2Pix, we randomly make Ø {u} 2, or both = Ø for 5% of data during training. After we have the generated latent of through the denoising process by c0. we can obtain the editing result
O' = Decvag During inference, we use ay = 1.5 and ax = 7.5.
3.3
LEARNING OF MGIE
Algo. presents the learning process of the proposed MGIE. The MILMM learns to derive concise & via the
Algorithm 1 -LLLM-Guided Image Editing
instruction loss Lins- With the latent imagination from [IMG), T transforms their modality and guides F synthesize the resulting image. The editing loss Lodit applied for diffusion training. Most weights can be frozen (self-attention blocks inside the MLLM), lead-
I:
while TRAIN do
2:
V. X, O input/instruction/gaal triple
3:
{w} summarized explanation
[w"] } MLLM(VV X))
5:
Lim + instruction loss
Eq
6:
U T([[
ing to parameter-efficient end-to-end training. Over-
7:
O' F(V.
all optimization of Call Lins + 0.5 Ledit can be:
8:
Lesn + editing loss
Eq
9:
Can f- overall training loss
min MLLM,W. W.
Call
(6)
10: end while
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Evaluation Metrics. We use IPr2Pr (Brooks et al.
20230 as our pre-training data.
contains 1M CLIP-filtere data, where instructions are extracted by GPT-3 (Brown et al., (020)). and
2023). For comprehensive evaluation, consider various editing aspects. EVR (Tan et al. 2019 collects 5.7K triples from PhotoshopRe- quest. We treat the standard pixel difference (L1) and visual feature similarity from DINO (Caron) et al. 202 or the CLIP visual encoder (CVS) between generated images and ground
images are synthesized by Prompt-to- Prooppp (Hertz et al.
goals as the evaluation metrics. GIER (Shi et al., 2020) crawls a larger-scale 29.9K triples also from online forums. Since there are more examples about global optimization, we apply L1, CVS, and Structural
5

---

Published as a conference paper at ICLR 2024
Trade-of off Curve on MagicBrush
Arch. Method
MA5k
MagicBrush
100
L1L
SSIMI LPIPS!
L11
DINOT CVSt CTS1
Input Image Similarity
90
InsPix2Pix
0.176
58.92
0.359
0.101
71.46
85.22 29.34
LGIE
0.178
57.26
0.372
0.133
67.53
82.49 28.79
FZ
MGIE
0.163
57.54
0.366
0.128
71.65
86.00 29.43
80
LGIE
0.166
60.11
0.357
0.124
71.04
85.47 29.37
FT
MGIE
0.163
61.38
0.348
0.101
74.79
87.12 29.68
IndPRk2PPx
70
LGIE
LGIE
0,144
64.60
0.327
0.084
80.90
88.87 30.10
MGIF
E2E
MGIE
0.133 66.25
0.298
0.082
82.22
91.14 30.40
5
10
15 20
Text-Image Direction Similarity
Table 3: Ablation study. We attempt FZ, FT, or E2E utilize expressive instructions. FZ directly treats expressive instruc- tions as the inputs to frozen InsPix2Pix. FT further fine-tunes InsPix2Pix and makes it adapt to expressive instructions. Our E2E learns expressive instructions along with the MLLM and trains the diffusion model in an end-to-end manner.
Figure 3: Trade-off curve for im- age editing. We set xx as 7.5 and vary ay in [1.0, 2.2]. For both edit (X-axis) and input consistency (Y- axis), higher is better.
Similarity Index (SSIM). MA5k (Shi et al.
2022) consists of 24.8K triples and aims at changing the contrast, brightness, saturation of a whole photo. We leverage L1, SSIM, and Learned Perceptual Image Patch Similarity (IPIPS) (Zhang et al. 2018) as the photo difference MagicBrush (Zhang al. 2023a annotates 10.5K triples. We follow them to use LI, DINO, CVS, and text-visual feature similarity (CTS) (Hessel et al. (02T) between goal captions and resulting images. We treat the same saining/validationneessst split as the original settings. Without specific mention, all evaluations are averaged from 5 random seeds in a zero-shot manner, where models are only trained on IPr2Pr.
Baselines. We treat InsPix2Pix (Brooks et al. (023), built upon the CLIP text encoder with a diffu- sion model for instruction-basee image editing, our baseline. We consider similar LLM -Lgguddded image editing (LGIE) model, where LLaMA-7B (Touvron et al. 2023) is adopted for expressive instructions & from instruction-only inputs but without visual perception.
ins
CLIP
ins
LLM
ins
MLLM
E
img
Diff
img
Diff
img
Diff
InsPix2pix
LGIE
MGIE
Implementation Details. The MLLM and diffusion model F. are initialized from VLa7 Liu al. 2023) and StableDiffussin-vl.. (Rombach et al., 2022). We jointly update both for the image editing task. Note that only word embeddings and LM head in the MLLM are trainable. Following GILL (Koh et al., | 2023). we use N=8 visual tokens. The edit head
a 4-layer Transformer, which transforms language features into editing guidance. We adopt Adam W (Loshchilov & Hutter] 2019) with the batch size of 128 to optimize MGIE. The learning rates of the MLLM and J are 5e-4 and le-4, respectively. All experiments are conducted in PyTorch (Paszke et al. 2017) on 8 A00 GPUs.
4.2 QUANTITATIVE RESULTS
Table I shows the zero-shot editing results, where models are trained only on IPr2Pr. For EVR and GIER that involve Photoshop-style modifications, expressive instructions can reveal concrete goals instead of brief but ambiguous commands, which makes the editing results more similar to intentions (e.g., higher 82.0 CVS on EVR by LGIE and higher 59.2 2 SSIM on GIER by MGIE). For global photo optimization on MASK, InsPix2Pix is hard to deal with due to the scarcity of related training triples. Though trained from the same source, LGIE and MGIE can offer detailed explanations via learning with the I.MM, but LGIE is still confined to its single modality. With access to images, MGIE derives explicit instructions such as which regions should brighten or what objects are more distinct. It can bring significant performance boost (e.g., higher 66.3 SSIM and lower 0.3 photo distance). Similar
As there is no object alteration in MASK feature-based DINO and CVS cannot clearly tell the difference.

---

Published as a conference paper at ICLR 2024
CLip-S(image, Expressive Ins.)
Expressive Instruction
Instruction-baser Image Editing
30
75
Inout
LGI
NGE
LGIE
KGE
25
26.5
50-
$3.7
57.0
2.33
2-
22.6
34.3
2.01
25
1.72
15
10.0
8.7
Input Image
Goal Image
More Practicality
Less Hallucination
Instruction Ground-truth
Overall
Figure 4: CLIP-S across im- ages (imput/ goal) and expres- sive instructions.
Figure 6: Human eval of image edit- ing results in terms of instruction fol- lowing, ground-truti relevance, and overall quality.
Figure 5: Human eval of ex- pressive instructions quality.
results are found MagicBrush. MGIE also achieves the best performance from the precise visual imagination and modifies the designate targets as the goals (e.g., higher 82.2 DINO visual similarity and higher 30.4 CTS global caption alignment).
To investigate instruction based image editing for the specific purpose, Table 2 fine- fine-tunes tunes models on each dataset. For EVR and GIER, all models obtain improvements after the adaptation to Photoshop- style editing tasks. Since fine-tuning makes expressive instructions more domain-speciff as well, MGIE increases the most via learning with domain-related guidance. This also helps our diffu- sion model to demonstrate concrete edited scenes from the fine-tuned MLLM, which benefits both global optimization and local modification (e.g., notably lower 0.24 LPIPS on MA5K and higher 95.3 CVS on MagicBrush). MGIE is consistently superior to LGIE in all aspects of editing since our visual-aware guidance is more aligned with the intended goal. From the above experiments, we illustrate that learning with expressive instructions can effectively enhance image editing, and visual perception plays a crucial role deriving explicit guidance for the greatest enhancements.
Trade-off between ox and cry. There are two goals in image editing: manipulate the target as the instruction and preserve the remaining as the input image. Fig. plots the trade-off curves between the instruction (axx) and input consistency (ar)) We fix ox as 7.5 and vary ay in [1.0,2.2] Higher ay will make editing result more similar to the input but less aligned with the instruction. X-axis calculates the CLIP directional similarity as how much the editing follows the instruction; Y-axis is the feature similarity to the input image from the CLIP visual encoder. Through concrete expressive instructions, we surpass InsPix2Pix in all settings. Our MGIE additionally results in comprehensive enhancements by learning with explicit visual-related guidance. This supports robust improvement, whether requiring higher input correlation or edit relevance.
4.3 ABLATION STUDY
MLLM-Guided Image Editing exhibits encouraging improvement in both zero-shot and fine-tuning scenarios. Now, we investigate different architectures to use expressive instructions. Table ers FZ, FT, and our E2E. FZ directly uses the derived expressive instructions as the input prompts to the frozen InsPix2Pix. In spite having additional guidance, the scenario still differs from the trained editing instructions, which makes it difficult to deal with. LGIE even hurts the performance as it may mislead due to the shortage of visual perception. FT fine-tunes InsPixPix and adapts it to expressive instructions. These results support that image editing benefit from explicit guidance along the derivation of instructions from the LLM/MLLM. E2E updates the editing diffusion model in conjunction with the LM, which learns extract applicable guidance and discard irrelevant nar- ration simultaneously through the end-to-end hidden states. In addition, our E2E can also avoid the potential error that may be propagated from the expressive instructions. Hence, we observe the most enhancements in both global optimization (MA5k) and local editing (MagicBrush). Among FZ, FT, and F2F, MGIE consistently surpasses LGIE. This indicates that expressive instructions with crucial visual perception are always advantageous across all ablation settings.
Why MILM Guidance is Helpful? Fig. 4 presents the CLIP-Score between input or ground-truth goal images and expressive instructions. A higher CLIP-S input images indicates that instructions
During the ablation study, we employ concise summarized expressive instructions for fair comparison.

---

Published as a conference paper at ICLR 2024
EVR
add lightning and make the water reflect the lightning
GIER
remove the christmas tree in the background
MASK
brighten image lot, sharpen photo
MagicBrush
let the donuts have strawberry glaze them
Input Image
InsPix2Pix
LGIE
MGIE
Ground Truth
Figure 7: Qualitative comparison between InsPix2Pix, LGIE, and our MGIE, For the 1st example, MGIE can showcase the clear "lightning" in the sky and its reflection on the water. For the 2nd one, although LGIE accurately targets the Christmas tree, only MGIE removes it in the background. For photo optimization (the 3rd example), InsPix2Pix fails to adjust the brightness, and LGIE makes the whole photo white and obviously distinct. contrast, MGIE follows the instruction to brighten as well as sharpen it. Moreover, in the 4th one. MGIE puts the "glaze" only on the donuts, but basclines even draw the entire image in strawberry pink.
relevant to the editing source. Better alignment with goal images provides explicit and correlated edit guidance. Without access to visual perception, expressive instructions from LGIE are limited to general language imagination, which is not tailored to the source image. The CLIP-S even lower than the original instructions. By contrast, MGIE is more aligned with inputs/goals, which explains why our expressive instructions are helpful. With clear narration of the intended result, our MGIE can achieve the greatest improvements in image editing.
Human Evaluation. Apart from automatic metrics, we conduct human evaluation to study gen- erated expressive instructions and image editing results. We randomly sample 25 examples for each dataset (100 total) and consider humans to rank across baselines and MGIE. To avoid potential ranking bias, hire 3 annotators for each example. Fig. plots the quality of generated expressive instructions. Precise guidance is informative and aligns with the intended goal (More Practicality). At the same time, should avoid incorrect unrelated explanations (Less Hallucination). Firstly,
8

---

Published as a conference paper at ICLR 2024
EVR
change the background and add star wars background
GIER
remove all people except baby from image
If you were to change the background to Star Wars movie, the image would look like a star Trek background.
Add a background that related to the Star War franchise, such as lightsaber spaceship.
The anly people still the image would be two people the foreground. The rest needs be removed.
If we remove all, we would baby sitting the beach with shark and balloons,
with
MASk
the desired image has more light contrast
MagicBrush
close the dog's 's mouth
Adjust contrast and brightness of desired area, making look darker and more visible.
Adjust between light and dark areas to bring out the details of the leaves and the tree trunk.
picture of dog bitten object with the dog the foreground and dark background.
dog in image had its mouth covered, it would appear were "playing with" the stick or chewing on it,
Input Image
LGIE
MGIE
Input Image
LGIE
MGIE
Figure 8: Qualitative comparison of expressive instructions by LGIE and our MGIE. Due to the limitation of the single modality, LGIE only have language-basee insight but may derive irrel- evant or even wrong explanations for image editing (e.g., "two people still in the foreground" for GIER). With access images, MGIE provides explicit visual imagination after the editing such as "baby on the beach with shark" or "bring out details of leaves and trunk". More surprisingly, we can link "lightsaber or spaceship" from Star Wars and describe "chewing on the stick" for the dog, which is aligned with the intended goal.
over 53% support that MGIE provides more practical expressive instructions, which facilitates the image editing task with explicit guidance. Meanwhile, 57% of labelers indicate that our MGIE can prevent irrelevant descriptions from language-derived hallucinations in LGIE since it perceives the image to have a precise goal for editing. Fig. 6 compares the image editing results by InsPix2Pix, LGIE, and our MGIE in terms of instruction following, ground-truth relevance, and overall quality. The ranking score is ranging from I to 3, higher is better. With derived expressive instructions from the LLM or MILM, LGIE and MGIE both outperform the baseline and perform image editing that is correlated with the instruction as well as similar to the ground-truth goal. Additionally, since our expressive instructions can provide concrete and visual-aware guidance, MGIE has the best human preference in all aspects, including the overall editing quality. These performance trends also align with automatic evaluations, which support our usage of metries.
Inference Efficiency. Despite relying MILM facilitate image editing, MGIE only rolls out concise expressive instructions (less than 32 tokens) and contains feasible efficiency as InsPix2Pix. Table presents the inference time cost on an NVIDIA A A100 GPU. For a single input, MGIE can accomplish the editing task in 10 seconds. With greater data parallelization, we take similar amount of time (e.g.. 37 seconds when batch size 8). The entire process can be affordable one GPU (40GB). summary, our MGIE surpasses the baseline on quality yet maintains competitive efficiency, leading to effective and practical image editing.
BS InsPix2Pix MGIE
1
6.8
9.2
4
16.5
20.6
8
31.5
36.9
Table 4: Time cost.
Qualitative Comparisons. Fig. illustrates the visualized comparison on all used datasets. Fig. further compares the expressive instructions by LGIE or MGIE. Our superior performance benefits from the explicit guidance of visual-related expressive instructions. Please visit our project website for more qualitative results.
CONCLUSION
We propose MLLM-Guided Image Editing (MGIE) to enhance instruction- based image editing via learning to produce expressive instructions. Instead brief but ambiguous guidance, MGIE derives explicit visual-aware intention and leads to reasonable image editing. We conduct extensive studies from various editing aspects and demonstrate that our MGIE effectively improves performance while maintaining competitive efficiency. We also believe the MLLM-guided framework can contribute to future vision-and and-aanddnnnn language research.
Project website: https: //m1 1m- ie. g thub.

---

Published as a conference paper at ICLR 2024
REFERENCES
Omri Avrahami, Dani Lischinski. and Ohad Fried. Blended Diffusion for Text-driven Editing of Natural Images. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2LIVE: Text- Driven Layered Image and Video Editing. In European Conference Computer Vision (ECCV), 2022.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. InstructPix2Pix: Learning to Follow Image Editing Instructions. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In International Conference on Computer Vision (ICCV), 2021.
Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apidianaki, and Smaranda Muresan. Spy Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors. In Annual Meetings of the Association for Computational Linguistics (ACL), 2023.
Zheng Chang, Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, and Boxin Shi. L-CAD: Language- based Colorization with Any-level Descriptions using Diffusion Priors. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin. Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontuek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemt Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. In arXiv:2204.02311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Eestruction-Fintture Lan- guage Models. In arXiv:2210.1144
2022.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. DiffEdit: Diffusion- based Semantic Image Editing with Mask Guidance. In International Conference on Learning Representations (ICLR), 2023.
Katherine Crowson, Stella Biderman. Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. In European Conference on Computer Vision (ECCV), 2022.
10

---

Published as a conference paper at ICLR 2024
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W.Taylor. Tell, Draw, and Repeat: Gener- ating and Modifying Images Based on Continual Linguistic Instruction. In International Confer- ence on Computer Vision (ICCV), 2019.
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. LayoutGPT: Compositional Visual Planning and Gen- eration with Large Language Models. In arXiv:2305. /5393, 2023.
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang. SSCR: Iterative Language-Bane -aaeee Image Editing via Self- Supervised Counterfactual Reasoning. In Con- ference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang. Language-Driven Artistic Style Transfer. In
European Conference on Computer Vision (ECCV), 2022.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. In Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH), 2022.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In Conference on Neural Information Processing Systems (NeurIPS), 2015.
Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, Hyunjoon Jung, and Xin Erie Wang. Photoswap: Personalized Subject Swap- ping Images. In arXiv:2305. 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reasoning without training. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-Prompy Image Editing with Cross Attention Control. In International Conference for Learning Representations (ICLR), 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2021.
Jonathan Ho and Tim Salimans. Classifier-Free Free Diffusion Guidance. In Conference Neural Information Processing Systems (NeurIPS), 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Confer- ence on Neural Information Processing Systems (NeurIPS), 2020.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. International Conference Machine Learning (ICML), 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models. In Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating Images with Multimodal Lan- guage Models. In arXiv:2305. 17216, 2023.
Bowen Li, Xiaojuan Qi. Thomas Lukasiewicz. and Philip H. S. Tor. ManiGAN: Text-Guided Image Manipulation. In Conference Computer Vision and Pattern Recognition (CVPR), 2020.
11

---

Published as a conference paper at ICLR 2024
Controllable text-to-lmage Generation and Editing. In arXiv:2305.14722, 2023a. Dongxu Li, Jumnan Li, and Steven Hoi. BLif-Diffusion: Pre-trained Subject Representation for
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In International Confer- ence on Machine Learning (ICML), 2023b.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LLMM-rounnde Diffusion: Enhancing Prompt Understanding of Text-to-Image Image Diffusion Models with Large Language Models. In 1rXiv:2305.13655, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In arXiv:2304.08433 2023.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Con- ference for Learning Representations (ICLR), 2019.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-PHe Compositional Reasoning with Large Language Models. In arXiv:2304. 09842, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Er- mon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In International Conference for Learning Representations (ICLR), 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Null-text Inversion for Editing Real Images using Guided Diffusion Models. In Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language. In Conference Neural Information Processing Systems (NeurIPS), 2018.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference on Machine Learning (ICML), 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In International Conference on Learning Representations Workshop (ICLRW), 2017.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text- Driven Manipulation StyleGAN Imagery. In International Conference on Computer Vision (ICCV), 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Inter- national Conference on Machine Learning (ICML), 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. In arXiv:2204.06122, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative Adversarial Text to Image Synthesis. In International Conference on Machine Leam- ing (ICML), 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

Published as a conference paper at ICLR 2024
Chitwan Sahari, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans. Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Jing Shi. Ning Xu, Trung Bui, Franck Dernoncourt, Zheng Wen, and Chenliang Xu. A Benchmark and Baseline for Language-Driven Image Editing. Asian Conference on Computer Vision (ACCV). 2020.
Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Dernoncourt, and Chenliang Xu. Learning by Planning: Language- Guided Global Image Editing. In Conference Computer Vision and Pat- tern Recognition (CVPR), 2022.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative Pretraining in Multimodality. In arXiv:2307.05222. 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. In arXiv:2303.00 081288 2023.
Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing Visual Relation- ships via Language. In Annual Meetings of the Association for Computational Linguisties (ACL),
2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. In arXiv:2302.1397/ 2023.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A Generative Image-to-tex Transformer for Vision and Language. In Transactions on Machine Learning Research (TMLR), 2022.
Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. InstructEdit: Improving Automatic Masks for Diffusion- based Image Editing With User Instructions. arXiv:2305. 18047, 2023a.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Ya- sumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. Imagen Editor and EditBench: Advancing and Evaluat- ing Text-Guided Image Inpainting. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Shuchen Weng, Peixuan Zhang, Zheng Chang, Xinlong Wang, Si and Boxin Shi. Affective Image Filter: Reflecting Emotions from Text to Images. In International Conference on Computer Vision (ICCV), 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models. In arXiv:2303.0467., 2023.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action. arXiv:2303.1 13381, 2023.
Kai Zhang, Lingbo Mo. Wenhu Chen, Huan Sun, and Yu Su. MagicBrush: A Manually Annotated Dataset for Instruction-Guider Image Editing. arXiv:2306. 100/2, 2023a.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao. Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. In arXiv:2303. 16/99, 2023b.
13

---

Published as a conference paper at ICLR 2024
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as Perceptual Metric. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, and Ran Xu. HIVE: Harnessing Human Feedback for Instructional Visual Editing. In arX1v:2303.09611, 2023c.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models. In arXivv2205.01068 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal Chain-of-Thought Reasoning in Language Models. In arXiv:2302.0092 2023d.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In arXiv:2304. 10592, 2023.
14


Reviews:
[{"review_id": "xfkinZm6um", "reviewer": "Reviewer_BZAL", "summary": "This paper proposes a novel text-guided image editing method that builds on the instructpix2pix model, which fine-tunes a diffusion model on instruction-image pairs. Authors propose to leverage a multimodal (image + text) large language model to generate more precise and expressive editing instructions, and improve editing ability. The method is trained on the instructpix2pix dataset, and evaluated on 4 datasets, considering different types of edits.", "questions": "- Image editing performance can be influenced by the random seed. Results in tables 1-2 compared to results reported in Zhang et al (MagicBrush) show that there can be a noticeable difference for some metrics (e.g. 70->74 for DINO score on instructpix2pix). Several of these evaluations show performance scores that are relatively close, have authors investigated how consistent these rankings are across multiple seeds? \n\n- Are ground truth images (post edit) available for all datasets? Why were these specific sets of evaluation metric chosen? Why not e.g. measure image quality using FID?\n\n- Authors compare inference times, but do not mention training times. How much more more expensive is it to train this new model compared to pix2pix? This is relevant as noticeable performance gains can be observed when fine-tuning on a specific instruction/edit types.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Image editing is a timely and challenging task. Authors demonstrate that they are able to successfully carry out a large set of edit types across multiple datasets. Visual results look very promising, and suggest that the proposed changes can yield noticeable gains over instructpix2pix The evaluation is detailed and the method is analysed from a lot of different angles.\n\nThe idea of replacing the CLIP encoder with a more expressive vision-language model is sound, as providing more detailed instructions can help guide the diffusion model towards the desired output. Authors have made efforts to go beyond simple replacement of the text encoder and added functionality (summarization, adaptation to visual content) that improve performance.", "weaknesses": "My main concern with this work is the limited novelty. The crux of the innovation is the use of a multimodal language model instead of a CLIP model in the instructpix2pix setting. The second main innovation is the introduction of [IMG] tokens, which are processed by a transformer head to generate conditioning embedding for a LDM model. This approach is very strongly inspired from Koh et al. 2023, where they train a language model to generate image tokens, which are then transformed via a transformer architecture, and used as conditioning for stable diffusion based generation. The source of inspiration should be credited more clearly (the main reference in the method section simply mentions using a similar feature extractor architecture).\n\nThe presentation of the paper could be improved as well.  The paper is written in a confusing way, and lacks explanation and justifications for design decisions. For example, equation (5) is introduced without any justifications or intuitive explanation, and author do  not explain what they refer to as by score. Similarly, what authors refer to as MLLMs (pre-trained LLMs adapted to take visual inputs as well) is not clearly defined until section 3.1. Multimodal language models can be designed and trained in different ways (e.g. trained with vision-text inputs jointly), and authors should clarify that they refer to a specific type of models. Similarity, the edit head T was not clearly explained, I needed to read the GILL paper (Koh et al.) to understand how these features were generated. Another example is figure 2, which is mentioned at the beginning of chapter 3.2 and shows a MLLM* model without explanation, while this model is only introduced (in a footnote) in a later paragraph. \n\nWhile the evaluation is detailed with experiments carried out on many datasets, state of the art references are limited. The only pre-existing work that authors compare to is instructpix2pix, while the LGIE baseline is an overly poor baseline, which is expected to perform worse than any model with vision-language mappings (e.g. CLIP) in a lot of settings. It does not make a lot of sense to ask a pure language model to hallucinate detailed descriptions of an unseen image. Evaluations on prompt-based editing methods are available on the magic brush dataset, and could provide additional context. For consistency, instructions could easily be converted to a prompt using an LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JupnXBOgM2", "reviewer": "Reviewer_fMJn", "summary": "This submission proposes an approach towards improving the visual output quality of instruction-based image editing. Authors put forward the hypothesis that natural text-based edit instructions, provided by humans, are oft terse and this hinders contemporary learning-based models from successfully capturing and dependably following intended image-edit meanings. \n\nThe crux of the proposed strategy involves learning to map terse input text, representing image edit-instructions, to more expressive (evidently more verbose) output text, in order to provide more explicit image-editing guidance. The work explores the efficacy of leveraging cross-modal capabilities, contemporary language-models, and sensitivity to visual inputs in order to positively influence edit-instruction quality. This multi-modal component is evidenced to be important for text-edit quality and resulting downstream edited images. Quantitative and qualitative evaluation of the proposed strategy, across various metrics and human rankings, are reported in comparison with recent work and a relevant baseline, across four benchmark datasets.", "questions": "* Can the authors commit to a full source code release? The submission opted to stay silent on this issue. Code release would be of clear benefit to the community aiding reproducibility, public probing of method robustness, consistency, range of model abilities. This will undoubtedly increase the value of the contributions. If impossible; web page might be extended to allow public inference-time testing.\n\n* The property of 'elaborate descriptions' is posed as something one wishes to evade in natural commands and yet 'explicit yet detailed guidance' is conversely noted a sought after notion. Do the authors suggest that the former pertains largely to image space, in a similar fashion to e.g. regional masks? Concrete examples of the considered 'elaborate descriptions' may help to clarify this point.\n\n* How is the phrase 'reasonable image editing' to be defined?\n\n* Minor: Small icons (flame, blue cube) tagged to components in the system schematic (Fig.2) presumably represent learnable and frozen model components, respectively. Suggest to make this key explicit.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problems being addressed here are real and are important -- principled solutions and progress towards techniques capable of realising consistently high quality and complex (intended) image edits, that are also of low cost in terms of required human effort, will be of high value to the community and will additionally result in widely applicable and practical end-user benefits.  \n\nThe technical components of the work, the manner in which the various components are concatenated together, appears reasonably straightforward. Explanation of the guided image editing strategy is aided by basic yet clear schematics to aid understanding (Fig. 2, specifically). \n\nThe investigation can be considered reasonably thorough with the inclusion of experimental work covering (i) method efficacy (quant. & qual.), (ii) vision & language hyper-parameter sensitivity, (iii) ablating instruction generation components, (iv) compute. Reported qualitative results are quite compelling and show some good perceptual improvements over baselines. Further results (public anon-web page, supp. materials) are appreciated and provide some aid for apprehensions over method robustness, reliability.\n\nWriting is of a reasonable standard in general. I enjoyed reading the paper.", "weaknesses": "Leveraging multi-modal language-models provides an intuitively promising avenue, when striving to follow image-manipulation intentions that are defind by only terse human text instruction. Good evidence is provided that the presented strategy goes some way to tackling the observed short-comings however technical contributions (size, sufficiency) can be regarded as moderate. The overarching system makes use of an array of pre-existing components however the particular implementation that facilitates component concatenations can be regarded as somewhat novel.\n\nFor this venue, an important factor that would strengthen the submission might look to provide additional understanding, discussion on the long-term sustainability of the proposed style of approach. The requirement for intermediary 'on-the-fly' remappings, reconfigurations of text input cannot be regarded as a very elegant or pleasing solution. I explicitly note that this is not grounds for rejection, my point is rather that further thinking, discussion on the fundamental gaps that prevent base models from correctly realising terse (yet human-parseable) instruction would likely prove elucidating and of high value. An empirical avenue here might involve investigating pre- and post-hoc parts-of-speech (POS) distributions, or other statistical evaluation of the edit-text distributions. Is the long-term goal, alternatively, to fashion single models, capable to understand natural (terse) human instruction? The point touches on well-understood (dis-)advantages of modular-component systems c.f. end-to-end.\n\nI would be keen for authors to discuss these points and I am open to modifying my score. \n\nMinor suggestions:\n\n1. Authors may wish to update to an ICLR24 template (c.f. ICLR23) \n\n2. On pp.2: suggest explicitly expand acronym 'IPr2Pr' on first use (presumably Instruction Prompt-2-Prompt ?)\n\n3. Precision of some phrasing could be tightened, towards aiding reading understanding. See example suggestions in following section. \n\n4. The (useful!) model schematic image, found between Figure 3 and Figure 4, is unnumbered. Is this by design ? Suggest 'Baselines' paragraph might serve as suitable caption content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qTpbcoBQBk", "reviewer": "Reviewer_RXEx", "summary": "This paper proposes a new instruction-based image editing approach. The Multimodal large language models (MLLMs) are incorporated into the editing, formulating MLLM-Guided Image Editing (MGIE). Different editing operations are utilized for testing, including Photoshop-style modification, global photo optimization, and local editing.", "questions": "1.\tHow to balance the loss weights of $\\mathcal{L}_{ins}$ and $\\mathcal{L}_{edit}$ in Eq. 6?\n\n2.\tCan this approach add new objects into the image?\n\n3.\tWhat are the details of the user study’s participants? Like their ages, educations, and the time for taking part in the user study.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method can have effective performance on the chosen datasets of evaluation. Especially, a lot of evaluations are conducted with subjective assessment.", "weaknesses": "1.\tThe performance is limited by the dataset utilized for training, especially for learning effective instruction and editing performance. Will this approach complete the editing operation which is not appeared in the training data?\n\n2.\tThe template of this paper is wrong, it is still the template of ICLR2023.\n\n3.\tThere is no ablation analysis for the loss functions and their alternatives. \n\n4.\tThe editing effects are not ideal, for example, the woman in the background of Fig. 1 is removed, while there are still some residuary artifacts in the background.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0XYAyUFY6", "reviewer": "Reviewer_TXUn", "summary": "The paper introduces an image editing method guided by MLLM. This approach can learn from expressive instructions and offer explicit guidance. Comprehensive experiments demonstrate the method's effectiveness.", "questions": "I am curious about the method's potential in handling tasks like:\n\n1. Transferring image texture. Language-Driven Artistic Style Transfer, ECCV 2022.\n\n2. Modifying an object's color. L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors, NeurIPS 2023.\n\n3. Interpreting user-provided emotions. Affective Image Filter: Reflecting Emotions from Text to Images, ICCV 2023.\n\nIncluding comparisons or referencing these studies could further enrich the paper's depth and context.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written.\n\nThe introduced technique.is novel and interesting.\n\nThe experiments are sufficiently conducted.\n\nThe presented results seems promising.", "weaknesses": "While certain aspects of the work might appear less novel, its practical effectiveness compensates for this.\n\nThere are typographical errors. Specifically, \"Methods\" in Tables 1 and 2 should be placed at the top.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)