CAMILA: Context-Aware Masking for Image Editing with Language Alignment
Authors: Hyunseung Kim
Chiho Choi
Srikanth Malla
Sai Prahladh Padmanabhan
Saurabh Bagchi
Joon Hee Choi
Venue: NeurIPS 2025 poster ()
Keywords: Image Editing
Multimodal Large Language Model
Context-Aware
Primary area: deep_learning

Abstract: Text-guided image editing has been allowing users to transform and synthesize images through natural language instructions, offering considerable flexibility. However, most existing image editing models naively attempt to follow all user instructions, even if those instructions are inherently infeasible or contradictory, often resulting in nonsensical output. To address these challenges, we propose a context-aware method for image editing named as CAMILA (Context-Aware Masking for Image Editing with Language Alignment). CAMILA is designed to validate the contextual coherence between instructions and the image, ensuring that only relevant edits are applied to the designated regions while ignoring non-executable instructions. For comprehensive evaluation of this new method, we constructed datasets for both single- and multi-instruction image editing, incorporating the presence of infeasible requests. Our method achieves better performance and higher semantic alignment than state-of-the-art models, demonstrating its effectiveness in handling complex instruction challenges while preserving image integrity.

CAMILA: Context-Aware Masking for Image Editing with Language Alignment
Hyunseung Kim 1.2" Chiho Choi'
Srikanth Malla'
Sai Prahladh Padmanabhan'
Saurabh Bagchi² Joon Hee Choi'
'Samsung Semiconductor, USA Puurdue University
{kim4061, (bagchi)@purruue edu fchiho1.chols srikanth.n m, sai. sai.prahladh, jh4. jh4.choi) (chooi)ssssuun.coom
Abstract
Text-guided image editing has been allowing users transform and synthesize images through natural language instructions, offering considerable flexibility. However, most existing image editing models naively attempt to follow all user in- structions, even if those instructions resulting in nonsensical output. To address these challenges, we propose context- aware method for image editing named as CAMILA (Context-Awarr Masking for Image Editing with Language Ali Alignment). CAMILA is designed to validate the contextual coherence between instructions and the image, ensuring that only relevant edits are applied to the designated regions while ignoring non-executable instructions. For comprehensive evaluation of this datasets for both single- and multiiinstruction image editing, incorporating the presence of infeasible requests. Our method achieves better performance and higher semantic alignment than state-of-the-art models, demonstrating its effectiveness in handling complex instruction challenges while preserving image integrity.
inherently infeasible contradictory, often
method, we constructed
Introduction
In recent years, the growing demand for visual content has made image editing essential across various fields. With advancements in technology, text-guided image editing has emerged as a powerful tool, enabling to manipulate images using natural language instructions 4 16 42 122
This innovation has streamlined the editing process, enabling users to perform sophisticated edits. Among these advancements, diffusion-based models have particularly excelled in image generation [ 38
39
42 48). However, models relying on simple text encoders such as CLIP (35] struggle to achieve user-intended fine-grained edits. These difficulties become more apparent when the editing prompt involves multi-step instructions with intricate details.
30 48
45
3 and editing tasks [18 8 12
To address this limitation, recent research has introduced two notable improvements in model design. First, the CLIP-like text encoder has been replaced by Multimodal Large Language Models (MLLMs) [6 [O]. These models effectively parse user instructions and interpret textual prompts, improving the capabilities of natural language understanding. Second, regions requiring editing within the image identified and modified using various methods, such cross-attention maps and segmentation models, to align each edit prompt with its corresponding regions [T] 25). Although the region- based image editing model [I]] shows more effective results on sulti-instruction tasks than other state-of-the- art methods, its attention maps often fail to consistently align with intended editing regions. This misalignment is especially pronounced when modifications involve spatial relationships or regions not directly associated with primary instruction keywords.
Work done during an internship Samsung Semiconductor, USA.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

eingut Imagez
Tar the 150
b
Change the mibee into stufled
Turn frikee color blue.
Farthermone addo blaydl tree
[MAS4]
Figure 1:
Three scenarios demonstrate how our method handles context-aware multi-instruction editing across various combinations of feasible and infeasible prompts. By leveraging [MASK] and [NEC] specialized tokens, it accurately identifies executable instructions.
These limitations become evident in multi-instruction scenarios containing challenging instructions that cannot be directly applied to the current image. Such instructions may request alterations to non-existent objects, logically inconsistent modifications, or edits that are incompatible with the image's content. Parsing and interpreting such inputs makes editing systems impractical, introducing suboptimal edits or even unrealistic. incoherent images. Additionally, relying on pretrained Large Language Models [2] to parse reorganize these instructions introduces further complexity in the editing pipeline and increases the potential for erross at intermediate steps. Any misinterpretation or bias in LLM output may propagate downstream, leading to incorrect region selection over-editing.
Despite the growing research interest comprehensive image editing, most existing methods overlook instruction executability, often leading to over-edited results. Our proposed approach addresses these concerns by explicitly assessing the executability of the instruction throughout the editing process. Building on pioneering research in this domain, we leverage the MLLM to jointly interpret both text instructions and images, then we extend its capabilities to enable image editing with context awareness. Here, context refers to the model's ability to interpret the relevance of various instructions within given image, allowing it to focus on applicable regions while ignoring irrelevant areas. A key feature of CAMILA is the use of specialized tokens and broadcast mechanism. Our model assigns [MASK] tokens to editable regions and [NEG] tokens to suppress irrelevant edits. The following broadcasting module then consistently aligns token assignments with user prompts. Overall, our context-aware pipeline helps to validate the coherence of instructions, resulting in improved performance across all image editing scenarios, including non-executable prompts.
To properly evaluate our approach, we extend the conventional single- and ioltiiinsstruction image editing tasks by introducing the possibility of non-executable prompts. This results in new evaluation scenarios: context-Awart Image Editing that evaluate how the model handles the number of instruc- tions and the presence of infeasible requests within the same sequence. We compare against several state-of-theeart baselines, observing substantial improvements in editing accuracy, particularly L1 and I.2 distances, well as enhanced performance on CLIP and DINO scores, with human preference-base based evaluation also indicating strong performance.
method
Our main contributions to this work are follows:
We introduce context-aware image editing model that precisely identifies prompt executability and corresponding editing regions, allowing user-aligned and consistent modifications. We propose new task setting: Context-Aware Image Editing. New datasets are created to evaluate model behavior and context-awarenes, in challenging scenarios.
Our model demonstrates significant improvements over existing methods in varying evaluation scenarios, achieving lower pixel-level errors and higher semantic alignment, while also showing qualitative superiority in effectively handling complex instructions.
Note that we formally define 'non-executable instruction' as any request that cannot be executed given the visual constraints or inherent semantics of the image. Our source code is available at https://gi tttps://github... thub. com/hk hk wwpookerrrrallle
2 Related Works
Multimodal Large Language Models. Multimodal Large Language Models (MLLMs) 24
27
41 [1 26 integrate multiple modalities, such as images and text. Recent MILMs have advanced to handle complex tasks such as referring visual grounding [50] 23 43), which aims to distinguish specific objects based context. Additionally, MLLMs have been applied to image editing task [16]

---

[Step]]
Taken Decoder
Diffusion
Model
Xena
[Step2] Diffusion model
Ring
Hiee aquirnel be tcokang ot the vese. and change the color ware yellow
Token Broadcaster
Tranablee
Diffusion model Text EncoderE
Frezon
The architecture of CAMILA begins by jointly processing the image Ting and text instructions Tia using an MILM. Output tokens are classified as either [MASK] or [NEG], indicating regions to modify leave unchanged. These tokens aligned with the text embeddings using the Token Broadcaster, and the final binary mask is generated by the Token Decoder. The mask is then applied in diffusion model to produce the edited image.
Figure 2:
10].
For instance, SmartEdit [66 improves instruction comprehension with bidirectional interactions between image and text, while MGIE [00] jointly trains an MILMM and diffusion model to guide editing tasks with visual-aware instructions. However, these models often lack context-awareness and fail to distinguish between relevant and irrelevant prompts. We thus break new ground by being the first incorporate context-aware MLLM specially for image editing. Unlike prior research, we do not limit our scope to single instruction tasks, enabling our model to handle both multi and context-aware instructions.
Image Editing by Diffusion Model. Diffusion models have become prominent image editing B7
[ 30 29
49). While text-guided image editing enable basic modifications, instruction-based image editing offers more nuanced control by interpreting complex, user-directed commands via natural language. InstructPix2Pix 4 introduced dataset combining GPT-33-geerraeed texts 5 and Prompt2Prompt-base images 12), which powers natural language-guiin gudddee editing. MGIE 10 utilizes an MLLM with visual-aware instructions for editing, and Fol ([]] uses cross- attention maps for multi-instruction scenarios. However, these methods struggle with ambiguous or incorrect instructions, they lack mechanisms interpret prompt feasibility. This limitation often leads to unintended modifications when the model encounters unclear instructions.
18
g
12
42
48
3 Preliminary
We briefly introduce InstructPix2Pix (IP2P) [4, standard framework for instruction- guiied image editing and its cross-attention mechanism. This overview serves as the background for our work.
3.1 InstructPix2Pix
IP2P is built upon Stable Diffusion 38 to modify images based on textual instructions. In this framework, conditioning on both input image and text instructions is necessary for guiding diffusion network to produce editing results aligned with user instruction. The input image Zimg is first encoded into a latent vector by the encoder &. At each time step I, the noisy latent vector 2, is progressively denoised by the score network. Then, the denoised latent vector is decoded into the output image.
To achieve conditional generation, diffusion models often employ classifier-free guidance [14, which eliminates the need for an external classifier. In their score network, two conditioning factors are introduced for use during inference: the image conditioning e₁ and the instruction conditioning Cj and eT are the encoded outputs from the image encoder & and the text encoder ε,, respectively. The final score estimation (g Cr, cr) computed as follows:
Eo(2,, C1, CT) = co 22, 2,00 + 81 (co(ze, C1, 2)
21)
ST (co Cr, CT) - co(2t, cr, 2),
In this equation, ez, 2, Ø) represents the base score prediction without any conditioning applied. The second term modulates the score with image conditioning çr, where 8, modulates how much the model preserves the characteristics the input image. Similarly, the last term incorporates text conditioning сT, with sT controls the degree of adherence to the edit instruction provided.

---

3.2 Cross Attention in Stable Diffusion
IP2P employs cross-attentin network modulation within the denoising U-Net architecture of the Stable Diffusion network. A key component is the tross-attention layer, which generates attention maps
E Rexrxm where is the spatial size and m is the number of text tokens. Several have shown that cross-attention maps with r r= 16 capture the most significant semantic information, compared to maps at other spatial resolutions. Thus, by modulating the computation of these aross-attention layers, it is possible to alter the image, as adjustments in the attention maps guide the model's focus on specific aspects of the text and image content
A
studies
12 6 I1
44
4 Methods
We build our framework upon pretrained MILM (27 and diffusion model 38 but key contribution lies in explicitly assessing the exccutability of instructions and leveraging specialized tokens to guide editing process in diffusion model. A key feature of our approach is its ability to validate the contextual coherence between instructions and the image, ensuring that only relevant edits are applied to designated regions while ignoring non-executable instructions. This context-aware mechanism distinguishes our method from existing MLL.--base approaches [1O] establishing executability filtering and context-awareness as new modeling objectives for MLLM-based image editing.
4.1 Architecture
The architecture of CAMILA is shown in Figure 2 Given an image Ting and text instructions Zia both inputs are jointly processed by the MILM F. The model is designed to encode and combine the visual and textual inputs, enabling it to capture the relationships between the textual instructions and corresponding regions in the image. Specifically, the image is processed through vision encoder, while the text instructions are tokenized and processed by language encoder. These representations are then combined into unified sequence within the MILM architecture, which interprets the joint context of the image and instructions. The output sequence O is generated from the image input Zimg and text input Zxg. Each output token O in = {O1, O2, On ). where n denotes the number of generated tokens, is classified either [MASK] or [NEG] token. The [MASK] tokens correspond to regions of the image that are to be modified based on the text instructions, while the [NEG] tokens indicate areas of the image that should remain unaffected.
By combining the visual and textual inputs, the MLLM is able to determine the relevance of each instruction to specific regions in the image, ensur- ing that only applicable edits are applied. This joint processing aligns each generated output token, either [MASK] or [NEG], with specific instructions. The [MASK] tokens are decoded, resulting in masks that accurately highlight the regions in the image that require modification according to the instructions. This targeted approach improves the precision of the editing process by ensuring that modifications are applied solely to relevant areas. The following con- tent will elaborate on how [MASK] and [NEG] tokens are aligned with instructions by Token Broadcaster and how [MASK] tokens are decoded into the actual editing mask by Token Decoder.
Token
Broadcaster
aramas
Sym
Projection Jaeer
W;
Phaiss model!
Figure 3: Architecture of the Token Broad- caster. It calculates similarity between MLLM output tokens and encoded text fea- tures, assigning each output token to the text embedding that best matches its correspond- ing semantic region.
4.2 Token Broadcaster and Token Decoder
The output sequence generated by the MLLM is processed by the Token Broadcaster module to ensure that the [MASK] and [NEG] tokens align accurately with the corre- sponding text embeddings. As illustrated Figure the text instructions Tist are embedded through the text encoder Σ of the diffusion model, resulting in a set of text embeddings cr. Using the diffusion model's text encoder & allows the model to ensure that the generated editing masks will align preciscly with er, facilitating integration into the diffusion model.
Token Broadcaster.

---

The MLLM output tokens O and the text embeddings cy reside in different latent spaces, so we need to align them into single space, Many studies 2 46] use cosine similarity- -bsed alignment to measure and organize relationships or similarities between different modalities. We project them into a shared space for alignment by applying trainable transformations Wo and Wr each, directly within the similarity matrix:
Sr = (O, (er,Wr)
(2)
[o) (l Wr)ll
where each element St, represents the cosine similarity score between the i-th transformed output token (O,Wo) and the j-th transformed text embedding (ery Wr), indicating their compatibility in the shared latent space.
To convert similarity scores into alignment probabilities, softmax is applied along each column of S. For each text embedding j. we then determine the index Oy that maximizes this probability:
exp(St.))
= arg max
Σx exp(Sk.))) Vj E (1,2,...,m) m..
(3)
where m denotes the length of text embeddings. This alignment process ensures that each text embedding maps to the output token best reflecting its semantic region within the image.
Token Decoder. The Token Decoder processes tokens differently based their type: only tokens labeled as [MASK] are converted into editing masks, while [NEG] tokens are directly replaced with black masks, indicating regions where no modification is applied. Designed as a two-layer Transformer decoder, the Token Decoder generates a set of binary masks MI, M2,...,, M,, each specifying regions of the image to be edited according the text instructions.
In the first decoder layer, we employ a cross- attssttnntnnoonn mechanism between image and text embeddings. This allows the model to extract contextually relevant features from the image that aligned with the text instructions. By attending to both modalities, the decoder effectively maps the semantic content of the text to corresponding regions in the image. The second decoder layer further refines this information by incorporating the [MASK] tokens into the key and value projections of the attention mechanism. This enables the model focus more precisely on the regions identified by each [MASK] token. After the second decoder layer, these intermediate masks are passed through sigmoid thresholding to produce the final 0-1 binary masks, denoted as Mj. Through this process, Token Decoder is able to generate the final binary mask Mr, with each mask serving as an editing mask for the corresponding MLLM output tokens O, defining the specific areas of the image be modified.
4.3 Diffusion Model
For each text embedding j. the alignment index a, determines the specific binary mask May to be used. The individual masks are concatenated to form unified binary mask M, which is then used in the diffusion model to guide the editing process:
M concat( (Ma,+ Mag...., Mam.)
This binary mask M ensures that each region is modified according to alignment indices from the Token Broadcaster, enabling precise, context-aware edits that reflect the intended modifications.
We modulate the cross-attention layers of the diffusion model, focusing specifically on the 16- sized cross-attention map, which captures the most semantically relevant features, as explained in The U-Net's cross-attention map A is modulated using the following equation:
Section 3.2
X' X M yo MM)
A' softmax
(5)
Vd
where d is the latent projection dimension, X' = QI.TKTTT and y KT. In this formulation, Qr. and K1,r represent the query and key projections in eg(zt, cr, CT), respectively, while Qr, and K1 are the query and key projections in Eg(2,, c,, Ø).
This modulation approach leverages A to align each text embedding precisely with the regions specified by the concatenated binary mask M, enhancing editing accuracy by concentrating on the relevant areas as dictated by the instructions. Then, the binary mask M selectively applies the text-conditioned attention map X' to editable regions and ) to unaltered areas, ensuring that only the specified areas are modified. By modulating the attention layer as in Equation (5). we generate the final output image following the score estimation formulated in Equation
5

---

4.4 Training Details
Training Loss Function. The training of our MLLM-based approach is optimized with four primary loss components, each designed to target specific aspect of model performance for accurate token classification, alignment, and mask generation. The total loss muain is formulated as follows:
mainn = Cce
Chroaccastt +
Aa + CBCE
(6)
where A1, A2, Aa, A are hyperparameters that balance the influence of each loss component.
The first element, token classification loss CCE applies cross-entropy (CH) loss to the MILMM output tokens. The second element, broadcasting aligmment loss Cbroadcast also utilizes CE loss to align MLLM output tokens with their respective text embeddings, ensuring precise correspondence between instructions and image regions, For mask quality, the mask dice loss Ldice measures overlap between predicted and ground truth masks, encouraging accurate spatial targeting. Lastly, the binary cross-entropy loss LACF enforces accuracy at the pixel level in the generated mask.
Trainable Parameters. To efficiently fine-tune the pre- pre-trained MLLM while preserving its learned knowledge, we adopt the Low-Rand Adaptation technique [15). In our training, we freeze the vision backbone and text encoder of the MILM, while the remaining parts of the model are fine-tuned. Additionally, the Token Broadcaster and Token Decoder are also trained, ensuring that the model aligns the output tokens with the text instructions and generates accurate masks for the diffusion model. Training is more efficient since only the MLLM and lightweight modules are updated, unlike other methods that jointly fine-tune both MLLM and diffusion model. All other training details are provided in Section A
4.5 Surrogate Module Training for Enhanced Masking
To further improve the quality of the binary mask M provided to the diffusion model, we conduct additional training beyond the initial MLLM training. Through empirical analysis, we found that certain outputs misalign with the description of the goal image. To better align the generated image with the intended modifications, we consider it useful to focus on improving CLIP-T score, which measures the similarity between the global description and the generated image. By optimizing the model for a higher CLIP-T score, we aim to generate higher quality binary masks, which lead to improved quality in the final output image.
However, due to the inherent complexity and the large number of steps involved in the forward pass of the diffusion model, directly backpropagating the loss from the final output image through the diffusion model to the MILM is infeasible. To address this limitation, we develop a lightweight surrogate module that approximates the CLIP-T score based on the input image Ting. the edit instruction Fix- and the binary masks M. Designed as a single-layer transformer, the surrogate module offers streamlined alternative to the complex, multi-step diffusion model. It is trained using a mean squared error (MSE) loss between the actual CLIP-T score and the predicted CLIP-T score. During this training phase, all other parts of the model are kept frozen, and only the surrogate module is updated. The overall loss function for training the surrogate module is formulated as:
Laurogate = E (CLIP-Toutpu CLIP-T CLIP-Tsunogatee
where CLIP-T and CLIP-T denote the actual CLIP-T score of the target output and the predicted score, respectively. This approach ensures that the surrogate module learns to accurately estimate the CLIP-T score without requiring multi-step backpropagation of the diffusson model.
Refining Mask Generation via Surrogate Module. Once the surrogate module is fully trained, we use estimated values to fine-tune the MLLM, Token Broadcaster, and Token Decoder. In this stage, the surrogate module is kept frozen, and the focus is on improving mask generation to maximize the predicted CLIP-T score. The objective to modify the MLLM's 's outputs to generate binary masks with higher CLIP-T score when processed by the diffusion model.
During training, the loss function is augmented include both Lmain as well the MSE loss between the predicted CLIP-T score and the oracle CLIP-T score. The updated loss Lupdated is defined as:
Lupdattd = Emain + > CMSE:
(8)
where LMSE is the MSE loss between the predicted and oracle CLIP-T score, and As is a hyperparam- eter controlling the weight of the CLIP-T score loss.

---

Table 1: Quantitative comparison across multi-instruction and context-aware instruction tasks. Our model demonstrates overall superior performance, especially excelling in the context-aware instruction task. This highlights our method's superb capability to handle context-aware instructions with high precision, applying edits that closely align with the intended modifications without over-
editing. Bold and underlining indicates the best and the second-best performance for each metric.
Multi Instruction
Context-Aware Instruction
Method
L14
L24
CLIP-I
DINO
CLIP-TT
L1
L24
CLIP-It
DINOT
CLIP-TT
IP2P
0.1402
0.0526
0.8327
0.7122
0.2977
0.1460
0.0514
0.7975
0.6429
0.2715
MGIE 110
0.1639
0.0777
0.8205
0.6723
0.2787
0.1592
0.0750
0.8090
0.6519
0.2637
SmartEdit 16
0.1295
0.0573
0.8630
0.7516
0.2971
0.1111
0.0495
0.8739
0.7726
0.2824
Fol 11
0.1054
0.0385
0.8811
0.8096
0.2941
0.0891
0.0284
0.8895
0.8190
0.2888
CAMILA (ours)
0.0945
0.0366
0.8980
0.8392
0.2984
0.0661
0.0222
0.9296
0.8932
0.3006
5
Evaluation
5.1 Task Categorization
For comprehensive assessment, we evaluate our method both single-instruction tasks aligned with standard benchmarks, and multiiinstruction image editing tasks that require multiple edit turns in single sequence. In single-instruction scenario, single directive is tested either in single-turn or multi-turn setting, whereas multi- multi-instruction tasks involve multiple directives that must be applied simultaneously. We further divide multi-instruction tasks into two types: Multi-instruction Image Editing, which includes only applicable instructions, and Context-Aware Instruction Image Editing, which includes mix of applicable and non-appliaable instructions.
5.2 Evaluation Settings
Datasets:
For evaluating single instruction tasks, we use the MagicBrush [47] dataset, which covers both single-turn and multi-turn scenarios as detailed in Section 6 along with the EMU [40] dataset. However, the literature lacks dedicated benchmark datasets for multi-instruction or context-aware instruction editing. To address this gap, we introduce two new tasks and curate corresponding datasets: Multi-in inlliiiisttuution Image Editing and Context-Aware Instruction Image Editing as detailed in Section [5.3] In Multi iulti-instruction Image Editing, we concatenate applicable instructions from MagicBrush' multi-turn dataset into a single instruction sequence. In the Context-Aware Instruction Image Editing task,
we introduce non-applicable instructions generated with ChatGPPT-V((sion)) 32 alongside images. More details on data creation are detailed in Section C
Metrics:
To evaluate our proposed method, we employ a diverse set of metrics, including L1/L2, CLIP-I, DINO, CLIP-T, CLIP-dir, and PickScore [22]. Detailed descriptions of these metries are provided in Section B
Baselines: We compare CAMILA with five different state-of-the image editing methods: IP2P 4 EMILIE 17), MGIE [10], SmartEdit (SE) [16), and Fol
5.3 Main Results
Quantitative Result. As illustrated Table CAMILA demonstrates state-of-the-art results across both multi-instruction and context-aware instruction tasks, particularly excelling in metrics such as CLIP-I, CLIP-T, and DINO similarity, and overall distance metrics (L1 and L2). This indicates that model aligns closely with human perception in maintaining fidelity to edited images.
The existing methods exhibit notable limitations. MGIE, which relies on a summarization approach to compress instructions, proves to be vulnerable to non-applicable instructions, leading to potential inaccuracies in execution. While SmartEdit shows improved understanding due to the integration of MLLMs, it suffers from lack of robustness by feeding all instructions into the diffusion model simultaneously, which can lead to oversights in handling complex editing requests. Additionally, Fol struggles with imprecise attention maps, which reduces its performance below CAMILA though the most competitive baseline. stark contrast, our approach effectively manages multi- instruction editing tasks, demonstrating superior capability in processing context-aware instructions.
7

---

Input Image
IP2P 4
MGIR LO
Smart
LG
Fol I1
CAMILA (Ours)
MAAKK]
[MASK]
(a) Edit instruction:
"Trrn the brown horse into pink unicorn, and put river nearby."
[MAKK]
MASKK]
(b) Edit instruction:
Give the man a cowboy hat, and put a witch hat in the woman.
MEC]
[MASK]
(c) Edit instruction: "Put flower vase the table next to the chair. On top of have be wearing a yellow sweatshirt."
NEG]
[MASK]
(d) instruction: "Remove stack of pancakes from plate and put the food on a plate."
Figure 4: Qualitative comparisons: Fol needs to extract keywords from each instruction using pretrained GPT model before running the model. Furthermore, due to inaccuracies in the attention map of diffusion model, Fol often fails to make precise modifications. instructions, CAMILA accurately identifies applicable instructions by generating [MASK] and [NEG] tokens from MILM. We present the decoded mask results for each instruction of the [MASK] token.
the case of context-ane
This proficiency enables model to execute edits with high precision, aligning closely with the intended modifications while minimizing the risk of over-editing. Overall, our results underscore the advantages of our method in navigating the complexities inherent multi-instructiin tasks.
In addition to distance-based metries and similarity- bassd metrics, we further evaluate human perceptual alignment using the PickScore metric across two editing tasks: Multi- nulti-insttucctin and Context-Awarr image editing. CAMILA outperforms
strongest baseline, Fol, by 18. 1% and 24.0% in each setting, respectively. The advantage is more evident in the Context-Aware setting, demonstrating our model's ability effectively filter non-applicable instructions. More detailed results presented in Section E.I
Qualitative Result. As shown in Figure 4 we present qualitative results and observe the following: All models, except for Fol, frequently execute only single instruction when multiple instructions are provided. In (a), while Fol successfully performs the first instruction, it fails to generate an accurate attention map for the keyword 'river', resulting in the incomplete application of the second instruction. Similarly, in (b), the lack of fine- fine-grained attention map results the hat being placed incorrectly. The remaining models predominantly execute only one instruction and demonstrate tendency toward over-editing; for instance, IP2P and MGIE alter the background color in (a), and SmartEdit generates an additional unicorn.
In (c) and (d), most models exhibit erroncous edits in response non-applicable instructions. In (c). despite the absence of a chair or table in the input image, the models add incorrect floral elements or table. Similarly, in (d), although the input image does contain a pancake, erroneously appears due to the removal instruction, illustrating compared Fol, our model demonstrates greater precision in mask extraction for areas requiring modification, enabling more refined edits. Furthermore, CAMILA supports both localized object edits and global transformations. The [MASK] tokens dynamically adjust their spatial coverage based on each instruction, enabling edits that range from small regions to full-scene editings. Through the use of [MASK] and [NEG] tokens, our proposed model facilitates robust, context-aware image editing. Further qualitative results are provided in Section E.4
inability to correctly handle the instruction. Especially

---

Table 2: Quantitative comparison on EMU dataset. Achieving the highest CLIP-dir score in the Context- Aware task shows that our model effectively distinguishes non-executable instruc-
Table 3: Comparison of results before and after addi- tional training with the surrogate module. We apply the surrogate module to improve the CLIP-T score, which also enhances L1/1. losses, as well as CLIP-I and DINO scores.
Task
Config
L11
L22
CLIP-IT DINOF CLIP-TI
tions.
Single
before
0.0602
0.0194
0.9367
0.9067
0.3020
Task Method
Single-inst
Context-Aware CLIP-dir
Single Inst.
-Tum
after
0.0596
0.0191
0.9375
0,9069
0.3022
CLIP-TT CLIP-dirt
CLIP-IT
Multi
before
0.0931
0.0339
0.8969
0.8357
0.3011
IP2P 0.2616
0.075
0.2446
0.064
-Tum
after
0.0782
0.0268
0.9127
0.8659
0.3019
MGIE
0,2680
0.082
0,2543
0.066
before
0.0957
0.0372
0.8961
0.8329
0.2975
SE 0,2680
0.094
0.067
Multi
Multi
after
0.0945
0.0366
0.8980
0.8392
0.2984
Fol 0.2673
0.068
0.2651
0.054
Inst.
Context
before
0.0673
0.0228
0.9284
0.8910
0.3002
ours
0.2687
0.092
0.092
-Aware
after
0.0661
0.0222
0.9296
0.8932
0.3006
6 Ablation Study
Robustness of CAMILA. In our framework, distinguishing applicable from non-applicable instruc- tions is critical to prevent unintended edits. Each input may contain multiple instructions, which are classified by the MLLM into [MASK] and [NEG] tokens. On the Context-Aware Image Editing dataset, our model achieves a token classification accuracy of 90.21 %, highlighting the robustness of CAMILA in filtering non-applicable instructions. Furthermore, we evaluate the alignment of generated masks with ground-truth edited regions using standard segmentation metrics. For applicable instructions, classified as [MASK] token, our model achieves IoU of 0.3819 and Dice score of 0.4986. As described in Equation (6). our model is trained with multiple loss objectives, not solely for segmentation accuracy. The generated masks are designed as high-level guidance, reflecting our focus on instruction fidelity and plausibility rather than strict spatial matching.
Evaluation on Instruction-Folloving Accuracy. We evaluate our method on the EMU dataset for both singleiinsruccton and context-aware instruction tasks. Since the EMU dataset does not provide ground-truth target images, we utilize CLIP-T and CLIP-dir as evaluation metric. CLIP-dir measures how accurately the generated image aligns with the intended semantic direction of the instructions. As shown in Table 2 CAMILA achieves the highest CLIP--dr score in the context-aware instruction task, demonstrating its effectiveness in handling non-exccutable instructions.
Table 4: Quantitative comparison on single instruction tasks, CAMILA excels in single instruc- tion tasks by generating precise masks that accurately target modification areas.
Single-turn Instruction
Multi-turn Instruction
Method
L11
L21
CLIP-1
DINO
CLIP-T
L1J
L21
CLIP-11
DINOt
CLIP-TY
IP2P
0.1129
0.0373
0.8540
0.7423
0.2918
0.1538
0.0575
0.8103
0.6511
0.2866
EMILIE 117
0.1129
0.0373
0.8540
0.7423
0.2918
0.1268
0.0509
0.8557
0.7591
0.2916
MGIE 10]
0.0931
0.0383
0.8853
0.8088
0.2935
0.1312
0.0574
0.8571
0.7507
0.3013
SmartEdit [16]
0,0895
0.0353
0.9030
0.8308
0.3024
0.1333
0.0575
0.8567
0.7421
0.3021
Fol [T]
0.0699
0.0206
0.9207
0,8779
0.2980
0.1084
0.0379
0.8681
0.7838
0.2935
CAMILA (ours)
0.0596
0.0191
0.9375
0.9069
0.3022
0.0782
0.0268
0.9127
0.8659
0.3019
Single-Instruction Task Performance. CAMILA, optimized for multi-instruution tasks, also performs strongly on single-instruction tasks, as shown in Table 4 CAMILA achieves strong performance across most evaluation metrics. In contrast, SmartEdit shows a tendency toward over- editing. This reflects CAMILA's 's balanced approach, minimizing over-editing while maintaining high fidelity. As shown in Figure 5 most models exhibit over-editing issues. Although Fol is designed to minimize over-editing, it encounters specific issues as follows: inaccurate attention maps in (a) prevent precise modifications to the 'angry birds' object, while in (b), additional frosting is incorrectly applied to cupcakes. These cases show that CAMILA achieves accurate edits without over-editing.
Impact of Surrogate Module Training. We compare the results on the multi-instruction tasks and single-instruction tasks before and after additional surrogate module training. As shown in Table 3 we demonstrate that mask generation through this surrogate module improves model performance. Interestingly, the improvement is not just for CLIP-T but also for the other metries.

---

Inpux
IP2P
SmartEdit
CAMILA
Table 5: Large-scale baseline compar- ison on context-aware instruction task. CAMILA outperforms larger diffusion-basee models by generating contextually aligned and precise masks, highlighting that its context-aware design enhances editing quality without relying on increased model capacity.
(a) Edit instruction:
"Repface the angry birds with flowers"
(b) "Add a stranberry on top of the cupoake with frosting" "
Method
L14
L24
CLIP-If DINOT CLIP-TT
Figure 5: Qualitative comparisons for single in- struction task. CAMILA demonstrates success- ful editing even the single instruction task.
StepI--Edit 0.0830 0.0329
0.8096
0.8892
0.2984 0.3006
CAMILA 0,0661 0.0222
0.9296
0.8932
Comparison with Large-Scale Baseline We additionally compare CAMILA with recent large-scale model such as Step XX X-Edit (28), which use larger diffusion backbone (34) than Stable Diffusion [88.. As shown in Table [ CAMILA achieves better performance on the context-aware image editing dataset despite its smaller diffusion model size. This implies that the context-aware design of CAMILA effectively enhances editing fidelity without relying on large model capacity.
Additional ablation studies on the variation of the Token Decoder and the inference time comparison are provided in Section D. and Section D.2, respectively.
7 Conclusion
In this paper, we addressed the limitations of current text-guided image editing models, particularly their difficulty handling fine-grained edits, multi-instruction edits, and distinguishing between executable and non-executabbee instructions. Leveraging MLLMs, we generated specialized tokens (DMASK] and [NEG] and designed token broadcaster to ensure the validity of the contextual coherence between instructions and the image, that only relevant edits can be applied to the designated regions while ignoring non-executable instructions. For comprehensive evaluation, we created new datasets that can evaluate Context-Awan Image Editing task, where our approach achieves superior results across both qualitative and quantitative evaluations compared to state-of-the-ar solutions.
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF international conference on computer vision, pages 4432 4441, 2019.
[2] Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for eextto--mmage synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 2283-22222 00932293,722 2023.
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132 2022.
[4] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18492,18402, 2023.
[5] Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
[6] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen- Or. Attend-and-excite: Attention-bassd semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42 2023.
[7] Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 26573-26583, 2024.
10

---

[8] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022.
[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards ceneral-purpose Vision-language models with instruction tuning, 2023.
[10] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang. and Zhe Gan. Guid- ing inanuctioo-bbsell image editing via multimodal large language models. In The Twelfth International Conference on Learning Representations, 2024.
[11] Qin Guo and Tianwei Lin. Foeus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6986-6996, 2024.
[12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-pin to-mmppptppttt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. CoRR, abs/2006. 1239, 2020.
[14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.0960 2021.
[16] Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction- hased image editing with multimodal large language models. arXiv preprint urXiv:2312.06739 2023.
[17] KJ Joseph, Prateksha Udhayanan, Tripti Shukla, Aishwarya Agarwal, Srikrishna Karanam, Koustava Goswami, and Balaji Vasan Srinivasan. Iterative multi-granular image editing using diffusion models. In Proceedings of the IEFF/CVE Winter Conference Applications of Computer Vision, pages 8107-8116, 16, 2024.
[18] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEFSCVF Conference on Computer Vision and Patlern Recognition, pages 6007-6017, 2023.
[19] Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36, 2024.
[20] Byoungjip Kim, Sungik Choi, Dasol Hwang, Moontae Lee, and Honglak Lee. Transferring pre-trained multimodal representations with modal similarity matching. Advances in Neural Information Processing Systems, 04:30826-30339 2022.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026. 2023.
[22] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36666--666666 -66633, 2023.
[23] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. Proceedings of the IEEF/CVÉ Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.

---

[24] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language ainuage-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[25] Shanglin Li, Bohan Zeng, Yutang Feng, Sicheng Gao, Xiuhui Liu, Jiaming Liu, Lin Li, Xu Tang, Yao Hu, Jianzhuang Liu, al. Zone: Zero-shot instruction-uuided local editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6254 6263, 2024.
[26] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[28] Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Steplx-edit; A practical framework for general image editing. arXiv preprint arXiv:2504. 17761, 2025.
[29] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.00 01073, 2021.
[30] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 2221 2021.
[31] OpenAL. Gpt-4 technical report, 2023.
[32] OpenAl. Gpt-4v(ision) system card, 2023.
[33] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071993 2023.
[34] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763 8763. PMLR, 2021.
[36] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji. Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 2024.
[37] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: a stylegan encoder for amage-to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287-2296, 2021.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[39] Robin Rombach, Andreas Blattmann, and Bjöm Ommer. Text-guided synthesis of artistic images with retrieval- auggented diffusion models. arXiv preprint arXiv:2207.13038, 2022.
[40] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 8871-8879, 2024.
12

---

[41] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican. al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
[42] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-ines tomge translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[43] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858-3869, 2024.
[44] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-488-22337, 2023.
[45] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong. Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
[46] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang. Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages (995-7004, 2021.
[47] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guide image editing. Advances in Neural Information Processing Systems, 36, 2024.
[48] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-imaag diffusion models. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[49] Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang. Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024.
Recognition, pages
[50] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-moda 1lms. arXiv preprint arXiv:2307.085881 2023.
[51] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing usioo-language understanding with advanced large language models, arXiv preprint a304v:2304. 10592, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. The abstract outlines the high-level scope. Also, the introduction section clearly states the main contributions, accurately reflecting the paper's content and focus.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations of our method, especially in failure scenarios, are further discussed Section E.3
further analyze the computational efficiency of our model in
Section D.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not present theoretical assumptions or provide formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all the training details to reproduce the main experimental results in various sections including Section 4 Section (..2 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We describe how we generated the new dataset in Section C.1 The code and
dataset will be made publicly available after the paper is accepted for publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: provide the training details to reproduce the main experimental results in various sections including Section Section 5.2 and Section A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars as our evaluation relies on deterministic. metrics that yield consistent values across runs. Since the outputs are fixed given the same inputs, statistical variance is not applicable in this setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The inference time for image generation is reported in Section D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We address this in Section F for further details.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Although we fine-tune pretrained LLM, method does not introduce additional misuse risks beyond those associated with the base model. No safeguards were added beyond those provided by the original pretrained model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We address this in Section for further details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not release any new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve human subjects, and therefore does not require the details mentioned in this point.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve human subjects and does not require an IRB approval or equivalent.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method of CAMILA was developed without the involvement of LLMs.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) hrct::::/eerrrpp..cc//oofeeeeeee cc///oofeeeennee//222////.. for what should or should not be described.
20


Reviews:
[{"review_id": "vTOzPBL1wh", "reviewer": "Reviewer_T2mz", "summary": "This paper introduces CAMILA, a new approach for masked image modeling (MIM) that enhances global context understanding and local spatial coherence in visual representations. The key idea is to introduce a Context-Aware Aggregation (CAA) module, which improves the reconstruction of masked tokens by leveraging both global context and local neighborhood interactions.\n\nCAMILA is implemented on top of a ViT encoder and uses a contextual decoder which jointly refines visible and masked tokens. It incorporates local aggregation layers that propagate information across both spatial and channel dimensions. The method is tested on standard benchmarks (ImageNet-1K, ADE20K, COCO), and shows performance improvements over strong MIM baselines such as MAE and SimMIM.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper presents a simple and flexible design for CAMILA, which is modular and can be easily integrated with different encoder-decoder architectures.\n\n* The authors provide thorough ablation studies that specifically analyze the effects of context-aware decoding and local aggregation. Visualizations of reconstruction quality and attention maps further substantiate the main claims.\n\nWeaknesses:\n\n* Limited Theoretical Justification: Despite strong empirical results, the paper lacks formal analysis or theoretical motivation for why context-aware aggregation improves generalization. Even an empirical investigation into receptive fields or token mixing patterns would help clarify this aspect.\n\n* Missing Token Broadcaster Ablation: The impact of the Token Broadcaster module remains unclear, as no metrics or comparisons with simple baseline methods are provided.\n\n* Uncertainty in Mask Accuracy: The predicted binary masks are generated using a separate token decoder, rather than utilizing the attention masks from UNet cross-attention layers. It is not demonstrated whether the token decoder actually improves mask prediction. An ablation study, potentially using the PIE-Bench dataset (which includes ground truth masks of edited and unchanged regions), would strengthen the evaluation.\n\nReference:\n[1] PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code, Ju et al., ICLR 2024", "was_revised": true, "final_justification": "I would like to thank the authors for their comprehensive rebuttal. The clarification regarding the Token Broadcaster through the ablation study has addressed my primary concerns. Nevertheless, I continue to believe that the paper would be significantly strengthened by a direct evaluation of the proposed mask's effectiveness. To this end, I strongly recommend assessing the mask accuracy, for which the PIE-Bench [1] dataset, with its provided mask regions, would be a highly suitable benchmark.", "rebuttal": "Thank you for your review. However, we respectfully believe that there may have been some misunderstandings or discrepancies between your comments and the content of our submission.\n- The summary mentions \"Context-Aware Aggregation (CAA)\" and \"masked token reconstruction,\" but our paper does not contain these terms or such methods. Similarly, benchmarks such as ImageNet-1K, ADE20K, and COCO are not used in our work.\n- Our method, CAMILA, is not an encoder-decoder architecture and does not involve any reconstruction-based tasks or visualizations, which are mentioned in your listed strengths.\n- Additionally, the citation at the end of the review (\"PnP Inversion: Boosting Diffusion-based Editing…\") is not related to our paper and does not appear in our references.\n\nGiven these points, we are concerned that the review may not reflect a full reading or an accurate understanding of our submission. We would greatly appreciate it if you could revisit the paper to reassess its contributions based on its actual content. \n\nThank you again for your time and effort. Below, we respond to the reviewer’s points.\n\n---\n\n### **[Weakness 1] Limited Theoretical Justification**\n\nWhile we agree that theoretical analysis is always valuable, CAMILA is primarily proposed as a practical solution for context-aware image editing. Regarding the mention of “receptive fields or token mixing patterns,” we are not sure about the specific context in which this was raised. We would be happy to provide further explanation during the discussion period if the reviewer could clarify this point.\n\n---\n\n### **[Weakness 2] Missing Token Broadcaster Ablation**\n\nWe acknowledge the reviewer’s concern regarding the lack of formal proof of Token Broadcaster optimality. To address this point quantitatively, we measure the accuracy of token alignment through our Token Broadcaster module. Our Token Broadcaster achieves a token alignment accuracy of 94.46\\% on our Context-Aware Image Editing dataset, which demonstrates the precision of our alignment strategy.\n\n---\n\n### **[Weakness 3] Uncertainty in Mask Accuracy**\n\nWhen the attention map is extracted from a UNet cross-attention layer of a diffusion model, as FoI did, it does not effectively capture contextual relationships and is insufficient for understanding non-executable content. The UNet cross-attention layer is based solely on the entire prompt input, so it lacks the ability to understand the overall context."}, {"review_id": "eSD6YFeT0J", "reviewer": "Reviewer_1Qf6", "summary": "The authors present CAMILA to tackle the major limitation of pre-existing text-guided diffusion-based image editors, which follow every single instruction blindly, leading to the generation of illegitimate images. This is due to 'non-executable instructions' present in the sequence of instructions in the provided text prompts. Context-awareness is important to generate legitimate image edits. CAMILA introduces a context-aware mechanism where it creates masks, [MASK], for the perfectly coherent text instructions for image editing and does not perform any operation when an infeasible request is present in the same sequence ([NEG]), thus allowing it to validate what part of the sequence is feasible or not before making the required edits in the image. The authors test this via constructed datasets for single and multi-instruction image editing prompts to simulate the occurrence of infeasible requests in a given text sequence. \n\nThe authors utilize LLaVA-7B, a Multimodal Large Language Model (MLLM) which takes the source image (ximg) and the text instructions (xtxt). They used a pretrained MLLM and diffusion model, and the major contribution lies in assessing whether the provided text instructions are meaningful or not, based on which the diffusion-based image editing takes place only at the regions for which [MASK] was generated. To fine-tune the pretrained MLLM, the authors use the Low-Rank Adaptation technique (LoRA [16]). The output sequence O is generated from ximg and xtxt, and each token {Oi} is either classified as [MASK], depicting regions to be edited, or [NEG] regions where no edits will be performed. A Token Broadcaster processes the output sequence O and correctly aligns to its corresponding text embeddings (cT) based on maximizing the cosine similarity score, where both Oi and cT are projected onto a single latent space (Equations 2 and 3). This ensures accurate semantic correspondence between the text instructions and image regions. \n\nA token decoder processes only the [MASK] tokens and converts them to editing masks and replaces [NEG] tokens with black masks. The Token Decoder is designed as a two-layer transformer decoder block, where the first decoder layer uses a cross-attention mechanism between image and text embeddings to extract contextually relevant features from the image aligned with the text instructions. The second decoder layer refines this information by incorporating [MASK] tokens into the key and value projections of the attention mechanism. These are then passed through a sigmoid to generate final binary masks, Mi. \n\nBased on the studies of ([2], [6], and [11]), the authors use 16x16 attention maps to capture the most significant semantic information. The equation (5) modulates the U-Net's cross-attention map and applies the appropriate edits in the image. Authors also use a surrogate module, a single-layer transformer, which is trained to enhance the generated masks. They focus on improving the CLIP-T score that measures the similarity between the global description and the generated image. The lightweight surrogate module approximates the CLIP-T score based on ximg, xtxt, and binary mask M. When the surrogate module is trained, all other parts of the models are frozen, and only the surrogate module is updated. \n\nCAMILA achieves state-of-the-art results in L1, L2, CLIP-I, CLIP-T, and DINO scores in both multi-instruction and context-aware instruction scenarios.", "questions": "1. Mask Precision\n\nThe failure cases (Appendix E.3) show that generated masks sometimes under- or over-segment the target areas. This is significant for small and thin objects.\n\nOne suggestion would be to check under what conditions the mask quality degrades (maybe try to find a certain threshold for object sizes). Additional direction could be predicting soft masks to capture the thin edges of the objects. \n\n2. Generalization beyond the synthetic benchmarks\n\nThe non-executable prompts are generated via GPT-4V, but this may not capture the full range of how humans give unclear or problematic instructions. \n\nCould you evaluate CAMILA on a user-collected set of random instructions to test the robustness? The user-collected set need not be large. This can be collected via a form circulated among your peers by providing them a sample image and receiving the prompts for image editing. It would be interesting to see what failure cases are encountered when instructions are semantically ambiguous rather than simply impossible. \n\nIf the results still hold up when tested on this human-generated dataset, I would be willing to change my borderline accept to an accept. \n\n3. The ablation of the MLLM choice\n\nAll the said experiments are performed using LLaVA-7B as the MLLM backbone. \n\nHow much significance does a smaller or larger MLLM have on the obtained metrics? A small ablation could provide insight into the choice of MLLM and the associated strategies.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe authors address the standing limitation of prior methods by explicitly tackling the executability of instructions in the image editing process. They explicitly handle the non-executable instructions, thus mitigating the \"always obey\" paradigm via the inclusion of context-awareness. CAMILA introduces a clever solution using special [MASK] and [NEG] tokens that act as gates to control image editing. This creates a new way to handle infeasible instructions in text prompts. The key innovation is that the system can evaluate whether an instruction makes sense in a given context and use that assessment to guide the image editing process. In my opinion, this particular innovation truly stands out and paves a new research direction for text-guided image editing.\n\nThe CAMILA pipeline carefully integrates MLLM, Token Broadcaster, and Token Decoder, thus allowing for high-quality image edits in multi-instruction and context-aware instruction scenarios. The authors created a lightweight module that estimates the CLIP-T score directly from the input image. This avoids having to run the full diffusion model, which involves many computational steps. Without this shortcut, it would be impossible to train the system end-to-end because the diffusion model is too complex to backpropagate through from the final image back to the multimodal language model.\n\nThe authors also curate new datasets for context-aware instructions and multi-instruction editing, filling an important gap in the field. These datasets will enable future studies on how to handle infeasible text prompts.\n\nThe paper is well-written and well-structured, with clear separation of sections, and the abstract engages the readers. The methods are clearly described, and the paper can be followed without many issues. The empirical validations, supported by the inclusion of various metrics, provide credibility for the proposed pipeline. The visual representations of the failure cases in the appendix highlight the limitations of the paper. \n\nWeakness\n\nThe paper heavily relies on the usage of multiple complex modules, such as MLLM, diffusion models, and multi-layer transformer blocks, which tend to increase the overall training and inference time. Only a small discussion regarding the runtime overhead is present in Appendix D.2 and Table 9. The binary masks generated (Failure cases Appendix E.3 and Figure 6) show that the model misestimates the object size or sometimes misses the finer details. \n\nAdditionally, the authors don't thoroughly justify their choice of two-layer transformers for the Token Decoder - they only compare against SAM. A deeper analysis would help readers understand this design decision. Moreover, the infeasible prompts are synthetically generated using GPT-4V; a thorough evaluation using human instructions should be explored to represent the diversity in natural human speech patterns and errors. \n\nAll the core experiments utilize LLaVA-7B, and it is unknown how the performance of the entire pipeline will fare when using smaller or larger MLLMs and how much impact it will have on image edits. \n\nThe core ideas (attention modules, mask-guided editing) draw inspiration from prior diffusion-based works. The main novelty lies in how the masks are utilized to establish the text-prompt feasibility and discard the infeasible instructions.", "was_revised": true, "final_justification": "Authors have thoroughly addressed my primary concerns. Their rebuttal provides clarifies the training and inference efficiency trade-offs. Authors justify the design choice of the two-layer Token Decoder by relating it to SAM’s proven architecture. The additional evaluation on a small human-generated dataset demonstrates reasonable robustness beyond synthetic prompts.", "rebuttal": "We thank the reviewer for the thoughtful feedback. Also, we are encouraged by the recognition of our core contribution, modeling instruction executability with our specialized tokens, as a promising direction for text-guided image editing. Below are our responses to each comment.\n\n---\n\n### **[Weakness 1] Training and Inference Efficiency**\n\nThe concern about the training and inference overhead of CAMILA raises an important point. While CAMILA integrates multiple components, including an MLLM, Token Decoder, Token Broadcaster, and diffusion model, we would like to clarify the implications of this design in terms of inference runtime and training cost.\n\nAs shown in Appendix D.2 and Table 9, CAMILA exhibits a higher total inference time than MGIE and SmartEdit. However, it achieves the lowest MLLM inference time among all models, reflecting the lightweight design of our Token Decoder and Token Broadcaster modules compared to the more complex components used in MGIE and SmartEdit. While CAMILA and FoI show increased diffusion time due to attention modulation on the IP2P backbone, CAMILA provides the unique advantage of supporting single-shot, multi-instruction editing without preprocessing (e.g., keyword extraction or sequential handling). This leads to a more integrated and streamlined editing pipeline, despite slightly higher latency.\n\nFrom a training time perspective, CAMILA is significantly more efficient than other MLLM-based methods. Unlike MGIE and SmartEdit, which finetune both the MLLM and the diffusion model jointly, our approach updates only the MLLM, Token Broadcaster, and Token Decoder. This design substantially reduces training cost and memory overhead. Although FoI is a training-free method and thus not directly comparable, among MLLM-based editing frameworks, CAMILA offers a more compute-efficient training strategy while achieving competitive or superior editing performance.\n\n---\n\n### **[Weakness 2] Design Choice of Token Decoder**\n\nWe thank the reviewer for raising the question regarding our choice to use a two-layer Transformer in the Token Decoder. While we do not currently provide an ablation study on the number of layers, our design decision was guided by the architecture of the Segment Anything Model (SAM) [1]. SAM’s mask decoder is composed of two attention blocks, each consisting of self-attention, cross-attention, MLP, and cross-attention layers. To preserve comparable model capacity, we adopted a comparable two-block configuration in our Token Decoder.\n\nUnlike SAM, which does not take edit instructions as input, our decoder explicitly conditions on text instructions through a cross-attention mechanism. This design enables our model to produce spatially grounded editing masks that are semantically aligned with the given instructions. As shown in Appendix D.1, our model outperforms SAM in instruction-based editing tasks, demonstrating the effectiveness of this architecture.\n\nWe agree that further investigation into architectural choices, such as the number of decoder layers, would be a valuable direction for future work.\n\n---\n\n### **[Weakness 3] Main Novelty**\n\nWe acknowledge the reviewer’s concern regarding the novelty of our approach. While CAMILA builds on existing components such as pretrained MLLMs and diffusion models, its primary contribution lies in *identifying and tackling a previously underexplored challenge in text-guided image editing*, determining whether instructions are actually executable in complex, multi-instruction scenarios. To this end, we introduce [MASK]/[NEG] tokens along with the Token Decoder and Token Broadcaster modules to explicitly represent instruction executability and align text with image regions, enabling context-aware editing.\n\n---\n\n### **[Question 1] Mask Precision**\n\nWe thank the reviewer for pointing out the failure cases in Appendix E.3, particularly regarding under- or over-segmentation of small and thin objects. We agree this is an important challenge and believe the issue arises from two key factors:\n\n1. **Limitations of the ground-truth masks in the MagicBrush dataset**: We utilized the MagicBrush dataset because, to the best of our knowledge, it is currently the only publicly available dataset that supports multi-instruction editing with editing masks. While we are unable to directly share examples of the masks due to the review policy, we encourage the reviewer to inspect the dataset directly, as it is publicly accessible, and observe that many of the masks are relatively coarse. Since our model is trained on these masks, it naturally inherits their limitations, particularly in handling very small or thin objects, where accurate mask supervision is lacking.\n\n2. **Resolution bottleneck introduced by Stable Diffusion’s attention modulation**: Although the model inputs and predicted masks are in 512×512 resolution, the Stable Diffusion backbone internally downsamples masks to 16×16 resolution during attention modulation. This reduces the model’s capacity to precisely localize or attend to structures smaller than approximately 32×32 pixels in the original image. Consequently, extremely small or thin regions are often not sufficiently represented, which can lead to segmentation errors.\n\nThese two factors jointly contribute to the observed failure cases. We believe that future improvements could be achieved by incorporating higher-quality, fine-grained masks and adopting diffusion models with higher spatial resolution conditioning. We consider this a promising direction for future work.\n\n---\n\n### **[Question 2] Generalization beyond the Synthetic Benchmarks**\n\nWe appreciate the reviewer’s concern regarding the generalization beyond the synthetic benchmarks and the suggestion to evaluate CAMILA on a user-collected set. In response, we conducted an additional experiment using human-generated non-executable instructions.\n\nSpecifically, we randomly sampled 8 images from the MagicBrush test set, and we asked our peers to generate 5 non-executable instructions per image, resulting in a total of 40 human-generated instances. We mixed these non-applicable instructions with the original applicable ones from MagicBrush in random order. Human-generated non-executable instructions were intentionally designed to resemble realistic user mistakes or confusion. For example, requesting to “remove the zebra” when the image contains a black horse, or asking to “change the red green tree to yellow,” reflecting syntactically or semantically ambiguous phrasing.\n\nUsing this user-collected dataset, we evaluated CAMILA’s token classification performance. The results show that CAMILA effectively filters out non-applicable instructions, achieving a token classification accuracy of 70\\% (28/40) on this human-generated dataset. While CAMILA achieved a token classification accuracy of 90.21\\% on the original synthetic benchmark, its performance on the user-collected dataset was lower. However, this is expected, as the model was not trained on any human-generated non-applicable instructions, and the user-collected evaluation dataset was relatively small. Importantly, many of the user-generated non-executable instructions were more nuanced and semantically ambiguous than those in the synthetic benchmark. Despite this, the results still indicate a reasonable level of robustness in CAMILA’s ability to identify and filter real-world ambiguous instructions. We believe that incorporating such human-generated data into the training dataset could further improve performance, and we sincerely thank the reviewer for this valuable and constructive suggestion.\n\n---\n\n### **[Question 3] The Ablation of the MLLM Choice**\nWe appreciate the reviewer’s insightful suggestion regarding the choice of MLLM. We selected LLaVA-7B as the backbone for our experiments primarily to ensure fair comparison with MLLM-based image editing models such as MGIE, which also solely rely on LLaVA-7B. Currently, due to limited computational resources, we have not yet conducted experiments using larger MLLM models (e.g., LLaVA-13B). If feasible within the discussion period, we will report preliminary results. Otherwise, we plan to include such experiments as part of future ablation studies.\n\nThat said, we would like to clarify the role of the MLLM within our framework. Its primary function is to generate [MASK] and [NEG] tokens, which guide the editing process for the diffusion model. Since our pipeline does not fine-tune the diffusion model and relies on the accuracy of these token predictions, the effectiveness of CAMILA largely depends on how well the MLLM generates [MASK] /[NEG] tokens. In our current experiments, CAMILA achieves a token classification accuracy of 90.21\\%, indicating that the model already performs robustly in identifying infeasible instructions. While it remains an open question whether scaling the MLLM would yield further gains, our results suggest that the current configuration provides a strong foundation.\n\n---\n\n### **References**\n\n[1] Kirillov, Alexander, et al. \"Segment anything.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023."}, {"review_id": "uv5zsck6Kf", "reviewer": "Reviewer_QraL", "summary": "This paper introduces CAMILA (Context-Aware Masking for Image Editing with Language Alignment), a new method for text-guided image editing that addresses the shortcomings of existing models, particularly their inability to handle infeasible or contradictory instructions.\n\nThe core of CAMILA is its ability to assess whether an instruction is contextually appropriate for a given image. It uses a Multimodal Large Language Model (MLLM) to jointly process the image and text instructions. Based on this understanding, the model generates specialized tokens: [MASK] for regions that should be edited and [NEG] for areas that should remain unchanged, effectively ignoring non-executable instructions.\n\nTo further refine the editing process, the authors introduce a surrogate module trained to predict the CLIP-T score, which helps optimize the generated masks for higher quality results without the need for direct backpropagation through the complex diffusion model.\n\nThe main contributions of the paper are:\n- CAMILA Model: A context-aware image editing model that can distinguish between feasible and infeasible instructions, identify corresponding regions, and perform precise edits.\n- Context-Aware Image Editing Task: The paper proposes a new task and creates new datasets for evaluating models on their ability to handle both single and multiple instructions, including non-executable ones.\n- Superior Performance: CAMILA demonstrates significant improvements over existing state-of-the-art models in both quantitative metrics (like L1/L2 distance, CLIP, and DINO scores) and qualitative assessments, particularly in handling complex and context-aware instructions.", "questions": "N/A", "limitations": "Limitation:\n1. The proposed method lacks technical novelty. Most of the existing components are being heavily studied by existing literature. Therefore, the paper's scientific contribution is not quite enough.\n2. The paper only compares against a small set of baselines while ignoring many recent strong image editing models like SEED-Edit, Step1X-Edit, FLUX-based methods. \n3. The proposed method seems to be only applicable to local edits while failing for global edits. This limits the models' strength.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper is well motivated to address a known problem in complex image editing.\n2. The proposed method seems to work better than some other baselines.\n\nWeakness:\n1. The proposed method lacks technical novelty. Most of the existing components are being heavily studied by existing literature. Therefore, the paper's scientific contribution is not quite enough.\n2. The paper only compares against a small set of baselines while ignoring many recent strong image editing models like SEED-Edit, Step1X-Edit, FLUX-based methods. \n3. The proposed method seems to be only applicable to local edits while failing for global edits. This limits the models' strength.", "was_revised": true, "final_justification": "I think the evaluation scenario proposed by the work is rather limited. I believe more comprehensive comparisons on existing benchmarks like GEdit-Bench and Intelligent-Bench should be done against other methods to show the effectiveness of the model.", "rebuttal": "We thank the reviewer for providing insightful feedback. We are encouraged by the reviewer’s acknowledgement that our paper tackles an important challenge in complex image editing and that our method demonstrates superior performance over existing baselines. Below, we respond to the reviewer’s points.\n\n---\n\n### **[Weakness 1] Technical Novelty**\n\nWe appreciate the reviewer’s concern regarding the novelty of our proposed method. While CAMILA leverages established components such as pretrained multimodal large language models (MLLMs) and pretrained diffusion models, its core novelty lies not in introducing a new architecture per se, but **in defining and addressing a previously overlooked problem in text-guided image editing**: the executability of instructions in complex, real-world scenarios involving multiple instructions.\n\nExisting text-guided image editing models typically assume that all instructions are fully executable and attempt to apply them uniformly. This often leads to semantic failures when instructions are non-applicable or refer to nonexistent objects. CAMILA departs from this paradigm by introducing a context-aware mechanism that explicitly distinguishes between executable and non-executable instructions before any editing takes place.\n\nSpecifically, we introduce specialized tokens, [MASK] and [NEG] tokens, to gate editability. To the best of our knowledge, this is novel in the context of image editing, particularly in alignment with the visual semantics of the image. Furthermore, we designed the Token Broadcaster to ensure the coherence between the instructions and the image, enabling relevant edits to be applied only to appropriate regions while ignoring non-executable instructions.\n\nIn this way, CAMILA introduces a new modeling objective, context-awareness and executability filtering, along with new modules to implement it, and provides empirical results that demonstrate its effectiveness, particularly in failure cases that existing methods are unable to handle. We respectfully argue that this constitutes a meaningful and original contribution to the field of text-guided image editing.\n\n---\n\n### **[Weakness 2] Lack of Recent Baselines**\n\n\nWe appreciate the reviewer’s suggestion to include additional comparisons. However, we would like to clarify the following practical considerations.\n\nFirst, SEED-Edit [1] is not available as an open-source model, which makes direct experimentation and fair comparison infeasible at this time. Also, our model and all the baselines included in the paper are built on Stable Diffusion v1.5 [2], which operates on 512×512 resolution and has been widely used in text-guided image editing. In contrast, SDXL [3], used by SEED-Edit and other recent models, is pretrained on 1024×1024 resolution and differs significantly in both architecture and capacity. Direct comparison across models that rely on different diffusion backbones is not straightforward and may result in unfair evaluation, as the underlying architectures differ in model capacity and resolution.\n\nSimilarly, Step1X-Edit [4] employs a different diffusion backbone, DiT (Diffusion in Transformer), which again makes direct comparisons challenging. However, as the model is publicly available, we conduct experiments on Context-Aware Instruction dataset and include the comparison results in **Table 1**.\n\n\n|Method|L1|L2|DINO|CLIP-I|CLIP-T|\n|-|-|-|-|-|-|\n|Step1X-Edit [4]|0.0830|0.0329|0.8096|0.8892|0.2984|\n|CAMILA(Ours)|**0.0661**|**0.0222**|**0.9296**|**0.8932**|**0.3006**|\n\n**Table 1.** Additional baseline comparison on Context-Aware Instruction dataset\n\n\nCAMILA outperforms the state-of-the-art Step1X-Edit model across all metrics. This indicates that, like other models, the diffusion-based Step1X-Edit lacks the capability for true context-awareness. Furthermore, while Step1X-Edit takes 55.4 seconds to run on an NVIDIA A100 80GB GPU, our model only requires about 9.2 seconds. Despite being a significantly smaller diffusion model than Step1X-Edit, our model achieves better performance due to its context-awareness capabilities.\n\nWe hope this clarifies the rationale behind our baseline selection and the scope of our comparative analysis.\n\n---\n\n### **[Weakness 3] Scope of Edits**\n\nWe thank the reviewer’s observation regarding the scope of edits. Our Context-Aware Instruction Image Editing dataset includes global editing operations, such as background changes. Our model generates [MASK] tokens that are aligned with the semantics of the given instruction. Consequently, the size and coverage of the edited region are entirely determined by editing instruction. When an instruction implies a global transformation, such as changing the background, the model produces correspondingly large masks, enabling global edits.\n\nThis is demonstrated in the examples provided in the Appendix. Specifically, Figure 5 (c), (d) and Figure 9 (e) illustrate edits that affect the background of the image, showcasing the model’s ability to perform global modifications. We believe these examples support the model’s capacity to handle both local and global editing instructions, depending on the context provided.\n\n---\n\n### **References**\n\n[1] Shi, Yichun, Peng Wang, and Weilin Huang. \"Seededit: Align image re-generation to image editing.\" arXiv preprint arXiv:2411.06686 (2024).\n\n[2] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[3] Podell, Dustin, et al. \"Sdxl: Improving latent diffusion models for high-resolution image synthesis.\" arXiv preprint arXiv:2307.01952 (2023).\n\n[4] Liu, Shiyu, et al. \"Step1x-edit: A practical framework for general image editing.\" arXiv preprint arXiv:2504.17761 (2025)."}, {"review_id": "THws2qShyL", "reviewer": "Reviewer_AbnQ", "summary": "This paper presents CAMILA (Context-Aware Masking for Image Editing with Language Alignment), a method to improve text-guided image editing. The authors identify that most existing models struggle when given infeasible or contradictory instructions, often leading to nonsensical or over-edited results. CAMILA addresses this by using a Multimodal Large Language Model (MLLM) to jointly interpret the image and the text instructions. The model is designed to validate the contextual coherence of the instructions, distinguishing between what is executable and what is not.", "questions": "Could the authors provide a more detailed analysis of their synthetic data generation process to show that it is diverse and not easily gameable?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Streengths:\n1. The problem is interesting and important. Many image editing models are  over-edited, nonsensical garbage without detail thinking. The authors correctly identify that instruction executability is a massive blind spot, and directly tackle it. It's a practical problem that needed a solution.\n\n2. The \"Token Broadcaster\"  uses the diffusion model's own text encoder to create the text embeddings that are used for alignment. This  ensures the masks generated by the MLLM are aligned with the exact same semantic space the diffusion model uses for conditioning. \n\n3. The Results are good. CAMILA  consistently beating the baselines. And in Figure 4 (c), CAMILA  correctly identifies the instruction as nonsense and does nothing, which is precisely the right move.\n\n4. The authors create a benchmark to evaluate model behavior and context-awareness in challenging scenarios.\n\nWeaknesses:\nIn Section 4.5. The authors admit that backpropagating through a diffusion model to optimize for a CLIP-T score is \"infeasible.\" And build a separate, lightweight transformer to predict what the final CLIP-T score will be. Then use this predicted score as a loss to train the actual model.  The ablation in Table 3 shows the improvements are marginal at best—a CLIP-T score going from 0.3002 to 0.3006.", "was_revised": true, "final_justification": "Most of my concerns have been resolved. I will keep my score for accept.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and encouraging feedback. We are especially grateful for the recognition of the importance of instruction executability, the practical relevance of our problem setting, the effectiveness of our proposed Token Broadcaster module, and benchmark dataset design. Below, we provide our responses to the weaknesses and questions.\n\n---\n\n### **[Weakness 1] Marginal Improvement of Surrogate Module Training**\n\n\nWe acknowledge the reviewer’s observation that the improvement from the surrogate module is marginal. However, even without the surrogate module training, our framework outperforms other baselines on most metrics, so we introduced the surrogate module as an additional training for potential refinement. We had expected further gains in text alignment score, assuming that improving CLIP-T score would more capture user-intended modifications. While we did observe some improvement, further gains were inherently limited by the already strong baseline performance. We plan to investigate more advanced strategies to secure larger improvements in future work.\n\n---\n\n### **[Question 1]  Detailed Analysis of Synthetic Data Generation Process**\n\n\nWe appreciate the request for a more detailed analysis of our synthetic data generation process. We used the “gpt-4o-2024-05-13” model to generate non-applicable instructions. These were based on a controlled but flexible prompt covering various editing types (add, remove, replace, transform, etc.), requiring that the target object not be present in the image and that the instruction remains realistic and grounded. We include the full generation prompt below, which was paired with the input image to generate five non-applicable instructions for each image.\n\n\n> “I want to create image editing instructions for an image. Instructions like 'Put something', 'Remove something', 'Replace something', 'let there be', 'have there be', 'add something', 'make something something', 'Change something', 'What if something is', 'transform something to something' and similar. \\n\\n Generate five non-applicable image editing instructions for the given image. \\n1. The instructions must target objects or elements that are not present in the image.\\n2. Instructions can involve adding, removing, or modifying objects, but:\\nIf adding, it must involve placing something next to, inside, or interacting with an object that does not exist in the image.\\nIf removing or altering, the instruction should involve an object that is completely absent from the scene.\\n3. Ensure the instructions are grounded in reality and fit the theme of the image (e.g., no fantastical elements).”\n\n\nThis ensures that the generated non-applicable instructions are not only structurally diverse but also semantically grounded in the actual content of the image, avoiding generic or gameable patterns. Additionally, to ensure the instructions were not trivially gameable, we randomly sampled one instruction out of the five and used the GPT model to explicitly verify its non-applicability to the image. This two-stage filtering guarantees semantic correctness and variety. Furthermore, Appendix C.2 shows that non-applicable instructions are distributed across multiple editing types, supporting the conclusion that the dataset is not skewed toward a single pattern."}]

Author rebuttal:

Meta-review: The paper proposes a method CAMILA for the masked image modeling task. This method constructs a Context-Aware Aggregation module, which improves the reconstruction of masked tokens by leveraging both global context and local neighborhood interactions. Several reviewers agree that this method is simple but interesting and that it shows effectiveness on the evaluations. During the discussion phase, the authors provide further explanations and results, which address many reviewers' concerns. This paper is expected to reach the acceptance level of NeurIPS 2025, but authors need to complement the provided statements and experiments in the final version.
Decision: Accept (poster)