Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing
Authors: Zhuoying Li
Zhu Xu
Yuxin Peng
Yang Liu
Venue: ICML 2025 poster (2025)
Keywords: Image Editing
Evaluation Metric
Primary area: applications->computer_vision

Abstract: Instruction-based image editing, which aims to modify the image faithfully towards instruction while preserving irrelevant content unchanged, has made advanced progresses. However, there still lacks a comprehensive metric for assessing the editing quality. Existing metrics either require high costs concerning human evaluation, which hinders large-scale evaluation, or adapt from other tasks and lose specified concerns, failing to comprehensively evaluate the modification of instruction and the preservation of irrelevant regions, resulting in biased evaluation. To tackle it, we introduce a new metric Balancing Preservation Modification (BPM), that tailored for instruction-based image editing by explicitly disentangling the image into editing-relevant and irrelevant regions for specific consideration. We first identify and locate editing-relevant regions, followed by a two-tier process to assess editing quality: Region-Aware Judge evaluates whether the position and size of the edited region align with instruction, and Semantic-Aware Judge further assesses the instruction content compliance within editing-relevant regions as well as content preservation within irrelevant regions, yielding comprehensive and interpretable quality assessment. Moreover, the editing-relevant region localization in BPM can be integrated into image editing approaches to improve the editing quality, manifesting its wild application. We verify the effectiveness of BPM metric on comprehensive instruction-editing data, and the re- sults show that we yield the highest alignment with human evaluation compared to existing metrics, indicating efficacy. The code is available at https://joyli-x.github.io/BPM/.

Balancing Preservation and Modification:
A Region and Semantic-Aware Metric for Instruction-Based Image Editing
Zhuoying Li
Zhu Xu
Yuxin Peng
Yang Liu
Abstract
Instruction-basee image editing, which aims to modify the image faithfully towards instruction while preserving irrelevant content unchanged, has made advanced progresses. However, there still lacks a comprehensive metric for assessing the editing quality. Existing metries either re- quire high costs concerning human evaluation, which hinders large-scale evaluation, or adapt from other tasks and lose specified concerns, fail- ing to comprehensively evaluate the modifica- tion of instruction and the preservation of irrele- vant regions, resulting biased evaluation. To tackle it, introduce new metric Balancing Preservation Modification (BPM), that tailored for instruction-baset image editing by explicitly disentangling the image into editing-relevan and irrelevant regions for specific consideration. We first identify and locate editing-releves regions, followed by a two-tier process to assess editing quality: Region-Aware Judge evaluates whether the position and size of the edited region align with instruction, and Semantic-Aware Judge fur- ther assesses the instruction content compliance within oditing-relevan regions as well as con- tent preservation within irrelevant regions, yield- ing comprehensive and interpretable quality as- sessment. Moreover, the editing-relevant region localization in BPM can be integrated into im- age editing approaches improve the editing quality, manifesting its wild application. We verify the effectiveness of BPM metric on com- prehensive instruction-editing data, and the re- sults show that we yield the highest alignment with human evaluation compared existing met- rics, indicating efficacy. The code is available https: pettps:/jjont http///oolll/////// x wi..ii..ttittt.. git......... 1o/BPM/
Text Instruction:
Original Image
Evaluation Hetrics
Editeed
Edited Inage
0.859
9.177
Not Align!
CLIP-Score
9494
8.851
DINO-Score
0.989
(PN
0.62
0.95
Align!
Human Evaluation
1.767
46.667
(a) Results of evaluation metrics for identical sample.
Encemsially
aectraciioo Aed
Orsetet Zoppe
Paniced Oilaaed prestivatiin)
Fivered by
(b) Visualized Example of Ground Truth Test.
Figure 1: (a) The scores rated by existing metrics LPIPS, CLIP- Score and DINOScore all witness contradict trends with human evaluation, while our proposed BPM aligns with human evaluation. (b) Previous metrics favor excessively preserved or modified result, while our BPM favor the well-edited image.
1. Introduction
Instruction-based image editing (Hertz et al., 2022; Kawar et al., 2022; Brooks et al., 2022; Han et al., 2024; Wang et al., 2024) has gained increasing attention, which aims modify the given image via text instructions, requiring the edited image faithfully align with the modification requirements while preserving the unrelated parts of the origin image. Though a great success, the field still lacks a sufficiently standardized metric for comprehensively assessing the edit- ing quality. Ideally, we would expect the most authoritative evaluations to come from human assessments. However, the costs and overhead associated with human evaluation too high scale, hindering its practical value in large-scale evaluations. In response to it, a series of existing works (Hertz et al., 2023; Basu et al., 2023; Gal et al., 2022; Kim & Ye, 2021; Brooks et al., 2022; Ruiz et al., 2023; Kocasari et al., 2022) have proposed automated evaluation metrics.
Equal contribution 'Wangxuan Institute of Computer Tech- nology, Peking University. Correspondence Yang Liu <yan- thu@@pu..edun eku..cd..nn>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Bant Image Editing
One line of metries evaluate the alignment between edited image and text instruction, such CLIPScore (Hessel et al., 2021). Other metrics, like DINOScore (Liu et al., 2023), LPIPS (Zhang et al., 2018) and CLIP-I assesses the sim- ilarity between original images and edited images. But these metrics often perform contradict human evaluations in cases, leading their results deemed unreliable. As shown in Fig. I(a), two edited images are rated by different existing metrics, the right image successfully incorporates "human kayaking" to fulfill instruction requirements, gaining higher human evaluation score. However, all existing metrics show an opposite conclusion, highlighting their untrustworthiness. We emphasize that the flaws in current automatic metrics arise from the following reasons: (1) These metries are not specifically tailored for this task, typically originat- ing from task evaluation for image generation. They fail to simultaneously utilize < Origin Image, Edited Image, Instruction > elements of instruction-based editing task, ig- noring crucial information: overlooking of instruction fails to judge the modification quality of target editing element, while overlooking of origin image results in poor judgment of the preservation quality of regions should not be mod- ified. (2) These metries evaluate the entire image as a whole, without accounting for the fact that some regions of the image need to be preserved, while others require modification. As result, they fall short of assessing the different requirements for editing process. To verify it, we conduct a ground truth test': for an input image and editing instruction, apart from the well-edited result (which serves as ground truth), we introduce two types of images gener- ated by holistic operations performed on the entire image for comparison. One is directly adding noise to the original image, serving as an excessively preserved global edit. The other is directly generated by a Text-to-Imagee IRommah et al., 2021) model based on the instruction, serving as an excessively modified global edit. One visualized example is shown Fig. 1(b). Existing metries favor excessively modi- fied or preserved results that acquired from global editing entire image, manifesting their evaluation are biased without separate consideration for different image regions. Evalu- ating image as a whole also lead to the inability to assess whether the changes of edited region properties, such as size and position, follows the instruction. The above reasons reveal the urgent need to develop a more comprehensive metric to evaluate instruction-based image editing.
with comprehensiveness and interpretability. However, given the diversity nature of instructions, accurately localizing the editing regions is non-trivial: for complex instruction involving object size change and spatial relation requirement, the extent of scaling ("make the apple bigger") and displacement ("add an apple to the right of banana") during the editing process makes editing region highly flexible, necessitates referring to both original and edited image to determine it. To enable such accurate localization, we parse the instruction by large language model(LLM) to determine the source and target object during editing, and utilize detection and segmentation tools to locate the editing regions (both in original and edited images). Then, we propose a two-tier evaluation process: Region-Aware Judge verify whether the size and position of edited region aligns with instruction, ensuring the overall editing region is roughly correct; Subsequently. Semantic-Aware Judge is adopted to conduct more fine-grained evaluation from semantic perspective: for the localized edited region, we retain the directional CLIP similarity to assess whether the modification faithfully complies with the instruction; for irrelevant regions, we adopt L.2-distacce to calculate whether the original content has been successfully preserved. Such hierarchical evaluation facilitates comprehensive and nuanced assessment of all requirements for editing task. Experimentally, our BPM witnesses remarkable alignment improvement with human evaluation on diverse citruuction-bsse editing data, indicating that BPM can provide more trustworthy evaluation results in comparison with previous metrics. Furthermore, we find the modification region localization process within BPM can be seamlessly integrated into existing image editing approaches, guiding the editing more focused on target editing region while reducing unnecessary modifications to irrelevant regions, yielding more satisfactory results.
The major contributions are summarized as follows. (1) We delve into the drawback analysis of existing metrics for instruction-basee image editing; (2) We introduce novel metric BPM for instruction- -based image editing, which eval- uates editing quality from the region and semantic perspec- tive with explicit disentangled separate consideration for editing relevant and irrelevant region, yielding comprehen- sive and interpretable evaluation score; (3) Through exten- sive experiments, BPM demonstrates a high correlation with human evaluation and outperforms previous metrics, indi- cating its efficacy; (4) We utilize region localization process within BPM to provide a training-free enhancement for ex- isting image editing approaches, yielding more accurate edited results with fewer irrelevant modifications.
To this end, we propose Balancing Preservation Modification (BPM) tailored for Instruction-based image editing. The core idea is to explicitly disentangle the image region that should be modified or preserved, separately evaluating the semantical instruction compliance for the modified region and the content preservation for the irrelevant region, thus endowing the evaluation process
2. Related Works
Conventional Image Editing Quality Evaluation Metrics
'The full experiment and detailed analysis is in Sec.
2

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Basse Image Editing
Existing metrics evaluate the editing quality from two dif- ferent perspectives: the first line of metrics, such LPIPS (Zhang et al., 2018), DINOScore (Liu et al., 2023) and L2 distance, measures the similarity between the edited and original image, but they overlook to evaluate the modifi- cation quality for the given instruction; the other line of metrics, such as CLIPScore (Gal et al., 2021), evaluates the success of an edit by assessing the semantic alignment be- tween the edited image and the instruction. However, such metrics overlook that instruction-based editing requires both image content that should be modified and preserved. All above metrics fail to utilize the "origin image, edited image, instruction" " information for comprehensive evaluation. Be- sides, we emphasize that simply combining the two lines of metries is insufficient: they all serve the entire image as a whole for evaluation, which is against the fact that different image regions encompass different requirements, i... preservation or modification. In contrast to their ap- proach, our BPM not only explicitly separates the image into relevant and irrelevant regions for editing, allowing for distinct evaluation, but also incorporates all three elements of the editing process. This enables the assessment of both modification and preservation, as well as the accuracy of the edited regions and content, resulting in more interpretable evaluation score.
Algorithm 1 BPM Evaluation Process
1: Input: Original Image Lorigins Edited Image Ledst. text editing instruction Tedit
2: Functions: large language model LLM to analyze Tedit function " calculates the modification and preservation score for edited image. D(..:) detects and segments objects of specific category and return corre- sponding bounding box and mask. sssize and J Sposition verify the attribute alignment in terms of edited object's size and position.
3: Output: Region-Aware Evaluation Score Sregion and Semantic-Aware Score Sseaantin-
4: 01, 02, posst, size,t + LLM(Tedda) {parse the instruc- tion to acquire the source and target object 01, O2, and the size state change (lager, smaller, unchanged) and location change pos right, up, down, un- changed) during the editing process. }
5: Morigin. Borigin & D(Torigims 01)
6: Media Bedia t- D(Tedit 02)
7: Sposition + Sposition (Borigins Bedit, posst, Tedit)
8: Ssize t- Jaize (Morigins Medsts size,t)
9: SmodkJy Spreserve + F(Forigins Ledits Borigin Bedit Morigin: Medit)
10: Sregion Sposition + Syize
11: S Ssemantic x- Smodify fy + Spreserve
VLM-based Image Quality Evaluation Metrics
12: BPM + -a= Ssemantic + (1 -5-0 a) Svegion.
Considering the great reasoning ability and visual under- standing ability of Vision language models (VLMs), several prior works adopt VLMs as automatic evaluators for image quality. (Ku et al., 2024; Lu et al., 2024; Li et al., 2024; Xu et al., 2024; Hui et al., 2024) proposes to utilize vision language models to automatically evaluate text-to-image generation task by assessing image-text alignment. HQ- Edit (Hui et al., 2024) utilizes GPT-4o (OpenAI, 2024) to evaluate the image editing from coherence and alignment perspectives. However, even the state-of-the-a VLMs like GPT-40(OpeenAI, 2024), are reported to show decent capa- bility in such image quality assessment concerning image- text alignment, and mistakes are often (Orpeessed(OpenAA. 2023) during such evaluation process, hindering the trust- worthy of (LM-based approaches. Instead of relying on VLMs for the whole evaluation process, we only adopt them to parse the instruction to locate the edit-relevant regions, thus alleviating the above mentioned problem.
Torigin. edited image Tedit and instruction Tedst in existing metrics results in biased evaluation, we incorporate all three elements for calculation. We first explicitly decompose both Torigin and Tedit into editing relevant and irrelevant regions. It is accomplished by first adopting Large Lan- guage Model (Line 4) to parse instruction acquire the source and target object 01, 02 involving the editing pro- cess, then utilizing detection and segmentation tools D(-) to separately locate the source object o₁ and target object 02 in original image Lorigin and edited image Ledits acquiring bounding box Boriginjedat and mask Morigin/edst (Line 5- 6) to disentangle the whole image. We also guide the LLM to output the required edited object size and position chang- ing state size,t, prs during editing, which are utilized to conduct following Region-Awars Judge, which aims to ver- ify the editing process aligns with the instruction in terms of the size and location changing. We design specific func- tions Spsosition (-) and ssize(-) acquiring scores for region size and position alignment (Line 7-8), termed Sposition and Saise, which together form Region-aware score Sregion. To enable more fine-grained Semantic-Aware Judge, we adopt function F(-) to evaluate the semantical instruction compliance within editing region (Smodify) as well as con- tent preservation for irrelevant region (S (Sprescrue) (Line 8), which together as Semantic-Awen score Ssemantie- Finally, we obtain the BPM score by adding Ssemantic and Sregion
3. Method
3.1. Overview
This section presents BPM, novel evaluation metric that enables accurate and comprehensive editing quality assess- ment. The whole process is shown in Alg. 1. Considering the overlook of simultaneous utilization of origin image
3

---

Balancing Preservation and Modification; A Region and Aemantic-Aware Metric for Instruction-aase Image Editing
Editing Instruction: Turn the clock into street sign.
Judge the object before and after edited.
"clock"
LLM
Object Identify
(beforee: cloch 02(after): street sign
"street sign"
Stte
How is the size of edited object changed?
S stze
Unchanged.
Size State
Detector & Segwentor
posttiio
Is the location of edited object changed?
pasttion
Original Image
oorignn
Unchanged.
Region-Aware Judge
Position State
Crop Modification
Region
muulity
Instruction Complt iance
Barigin.
Bott
A
origin
A
edit
Semantic Judge
Editing
Ivrel Ivrelevant Region
Edited Image
preserve
Tadit
origin
adit
AP prigio
AP
Content Preservation
Figure 2: Overall Pipeline BPM, Firstly, LLM utilized to parse and analyze editing instruction, generates responses identify the source and target object, as well as the object size and position changing state requirement during editing. Then we conduct Region-Aware Judge to verify editing follows the instruction for region size and position, yielding region-aware score Sreyion. For Semantic-Aware Judge, utilize detection and segmentation tools locate and segment the edited object, subsequently apart the origin and edited images into edited regions and irrelevant regions. separately evaluating the semantical instruction compliance in edited regions and content preservation irrelevant regions, yielding semantic-aware score Ssemantic.
with different weights (Line 12).
position undergoes displacement (outputting "left", "right", "up". "down" or "unchanged", termed pos.t). In the exam- ple, the instruction requires no specific position and size change, and posst and sizest is "unchanged". Then for editing region localization, considering that the diverse edit- ing instructions necessitates simultaneous consideration of both the original image Lorigin and edited image Ledie to accurately determine the modified regions, employed detection and segmentation models to identify the position of 01 and in Torigin and Tedits represented as bounding boxes B Borigin/edit and instance masks M Morigin/eda /edit, respec- tively. By incorporating the modified region within original image Torigin as well as edited image Tedits we can accu- rately acquire the modified region within the editing process. Specifically, for the case of "add object", the original object 01 is set "None", and the corresponding mask Morigin and bounding box Borigin are set identical as Medit/ Bedit in default. Similarly, in the "remove object" case, the edited object 02 is set to "None," and Medi// Bedit are initialized to be identical to Morigin/ Borigin.
In the following, separately introduce how we parse the instruction and localize the modified regions in Sec. 3.2, the Region-Aware Judge (object position and size alignment ver- ification) in Sec. 3.3, the Semantic-Aware Judge (semantical instruction compliance within modified regions and content preservation for irrelevant regions) in Sec. 3.4. Finally, we delve into attempt to utilize the editing region localiza- tion process within BPM to improve the editing quality of image editing methods in Sec. 3.5.
3.2. Instruction Parsing and Editing Region Localization
We employ LLM to parse the editing instruction Teast based on three criteria, involving the editing object identification, object repositioning and size modifications judgement. One example shown in Fig. The model is required to first decide the exact object category involved in the editing pro- cess, outputting clock source object θ₃ and street sign as target object 02. What's more, L.M is also guided to determine whether the editing is required to affect the size of the corresponding edited object (outputting "larger," "un- changed," or "smaller", termed size.t xt) and how the object's
3.3. Region-Aware Judge
After acquiring the masks Morigin, Medie and bounding boxes Borigsn Bedit for original object or and edited object

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Basse Image Editing
02, we verify the size and position changing is aligned with text instruction Tedit, acquiring the region-aware evaluation score. We provide two rule-based functions, termed size judgment (faize in Alg. 1) and position judgment fposition in Alg. 1) respectively, as shown below.
(2)Saliency Check: The ratio of the front and rear area needs to be greater than (for "larger" sizest) or less than (for "smaller" as sizest) a certain threshold to check the chang- ing saliency. According to above criteria, we yield score Ssize as (coonformed) or O(disobey). Instructions with "un- changed" size,t is evaluated by Saliency Check, restricting the ratio of areas to be around I within a certain range.
The pipeline is shown in Fig. 3(a). We utilize LLM to acquire the position changing state posst re- quired in instruction, which denotes spatial relation between edited object and reference object. Specifically, the edited and reference object is identical for instructions without ex- plicit reference object, such as "Move the apple to the right." " With the bounding boxes of edited and reference object, we employ three criteria to determine whether the editing meet the requirement: Pl)sosition Assessment: The center coordi- nates of bounding boxes are used for direct position assess- ment; (2)Direction Verification: The difference between the x- and y-coordinates is required to check the existence of unwanted edits in unrelated directions; (3)Saliency Check:
Position Alignment:
Algorithm 2 Semantic-Aware Judge Process
1: Input: Original Image T Torigin: Edited Image Tedit,
mask Morigin Medit, bbox Borigin. Bedit-
2: Functions: Cn crops image regions following given bbox, CLIP1 and CLIPT r are CLIP image encoder and text encoder. Norm min-max normalization.
Output: modification score Smodify and preservation
score Spreserve.
4: A CB(loriginn Borigin)
5: A + CB(Tedia, Bedst)
6: A origin + CassTooriiinn MorrignnMorrggnl
A certain spatial distance between the edited object and the reference object (expressed as bounding box IOU less than a threshold) further required to check the editing saliency. If all three criteria are satisfied, we consider the editing comply with instruction and score Spasition as otherwise, score Spocition as O. Particularly, for "unchanged" in- structions, we only evaluate it through Saliency Check by requiring the bounding box IOU is greater than a threshold.
7: A f Cedieee Medit)
8: Abrigin t (1 (1-Morignn U Medit) @ Lorigin
9: A t (1 Morigin Medit) @ Ledit
10: Smodify cos_simmCII
(LIPT(02)-CLIPP)) (CM.(IIIIAAAAA (AAA) (
11: Spreserve & 1 -L2( Aoriginn Aedit) editt
12: Seemantic + Norm (S Spreserve) + Norm(Smodits)
cbtting Inttrcctin:: Ald the ride 45 the sarphaard.
right
3.4. Semantic-Aware Judge
Pasician Assessment
Furthermore, to conduct more fine- -raiind evaluation from semantic perspective, we leverage the obtained bounding box and mask information to disentangle the origin and edited image lorigin and Leais for separately evaluating the quality of semantical instruction compliance and content preservation. The process is shown in Alg. 2.
22
Direction Verificatian
[x2
1y2-22]
Salfency Chach
LWVCsurfoosed (eg) <
Originet
Cmage
Edited tange
Spoese
(a) Position Alignment verification.
Semantical Instruction Compliance:
the obtained bound- ing boxes Borigin: Bedit are utilized isolate the edited object's region, thereby excluding interference from un- modified areas and enhancing the quality assessment for modification, yielding cropped edited regions A Aorigin for Lorigin and A for Ledit (Line 4-5). Then, to measure the semantic consistency between the editing results and instruc- tional specifications, following CLIP Directional Similarity (Kim et al., 2022), we calculate the cosine similarity be- tween the difference of A Aprigsn Am and corresponding text embedding of 01, 02 in CLIP space as follows:
Editiog Instruction: fich higgrr.
sizeg Larger
Originat
Burigg areal
Area Exarination
Seliency Chech
areoZ/arenl z I+ 0
Kmage
Kuzi dree2
Sav
(b) Size Alignment verification.
Figure 3: Evaluation functions for region-aware judge.
The pipeline is shown in Fig. 3(b). The area of Morigin and Medit represents the original and edited object size, and we evaluate whether the size change con- forms to required size change state size,t with following criteria: (1)Area Examination: Size comparison of Morigsn and M ediit directly adopted to examine the requirements.
Size Alignment:
Smodify = cs (A CA (Andst)
CLIP - -CLIPr(0z)
where cs(a, b) a
| denotes cosine similarity,
CLIP((-) and CLIPr(-) is the CLIP image encoder and text
5

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Basse Image Editing
encoder. The rationale behind such directional similarity calculation is that by computing the difference in features before and after editing, we can not only assess whether the features requiring modification align with the instruction, but also implicitly verify that other attributes unrelated to the editing process are successfully retained, thus providing a more comprehensive reflection of correspondence between editing and instruction. Higher Semanniic indicates better semantic instruction compliance.
where the 2e is the denoised latent vector during editing process, t is the timestep, 81 and ST are the guidance scale for input image lorigin and instruction Tedit, respectively, O is hadamard product operation. By adopting such cless-free guidance, the instruction guidance is restricted within the region that is relevant to the editing process, thus reducing unnecessary modifications in other regions, subsequently yielding more satisfactory editing results.
Irrelevant Content Preservation: we utilize masks to ex- clude modified regions to only retain the preserved parts. Considering the potential object displacement during edit- ing, we utilize the union of Morigin and Medit to exclude all areas intended for editing, which is formulated as:
4. Experiement
4.1. Implementation Details
We employ a pre-trained CLIP-ViT 14/B(Radford et al., 2021) model for directional similarity calculation. For the LLM for instruction parsing, we select gemma-2-9b- it-SimPO(Meng et al., 2024) in seek of trade-off between cost and performance. For detector and segmentor, we use Grounding DINO (Liu et al., 2023) and Grounded SAM (Ren et al., 2024). For scaling weight for Sxemantics we set it to 0.7 (i.e. BPM = 0.7 Sxemantic + 0.3 * Sregion. More results with different LLMs, further details on prompt- ing the source and target descriptions and the impact of different a are referred to the appendix.
Aoriign = [1 - Morigin U Medst] 2 forigin
A = [1 - Morigin U Meait] O Iedit
(3)
where U is the union operation between masks, and ( is the element-wise matrix product. Considering the po- tential object displacement during editing, we utilize the union of Morigin and Medst to exclude all areas intended for editing, and subsequently obtain the preserved portion for original image Agrigán and edited image A Then, we measure the L2-distance between 4 origin and Aldit- subsequently acquiring preservation score Spreserve = L2(A"riggn Aldit) Higher S Spreserve denotes better content preservation.
Evaluation setting. We conduct Human Alignment Test and Ground Truth Test. Human Alignment Test assess the correlation between metric evaluation and human judgment. To implement it, we first conduct user study, asking users to rate the editing images from ((worst) to 5(best).² For two images Ledit and 1adse edited by two different editing models with identical input, we examine the consistency in the ranking of their human scores H1, H2 and metric scores M,, M2. Human ranking P, is calculated by com- paring Hi and H2, where Pn = HH2 Similarly, we calculate the metric ranking as Pm = 1 M2.- Finally we calculate the aligmment ratio of Pry. and Pn across all the samples as Alignment Score= N E Lpx= where N is the sample number. Higer Alignment Score indicates better alignment between human and metric evaluation, and we report it for each pair editing model for comprehensive comparison. Ground Truth test assesses whether the met- rics correctly identify the well-edited image within a triplet of images categorized as excessively modified, excessively preserved, ground truth(well edited). The ground truth are manually filtered and selected as well-edited image, while the excessively modified images are directly generated by a text-to-image model (Rombach et al., 2021) according to the instruction, the excessively preserved images are generated by adding Gaussian noise to the original image. We report the proportion of favoring for each metric across these three types of images, with the total sum equal to 1. A higher proportion of favoring for ground truth images indicates
Ultimately, we acquire the semantic aware score S Sseonntic by normalizing and combining Smodify fy and Spreserve. which accurately assess the instrution compliance in edited region, as well as the content preservation in irrelevant re- gions.
3.5. Application for Editing Quality Enhancement
Furthermore, considering that the process in Sec. 3.2 enables accurate editing region localization, we delve into an attempt to utilize the editing region located as the guidance for im- age editing model, leading the model to focus attention on the regions that need editing, thereby reducing modifications to unrelated areas and achieving results that better align with the editing instructions. Specifically, we first acquire the edited image using original editing model, and adopt the re- gion localization process in BPM to acquire mask of edited area in both Forigin and Iedat: Mall = Morigin U Medit- Then in the second-round editing quality enhancement pro- cess, Mal serves as explicit guidance through classifier-fres free guidance (Ho& & Salimans, 2022) with following formula- tion.
ca( lrrgins Teia) ca( 2,
81 + (colz,, Iurigin, 2)- co(2,, 2, 2))+
(4)
"The user study details are provided in Appendix.
ST (colz,, 1, Trigin Tedi)) co( lorginn 21) @ Mall
6

---

Balancing Preservation and Modification; A Region and Semantic-Awann Metric for Instruction-aase Based Image Editing
Table Human alignment on BPM and existing image editing metrics for local edits. report the alignment of preference between metric and human evaluation for paired editing images. Bold and underline represent the highest and second-best performance.
MGIE vs PTpp
MGIE va IP2P1
IP2P vs. PTppee
DaL IP2P1
DALLE-2 FTiP2P
DALLE vs MGIE!
Average
1.2
0.651
0.614
0.724
0.762
0.659
0,801
0.702
LPIPS
0.687
0.602
0,665
0.731
0.626
0,791
0.683
CLIP-I
0.578
0.574
0.571
0.705
0.643
0.796
0.644
DINOScore
0.608
0.597
0.606
0.746
0.692
0.796
0.674
Modificttinn Metrics
CLIPScore
0.530
0.494
0.394
0.429
0.387
Vi---aased Metrics
GPT-4o
0.681
0.676
0.712
0.860
0.780
0.812
0.754
Rolonced Metrics
CLIPScore +CLIP4
0.648
0.667
0.607
0.691
0.573
0.660
0.641
0.663
0.659
0.818
0.943
0.802
0.911
Table 2: Human alignment test on BPM and existing image editing metrics for global edits.
PIE vs HQJEDIT
PIF vs FTiP2P
HQLEDIT FTIPeP
PTpgpes/PPP
IP2P vs HQLEDIT
TP2P vs PIE
Average f
Preservation Metries
LPTPS
1.000
0.917
0.375
0.467
0,480
0.900
0.690
CLIP-1
0.966
0.861
0.417
0.600
0,520
0.850
0.702
DINOScore
0.966
0.833
0.250
0.533
0.520
0.900
0.667
Modifection
CLIPScore
0,897
0.833
0.917
0.667
0.760
0.825
0.817
VIM--hased Metrics
GPT-4o
0.759
0.722
0.583
0.733
0.760
0.825
0.730
Bolanced
CLIPScore +CLIP-I
1.000
0.917
0.458
0.750
0.600
1.000
0.787
BPM
0,897
0.889
0.750
0.750
0.840
0.850
0.829
Table 3: The inference speed of metrics.
Table 5: Effectiveness verification for component score of
Ssemantics Spreserve and Smodify separately targets for content preservation and instruction compliance quality evaluation, compare them with corresponding human evaluation.
(Ours)
CLIP-T
DINO
GPT-4o
Speed (seconds per image)
1,500
0,255
0.350
0.067
0150
0x08
11:40
Table 4: Ground Truth test. We report winning rate out of three type of images for each editing metric. "EP" represents "Excessively Preserved",, "EM" represents "Excessively Modified", "GT" represents "Ground Truth". Each sums 1, indicating how many portion of images the metric that row favors.
JP2P [TTrrrr
DaLLE 2 IP2P
DALLE PT
pissevvaiion
Spraere
0,829
6,933
0,757
modification
CLIPSeas
0.634
0.444
B.499
Table 6: Effectiveness verification for Sposition and Ssize- We conduct study to verify the alignment between human and metric judgment towards editing region size and position change.
Metrics
EP
EM
0.17
LPIPS
1
CLIP-I
0.87
MGIE
IP2P
Dal.
FT
Saiee
0.89
0.89
0.93
0.92
DINOScore
0.98
0.02
3 Spacttion
0.71
0.69
0.89
0.75
CLIPScore
0.04
0.83
0.13
CLIPScore +CLIP-I
0.42
0.14
0.44
GPT-4o
0.08
0.14
0.78
of existing automatic metries: (1) Preservation Metries focuses on the similarity of original and edited image, in- cluding DINOscore (Liu et al., 2023), LPIPS (Johnson et al., 2016), CLIP-I and L2 distance. (2) Modification Metrics evaluate alignment between edited image and text instruc- tion, where CLIPScore (Hessel al., 2021) the only one. We also incorporate the trade-off between CLIPScore and other metrics, like hensive comparison. (3) VLM-based Metrics utilize VLM to automatically evaluate the editing quality, we select most advanced VLM GPT-4o (OpenAI, 2024) with the evaluation prompts HQ-Edit (Hui al., 2024) for comparison.
BPM-overall
0.12
0.01
0.87
better evaluation for that metric.
For evaluation setting that compares the correlation between human scores and metric scores, we explain in the supple- mentary materials the reasons why this type of experiment is not recommended in this task, as well as the relevant experimental results.
for more compre-
Compared Metrics. We compare BPM with three lines

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Bant Based Image Editing
Evaluation Datasets. For local editing, we sample image- instruction pairs from MagicBrush (Zhang al., 2024), which covers six mainstream instruction editing types: adding objects, replacing objects, changing actions, chang- ing colors, removing objects and changing patterns. Then image editing models IP2P (Brooks et al., 2022), MGIE (Fu et al., 2023), DALLE-2 (Ramesh al.. 2022), and FTIP2P (Zhang et al., 2024) generate edited images. For global editing, we sample image-instruction pairs from the global edits subset of PIE-Bench (Ju et al., 2023). In addition to the baseline of PIE-bench itself, we also used HQ-Edit (Hui et al., 2024), IP2P (Brooks et al., 2022), and 1212 (Zhang et al., 2024) to generate edited images. Finally, we get 960 entries with human score annotation.
(1) preservation metrics (L.2, LPIPS, CLIP-I, DINOScore) prefer excessively-preservee results, while modification met- rics (CLIPScore) prefer excessively-modifee results, indi- cating all these metrics exist biased evaluation, leading their evaluation deemed unreliable. (2) Our BPM -overall favors the ground truth images, i.e., the well-edited ones with 87% ratio and surpasses GPT-4o results, which is credit to our explicit disentanglement of editing relevant and irrelevant regions for separate consideration, alleviating the biased towards the preservation modification.
we employ four advanced instruction-base
Instruction: Make plass of cola.
4.2. Comparison with other evaluation metrics
Originsi Inage
Eétted Image
EAttel Irage
Fixeed Trage 3
UMPS&
0.091
0.246
0,052
Human Alignment Test result is shown in Table. 1 (local edits) and Table. 2 (global edits). For local edits, we can conclude that: (1) for each editing model pair comparison, overall score (Seemantic + Sregion ) of BPM yield the high- est alignment with human evaluation compared to existing editing metrics that evaluating editing quality from preser- vation and modification perspective in most cases(Line 1-5), indicating BPM can serve as a more trustworthy and authen- tic metric for image editing evaluation; (2) we also merge the score of CLIPScore and CLIP-I serve as a stronger comparison (Line 7) with both consideration of preservation and modification, and BPM still outperforms it, indicating simply summarizing preservation and modification metrics is insufficient, as they overlook to separate consider the requirements for region relevant or irrelevant to editing pro- cess, and BPM resolves it by explicitly disentangling the image for separate evaluation, yielding more satisfactory results. (3) BPM outperform the VL.M-based evaluation powered by GPT-40(OpenAA, 2024) (Line 6), indicating that directly adopting VLM for automatic evaluation exist untrustworthiness, and we mitigate such problem to enable more authentic evaluation. Besides, as shown in Table. 3. our BPM shows much faster inference speed than GPT-4o, validating our practicality. For global edits, we can conclude that: (1) Our BPM still yields the highest human alignment compared to other metrics, showing its effectiveness and wild application scope. (2) It is noted that the ClipScore shows a significant improvement compared to local edits. This may be because, in global edits, a larger number of pixels are modified, and the shortcomings ClipScore in terms of overlooking original image are less pronounced compared to local edits.
CUPScove
6,178
0.161
6.360
4o7
94
82.5
50
SanceCUPPSSmee
0.117
0.804
0:360
BisnceCLPP S
1154
1.000
1,611
BlanceCL
BlanetLLPP S
0
DisncetUP-C SirtetuP---c.c.. Synert
1154
1.100
3,111
Figure 4: Visualized example of evaluation metrics comparison.
4.3. Ablation Study
Several ablation studies are conducted to separately verify the effectiveness of (1) each component score in BPM (2)
LLM instruction parsing. (3) our directional similarity.
To verify the editing quality from each perspective respectively, we further ask human annotators to separately rate the edited image quality for preservation, modification, size, and position, and report the componential evaluation alignment. (1) for preserva- tion and modification, we compared our scores with 1.2 and CLIPScore to validate the necessity of our explicit image re- gion decomposition, the results are in Tab. 5, S Spreserve and Smodkfy consistently outperform I.2 and CLIPScore, high- lighting that decomposing the image into editing relevant or irrelevant regions for separate consideration is crucial to acquire accurate assessment for modification and preser- vation, as it removes the interference of regions that owns opposite requirements; (2) for size and position, we report the alignment between human and our metrie evaluation for size and position change, as shown in Tab. 6. Our Sposition and Swize both yield relatively high alignment with human evaluation, indicating our design is effective in judging the position and size changing during editing.
Effectiveness of Component Scores:
Ground Truth Test performance is shown in Tab. 4, each row represents the metric favoring proportion for three types of images and sum up to 1, higher proportion for GT indi- cates less biased evaluation results. We can conclude that:
Effectiveness of LLM Instruction Parsing: we validate it from two aspects, the accuracy of editing region local- ization as well as the size and position changing judgment
8

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Basse Image Editing
Table Ablation for our directional Similarity. "CS" denotes vanilla CLIPScore for the cropped edited object region, "DS" de- notes our directional similarity. We report the alignment score hetween human and metric evaluation.
improve the modification performance. We also validate the quality enhancement through user study, asking users select better edited image from original and our enhanced edited images. Our method achieved 91 % of the samples being rated as better or equally good as the original, refer to Appendix for details.
IP2P va PTgge f DaLLE-2 vs IP2P1
DALLE-2 va Playe DALLE vs MGIEE
C5
0635
B619
0.79
0.536
18
0,659
16676
0L383
Table 8: Results for Editing Quality Enhancement.
alignment with instruction. To verify it, we randomly se- lect 100 data samples along with their edited region mask Morigins Medit and required position and size changing state Posst, Size,t, then manually judge (1) whether the M Morigin and Medit accurately locate the editing region. (2) whether the Pa and Size,r express the requirement in instruction. Finally, we yield 97% accurate editing region localization and 99% correct position and size changing requirement expression, validating the trustworthiness of our LLM instruction parsing, subsequently facilitating the ultimate editing evaluation.
Methods
Preservation Origin
Modification
Ours
Origin
Ours
TP2P
0.792
0.927
0.318
0.334
FT_IP2P
0.907
0.922
0.361
0.364
MGIE
0.876
0.923
0.327
0.341
4.5. Qualitative Results
We present qualitative results separately verify (1) BPM can comprehensive evaluating the edited images with trust- worthy scoring. In Fig. 4, we compare three editing images for identical input image and instruction. The edited im- age owns accurate editing within the region of source object "glass", as well as good content preservation in other irrelevant regions. Our BPM favors in terms of preserva- tion, modification well as overall quality, highlighting the alignment of our evaluation towards human judgment; (2) the edited region localization process can enhance edit- ing performance. In Fig. 5, our editing region mask Medit accurately locate the region should be edited, guiding the model to focus within it, reducing unnecessary modifica- tions in other regions, thus harvesting more high-quality editing results. More qualitative results are in Appendix.
Instruction: Change the conpater car
Instruction: televiston fish equartus.
Source Image
Original sdited Resslt Dur Editiey Moh Ma Fnharced Edited Ressit
Figure 5: Visualized example of image editing mask guidance.
Effectiveness of Directional Similarity: we compared our directional similarity with original CLIPScore, i.e., the image-text alignment of edited object. From the results in Table. 7 we see that directional similarity yields higher alignment with human evaluation, validating that directional similarity indeed implicitly incorporates the ver- ification of aditing-irrelevan attributes maintenance, thus providing more reliable reflection of editing quality.
5. Conclusion
In this paper, we propose a novel metric BPM tailored for instruction- based imaged editing, which explicitly disentan- gle the image into editing-relevan and irrelevant regions for separate evaluation according to their requirements. Two- tier evaluations perspectives, yielding comprehensive editing quality assess- ment. What's 's' more, the editing region localization in can be integrated into image-editing models to enhance the edited image quality, indicating its wild application. Experiments upon editing data with diverse instructions verify the effec- tiveness of BPM serving as a trustworthy evaluation metric.
conducted from region and semantic
4.4. Application as Image Editing Guidance
We adopt three advanced image editing models MGIE(Fu et al., 2023), FT (Zhang et al., 2024) and In- strucPix2Pix(Brook et al., 2022), and separately implement our enhancement in Sec. 3.5 for them. The performance is shown in Tab. 8. For both preservation and modification, steady alignment improvement are observed across all three editing models, indicating that by adopting our mask guid- ance, the model can focus on correct editing region, thus reducing unnecessary modifications in irrelevant regions to enhance the preservation performance, as well as harvesting more high-quality modification within editing regions to
Impact Statement
Image editing quality evaluation has broad impacts: while facilitating community development, it also raises privacy and misinformation concerns that require careful attention to ensure responsible deployment and mitigate risks.

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Intrruction-Basce Image Editing
Acknowledgements
Hui, M., Yang, S., Zhao, B., Shi, Y., Wang, H., Wang, P., Zhou, Y., and Xie, C. Hq-edit: A high-quality dataset for instruction-based image editing, 2024, URL https: www.wi.wwwwwas////////. . 09990.
This work was supported by the grants from National Nat- ural Science Foundation of China (62372014,6252522) 62132001, 62432001) and Beijing Natural Science Founda- tion (4252040, L247006).
Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
References
Basu, S., Saberi, M., Bhardwaj. S., Chegini, A. M., Mas- siceti, D., Sanjabi, M., Hu, S. X., and Feizi, S. Editval: Benchmarking diffusion based text-guided image editing methods. arXiv preprint 1rxii:23110024226. 2023.
Ju, X., Zeng, A., Bian, Y., Liu, S., and Xu, Q. Direct inversion: Boosting diffusion- based editing with 3 lines code. arXiv preprint arXiv:2310.01500 2023.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXhv:2210.092760 2022.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:221 .98000, 2022.
Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210. 14222 2022.
Kim, G. and Ye, J. C. Diffusionclli:: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2110.027111 2021.
Fu, T.J., Hu, W., Du, X., Wang, W. Y., Yang. Y., and Gan, Z. Guiding instruction based image editing via multimodal large language models. arXiv preprint arXiv:2309. 17102, 2023.
Kim, G., Kwon, T., and Ye, C. Diffusionclip: Text- guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2426-2435, 2022.
Gal, R., Patashnik, O., Maron, H., Chechik, G., and Cohen- Or, D. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00900 2021.
Kocasari, U., Dirik, A., Tiftikei, M., and Yanardag, P. Styleme: Multi-channel based fast text-guided image generation and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 895-904, 2022.
Gal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., and Cohen-Or Orh D. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Gruphics (TOG), 41(4):1-13, 2022.
Ku, M., Jiang, D., Wei, C., Yue, X., and Chen, W. Viescore: Towards explainable metrics for conditional image syn- thesis evaluation, 2024. URL https: /atos://areee //aaxiiivv.orr.////e abs 2212.212222 14867.
Han, Z., Jiang, Z., Pan, Y., Zhang, J., Mao, C., Xie, C., Liu, Y., and Zhou, J. Ace: All-round creator and editor following instructions via diffusion transformer. arXiv preprint arXiv:2410.00086 2024.
Li, J., Li, H., Erfani, S., Feng, L.., Bailey, J., and Liu, F. Visual-text cross alignment: Refining the similar- ity in vision-language models. arXiv preprint Stth:2406.029113. 2024.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-promp im- age editing with cross attention control. arXiv preprint arXiv:2208.01626 2022.
Liu, S., Zeng, 7.,, Ren, T., Li, F., Zhang, H., Yang. J., Li, C., Yang. J., Su, H., Zhu, J., al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499. 2023,
Hertz, A., Aberman, K., and Cohen-Or, D. Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2328-2337, 2023.
Lu, Y., Yang, X., Li, X., Wang. X. E., and Wang, W. Y. Limscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024.
Hessel, J., Holtzman, A., Forbes, M., Bras, I.., and Choi, Y. Clipscore: A reference-free evaluation metric for im- age captioning. arXiv preprint arXiv:2104.087. 2021.
Ho, J. and Salimans, T. Classifier-free diffusion guid- ance, 2022. URL https: /wtpiv.ivvvv.////////..///aaaav. org/ abs/
Meng, Y., Xia, M., and Chen, D. Simpo:
Simple preference optimization with a reference free reward. In Advances in Newral Information Processing Systems (NeurIPS), 2024.
2207. 12598.
10

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Inssrection-aased Image Editing
OpenAl. Gpt-4v(ision) system card, 2023.
pettss:/ccd.ooeee com/ Satam/ppper//GPPPP Systen_Card.pdff [Accessed: 22-09-2024).
OpenAl. Introducing gpt-4o: our fastest and most afford-
able flagship model, 2024, patps://openail com/ i nndex/hell 1aeelll--ppp-p-
gpt- 4o/ [Accessed: 22-09-2024).
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0003 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C.,, and Chen, M. Hierarchical text-conditional image generation with clip latents, arXiv preprint ar Xix::2204.006255 2022.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang. H., Li, F., Yang, J., Li, H.. Jiang, Q., and Zhang, L. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-reson righ-reesolutinn image synthesis with latent diffusion models, 2021.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image dif- fusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Wang, Z., Li, A., Li, Z., and Liu, X. Genartist: Multimodal Ilm as an agent for unified image generation and editing. Advances in Neural Information Processing Systems, 37:
128374-128395. 2024.
Xu, Z., Chen, Q., Peng, Y., and Liu, Y. Semantic-aware human object interaction image generation. In Forty- first International Conference Machine Learning, pp. 19065-19075. 2024.
Zhang, K., Mo. L., Chen, W., Sun, H.,, and Su, Y. Mag- icbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metrie. In CVPR, 2018.

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Basse Image Editing
In the appendix, first provide more information concerning method details, LLM prompts and user studies in Sec. A. Then we provide extra quantitative results for (1) impact of different scaling weight in Sec. B,1; (2) results of correlation between metrics and human evaluation score in Sec. B.2. (3) ablation study about the choice of LLM in Sec. B.3; (4) validation of LLM parsing result in Sec. B.4 (5) user study of masks generated by different pipeline in Sec. B.5 (6) more comprehensive component score ablation in Sec. B.6. (7) user study results for masking guidance enhancement Sec. B.7. Finally, we provide qualitative results to validate (1)the quality of our editing region mask(in See. C.I); (2) the effectiveness of our BPM evaluation (in Sec. C.2) (3) the effectiveness of our mask guided enhancement. Sec. C.3).
A. Method Details
A.1. LLM instruction parsing Prompts
The prompts for LLM instruction parsing is shown in Figure. 6. Our prompts include three parts, separately targets for object identify, size state and position state judgement.
A.2. User Study for Human Alignment Test
In this part, we provide details about the user study regarding the degree to which the metrics align with human ratings.
User Study Implementation Details: We invited 3 participants to rate the quality of the provided editing images through a questionnaire. We randomly select 100 image-instruction pairs from MagicBrush(Zhang et al., 2024), and adopt IP2PoBrooks al., (0222.. et al., 2023), DALLE-2(Ramesh et al., 2022), and FTTreer(Zhang et al.. 2024) to generate edited images. Participants were instructed rate images from 5 different perspectives, each score from I to 5, representing bad quality to perfect quality. Evaluation criteria are (1) modification quality, which assesses whether the editing accurately comply the instruction; (2) preservation quality: which assesses whether regions should not be edited is successfully preserved; (3) size accuracy, which assesses whether the editing region/object size change follows the instruction; (4) position accuracy, which assesses whether the editing region/object position change follows the instruction; (5)overall accuracy, which assesses the overall editing quality by considering above perspectives together. Detailed user study instruction and standards are shown in Fig. 7. We ultimately yield 960 user responses, which comprehensive enough to conduct the human alignment test. We also provide one screen shot for our user study page in Fig. 8.
A.3. User Study for Mask Guided Enhancement Quality
To analyze whether the editing performance have improved after adding mask guidance, we conducted preference tests with three human evaluators comparing images produced by the original models and those generated by models with mask guidance. We randomly selected 100 original image-instruction pairs. For each pair, randomly chose one model from IP2P (Brooks et al., 2022), MGIE (Fu et al., 2023), and FT (Zhang et al., 2024) to perform image editing, and compared it with the version that included mask guidance. The users were asked to choose the better image between the edited image produced by the original model and the edited image produced by the mask-guided model, or to consider them equivalent. The results from the three evaluators were then aggregated by majority vote; if all three chose differently, that sample was discarded. Detailed user study instruction and standards are shown in Fig. 9. We also provide one screen shot for our user study page in Fig. 10.
A.4. faize and posiiioo corner case discussion:
There exists some corner cases for size and position judge process: Fo) the add type instruction, where the source object in the original image is "None", fsize directly assigns full score to size state. (2)For the remove type instruction, where the target object in the edited image is "None", fynsition directly assigns full score to position state.
B. More Experimental Results
B.I. Ablation on Different Balance Factors
We conduct experiments with different weight factor a between Ssemantic and Spegions i.e., BPM = a Ssemantic + (1 (1-a) a)= * Sregion. the results are shown in Table. 9. Among all combinations, a = 0.7 reaches the best human evaluation alignment, verifying that higher yet moderate semantic scale would yield better evaluation results.
12

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for mnstructionBassed Image Editing
Given an instruction for image editing, output the parts of the original image that should be modified and the modified parts in the edited image compared to the original image. there no parts that should be modified, output None. Don't output the reasons. Here are some examples:
Object identify
1. Instruction:
"Replace the bike with bear." Output: Original image: bike. Edited image: bear.
Instruction: "Add bear." Output: Original image: None. Edited image: bear.
Instruction: "Remove the bike." Output: Original image: bike. Edited image: None.
Here instruction:
Given an instruction image editing, output what size changes of the objects involved, such as "larger", ; "smaller", "unchanged", Don't oulput the reasons.
Here are some examples:
Size state
1. Instruction: "give the bear bigger claws." Outpul: larger.
2. Instruction: "Make the hydrant blue." Output: unchanged.
3. Instruction: "Make the vase smaller." Output: smaller.
Here is the instruction:
Given an instruction for image editing and edited area the original and edited image, output the positional relationship between edited area in the edited image and the edited area the original image. The optional positional relationships arc: "left", "right", "up". "down", "inside", "unchanged", "near".
Here are some examples: 1. Instruction:
Position state
"put on the screen of the laptop". Edited area in original image: screen of the laptop. Edited area in edited image: Output: inside.
Instruction: "put bird on top of the tower". Edited original image: tower, Edited in edited image: bird. Output: up.
Instruction: "replace the cat with dog". Edited area original image: cat, Edited area edited image: dog. Output: unchanged.
Here is the instruction:
Figure 6: Prompts for parsing instruction.
Table 9: Ablation on different balance factors between S Seeanntic and Sregion. Experiment conducted on a 100-entries subset.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Human Alignment Score
0.302
0,423
0.598
0.659
0.738
0.793
0.819
0.822
0.776
0.769
0.761
B.2. Correlation between metrics and human evaluation score
We report metrics and human scores correlation for each single model in Tab. 10. Though our BPM yield highest correlation this format as well, such calculation is questionable as output from distinct instruction lacks comparability, and all metrics show performance not high.
The rationale behind our pair model preference calculation a) Reduce Subjectivity influence: Scores among users exhibit variance of 1.06(out of 0-5 range), thus using absolute values for single model correlation calculation may introduce bias. In contrast, employing relative preferences, by introducing reference, reduces subjectivity in user study to enhance the credibility of the results. b) Reduce cross-sample bias: Note our metric evaluates which model performs better for the same instruction, rather than comparing scores across different instructions, as we believe outputs from distinct instructions lack absolute comparability, as shown in Figure. 11. By pair-wise comparison on identical instructions, we enable more accurate model-to-modd comparisons explicitly than correlating scores from a single model's diverse outputs.
B.3. LLM Selection Influence
We alter different LLM to parse the instruction, and the performance comparison shown in Table. 11. From the result we can conclude that for the task of parsing instructions, current advanced LLM show similar alignment performance between metric and human evaluation. Specifically, Gemma-9B can achieve performance on par with GPT-4o.
13

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Bass Image Editing
Instruction
Please assess whether the edited image meets the editing requirements. The assessment is divided into five dimensions: overall editing quality, quality of the edited area, consistency of the non-edited area, whether the size of the edited object meets the editing instructions, and whether the position of the edited object is correct.
The assessment criteria are follows:
Overall editing quality: Evaluate the overall quality of the generated image, considering the realism of the edited image, whether the edits align with the editing instructions and appear natural, and whether the edited areas remain consistent with the original image.
Quality of the edited area: Assess whether the modified areas required by the instructions meet the textual editing instructions and appear natural.
Consistency of the non-edited area: Evaluate whether the areas that should not be edited remain
consistent with the original image.
Size: Whether the size of the edited object meets the editing instructions, For example, "make the fire hydrant blue" means that the size of the fire hydrant should remain unchanged; while "make the vase bigger" means that the vase should increase in size.
Position: Whether the position of the edited object is correct, For example, "make the hydrant blue," the position of the hydrant should remain unchanged; "put apple the table", the apple should be top of the table.
The scoring range is an integer from 1 to 5:
point: Very Poor
2 points: Poor
3 points: Average
4 points: Good
5 points: Very Good
Figure 7: The instructions for user study.
Editiing instruction:
Make the hydrant blue.
Original image::
image
Image
Image
Image
Ovenall eciting quality:
Oversll editnng quality:
Overal ediling qually:
Overall editing quality
Quality the edited area:
Quality of the edited area:
Quality of the edited area:
Quality of the edited area:
Consistancy of the non-editad area:
Consistency of non-edited area:
Consistancy of the non-edited area:
Consistincy of the nan-edled areac
Size:
Size:
Size:
Size:
Posttio::
Posttoo::
Position:
Position:
Figure 8: Screen-shot of our user study page.
14

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Bass Image Editing
Instruction
Please judge which the two edited images is better edited. The assessment is divided into five dimensions: overall editing quality, quality of the edited area, consistency of the non-edited area, whether the size of the edited object meets the editing instructions, and whether the position of the edited object is correct.
The assessment criteria are as follows:
Overall editing quality: Evaluate the overall quality of the generated image, considering the realism of the edited image, whether the edits align with the editing instructions and appear natural, and whether the non- edited areas remain consistent with the original image.
Quality of the edited area: Assess whether the modified areas required by the instructions meet the textual editing instructions and appear natural.
Consistency the non-edited area: Evaluate whether the areas that should not edited remain consistent with the original image.
Size: Whether the size of the edited object meets the editing instructions, For example, "make the fire hydrant blue" means that the size the fire hydrant should remain unchanged; while "make the vase bigger" means that the vase should increase in size.
Position: Whether the position of the edited object is correct, For example, "make the hydrant blue," the position the hydrant should remain unchanged; "put an apple on the table", the apple should be on top the table.
The scoring range an integer from -1 to
the left edited image is better than the right one points: equal
point:
-1 points: the right edited image is better than the left one
Figure 9: The instructions for user study of preference test.
Table 10: Correlation between metrics and human evaluation score on each single editing model.
MGIE
FT
IP2P
IP2P
DALLE2
GenArtist
ACE
Average
CLIP-T
0.114
-0.028
-0.146
0.079
0.280
0.075
0.062
CLIP-1
0.207
-0.069
0.201
0.018
0.632
0.260
0.208
DINO-Score
0.315
0.039
0.252
0.115
0.695
0.299
0.285
LPIPS
0.264
-0.034
0.276
0.153
0.735
0.198
0.265
L2
0.210
0,082
0.156
0,256
0.628
0.151
0.247
GPT-4o
0,512
0.464
0.491
0.338
0,425
0.430
0).43
BPM( Ours)
0.474
0.374
0.452
0.488
0.537
0.324
0.444
B.4. Validation of LIM Parsing Result
We randomly sampled 100 examples, used Gemma-9b for parsing, and manually evaluated the accuracy. As shown in Table. 12, gemma's parsing accuracy has reached over 97%. Specifically, the failure cases of the parsing of object identify "Could be eggs?" and "It could be just french fries." The errors occurred because the instructions contained "it" and since the LLM did not have information from the original image, mistakes. For the parsing of positions, the errors were due to the LLM''s difficulty in judging other spatial relationships that were not included in the types we provided, such as "among."
unable determine what "it" referred to, leading to
B.5. Mask Comparison
To quantitatively confirm the quality of our masks (i.e., edited regions), we randomly sampled 100 entries and asked 3 human evaluators to score the masks generated by different methods on a scale of I(worse) to 5(best). These methods include the bounding boxes generated by GPT-4o, the masks produced by DiffEdit(Couairo) et al., 2022), and the masks generated by our pipeline. As shown in Table. 13, our average mask quality score is 4.01, which is significantly higher than the other two methods. In addition to quantitative experiment, some visual results are presented Figure. 12.
15

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Bass Image Editing
Editing instruction:
Make the hydrant blue.
Original image:
Generated image 1:
Generated image 2:
Quality of the edited area:
Consistency of the non-edited area:
Size:
Position:
Overall editing quality:
Figure 10: Screen-shot of study page for preference test. Users asked to evaluate whether the left edited image is better than the right one from five dimensions.
B.6. Additional Component Score Verification
Due to limited space, the full component score verification shown in Table. 14 as complement for Table. 5, Our Spreserve and Smodify consistently yield better alignment towards preservation and modification compared to L2 and CLIPScore for each pair of editing model comparison.
B.7. User Study Results for Masking Guidance Editing Enhancement
The detail for this user study can be found in Sec, A.3. As shown in Table. 15, the results of user study indicated that for all evaluation dimensions, the number of mask-guided models that surpassed original models was significantly higher than those that performed worse. Notably, mask guidance provided remarkable improvement in maintaining regions outside the editing area, with 70% of the samples performing better than the original models.
C. Qualitative results
C.1. Visualization of Mask Quality Comparison
In Fig. 12, we compare our mask with GPT-4o generated bounding box and DiffEdit(Couairo et al., 2022) generated mask, our mask and bounding box can more accurately locate the editing region, providing more precise separation for evaluation, thoroughly yielding more trustworthy evaluation score.
16

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for mnstruction-Based Based Image Editing
Editing Instruction:
Into bow
Original Image:
image
image
image
image
Overal ociting qualit::
Overal ediiing quality:
Overall ectting quality:
Overal lciting quality:
Editing instruction:
Makve the television a fish aquarium
Original image:
Gonerated Image f:
Generated Image 2:
Gonerated Image 3:
Gonerated Image 4:
Overall eotting qualit::
Overal editigg qualiy:
Overail ectting quslit:
Overall edliing qualit:
Figure 11: Visualized of human score collection procedure. By offering references for identical input (Generated image 1-4 in each line), the annotators can judge the editing quality more reasonably. On the contrary, cross-sampled score collection lacks absolute comparison. For instance, scoring samples for two distinct instructions column (Generated image 2) is difficult to determine whether the "tie change case"(in first line) or "television change case"(in second line) is better in editing quality.
C.2. Visualization of BPM evaluation
More qualitative results of BPM evaluation is shown in Fig. 13. For each input image and instruction along with their different editing results, BPM can accurately verify the editing images' quality from different perspectives. For example, in Fig. 13(a), edited image 4 owns totally clean plate, which accurately follows the instruction "mask the plate empty", and the other region remains perfectly unchanged. BPM rate it with highest score among four editing images, validating the authenticity of BPM serving as evaluation metric.
C.3. Visualization for mask-guided enhancement
Visualization of mask guided enhancement in shown in Figure. 14. Our mask accurately locate the region that should be edited, and successfully maintain the content of irrelevant regions, yielding more satisfactory editing result.
17

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Base Image Editing
Instruction: Make the television a fish aquarium
Instruction: Turn off the computer
Instruction: Make the hydrant blue
Original Image
GPT-40
DiffEdit
(PMMOurs)
Figure 12: Visualization of mask comparison. We compare our editing region mask with two approaches, GPT-40(OpenAlL, 2024) and DiffEdit(Cuuairo et al., 2022). For GPT-4o, we prompt to output the bounding box coordinates of editing region; for DiffEdit, we visualize the attention map that serves as their mask guidance during editing process.
18

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Baree Based Image Editing
Table 11: The influence of the choice of LLM. Experiment conducted on a 100-entries subset.
mgie vs fppp
mgii vs ip2p
ft_ip2p vs ip2p
dalle2 vs ip2p
dalle2 vs fppip
dalle2 vs mgie
Average
gemma
0.761
0.700
0.831
0.866
0.823
0.773
0.793
gpt-4o
0.761
0.689
0.854
0.876
0.792
0.804
0.796
gemini
0.739
0.689
0.854
0.866
0.792
0.784
0.789
doopscek-v3
0.739
0.689
0.854
0.887
0.823
0.804
0.799
Table 12: The accuracy of instruction parsing.
Object identify
Size
Position
Accuracy
0.98
0.99
0.97
Table 13: Human ratings of mask generated by different methods.
GPT-4o
DiffEdiit
Ours
Human rating
2.66
1.45
4.01
Table 14: Effectiveness verification for component score of BPM.
The component scores BPM targets for different perspective evaluation of editing quality, we compare them with corresponding human evaluation(specife scores in the target perspective).
MGIE FTP
MGIE IP2P
IP2P FT
Dal IP2P
DALLE-2 vs FTTP
DALLE-2 MGIE
L.2
0.756
0.816
0.816
0.893
0.743
0.927
preservation
S Specseuee
0.729
0.839
0.829
0.893
0,757
0.927
CL.PP
0.617
0.645
0.624
0.484
0.479
0.515
modification
Smodify
0.678
0.686
0.659
0.67
0.606
0.583
Table 15: User Study for our Mask Guidance Enhancement. randomly select 100 image samples comparison. "Ours" denotes edited image under our mask guidance is preferred by user, "Equal" denotes no obvious difference between original and our edited image, "Origin" denotes original edited image is preferred.
Ours
Equal
Origin
Position
38
54
8
Size
23
75
2
Modification
38
48
14
Preservation
70
26
4
Overall
57
34
9
19

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction BasseectionBaaeed Image Editing
Original Image
Edited Imoge 1
Edited Image
Image
Image
Instruction: make the plate empty.
LPip5
0.158
0.102
0.063
0.086
L24
0.093
0.054
0.051
0.047
CLIPScore
0.180
0.159
0.159
0.144
GPT-4o
78.5
40.0
47.5
84.0
BlanceCUPPSS
0.128
0.110
0.071
0.283
BianceCUP-5, mrmm
0.865
0.915
0.914
0.964
BlancoCLIP-5 pernaoi
0.993
1.025
0.985
BlanceCLUP-S...
BlanceCLiP-S
HlancaCLI--suwnde
2.993
1.025
0.985
3.248
a
Original Image
Image
Edited Image
Edited Image
Edited Image
Instruction: Add car the screen.
LPIPS
0.687
0.051
0.534
0.052
L24
0.283
0.003
0.215
0.003
CLIPScore
0.189
0.167
0.187
0.167
GPT-4o1
25.5
77.5
$6.5
BlanceCLIP-S_... T 1
0.274
0.391
0.264
0.525
BlanceCUP-5 Spssone
0.342
0.952
0,482
0.973
BlanceCLIP-S,
0.617
1.343
0.746
BlanceCUIP-S...1
1
1
BlanceCLIP-5 Spotion
BlanceCUIP-S... T
1.617
3.343
2.746
3.499
(b)
Figure 13: Visualized Evaluation comparison among different metrics.
20

---

Balancing Preservation and Modification; A Region and Semantic-Aware Metric for Instruction-Bass Image Editing
Original image
Original edited image
Mask
Added mask guidance
"Let the bear smile"
"Add curtains the window"
"Have the wear party hat"
"Let the woman wear sweater"
"Turn the laptop orange"
Figure 14: More visualization of mask-guided editing.
21


Reviews:
[{"review_id": "02XfifzCQd", "reviewer": "Reviewer_xJQS", "summary": "The authors proposed a new metrics to balance the evaluation of preservation and modification for instruction-based image editing tasks. It is supposed to address some practical problems of image editing evaluation, by disentangling and measuring both the edited and untouched regions. The results show that the newly proposed metrics demonstrate a high correlation between human judgement and automatic numeric numbers. It is expected to have a higher impacts on image editing domains.\n\n## update after rebuttal\nI appreciate the efforts on designing the pipeline for evaluating the editing tasks automatically with a higher human alignment. The proposed method is a heuristic pipeline without learning, but it can serve as an intermediate solution which unblocks many emerging editing models, as an auxiliary metrics of user study. I also read the rebuttals and like the experiments on global editing tasks. Though not perfect and having some issues of complexity and internal errors, having it released in public will still help the community. So i will keep my original scores and encourage the authors to add these new results into the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our work that \n\"expected to have a high impacts on image editing domains\", \"claims are well evaluated with sufficient evidences\", \"human alignment testing is effective\". \n\n**1.Robustness of BPM on more diverse image editing tasks:** (1)Our BPM can apply to global image style transfer like weather change and environment change as well, we provide visualizations in [Fig](https://anonymous.4open.science/r/r-2/4.pdf). For quantitative evaluation, we conduct our experiments on the global editing subset of PIE-bench, which contains 160 samples. The results are shown in Table1. Our BPM yields the highest human alignment compared to other metrics, showing its effectiveness and wild application scope.\n(2) We want to clarify the focus of this paper is introducing new metrics for Instruction-Based Image Editing, specifically targeting edits that require clear semantic changes.  Suggested tasks like image enhancement or flare removal—which primarily improve image quality rather than alter semantics, fall outside the scope of our work.\n|dataset |PIEvs HQ_Edit $\\uparrow$| PIE vs FT\\_IP2P$\\uparrow$ | HQ_Edit vs FT_IP2P $\\uparrow$ |FT\\_IP2P vs IP2P $\\uparrow$  |IP2P vs HQ_Edit $\\uparrow$|IP2P vs PIE $\\uparrow$ |Average$\\uparrow$|\n|---|---|---|---|---|---|---|---|\nCLIP-I | 0.966 | 0.861 | 0.417 | 0.6 | 0.52 | 0.85 | 0.702|\nCLIP-T | 0.897 | 0.833 | **0.917** | **0.667** | 0.76 | 0.825 | 0.817|\nLPIPS | **1.0** | **0.917** | 0.375 | 0.467 | 0.48 | 0.9 | 0.690  |\nDINOScore | 0.966 | 0.833 | 0.25 | 0.533 | 0.52 | 0.9 | 0.667 |\nL2 | 0.897| 0.889 | 0.333 | 0.4 | 0.52 | **0.925** | 0.661 |\nGPT-4o |  0.759 | 0.722 |0.583 | **0.733** | 0.76 | 0.825 | 0.730 |\nBPM(Ours) | 0.931 | **0.917** | 0.875 | 0.6  | **0.84** | 0.85 | **0.836** |\n**Table 1**\n\n**2.Apply balanced factor between preservation and modification:** Thanks for the suggestions. We conduct experiments with different balanced factor $\\alpha$ between $S_{preserve}$ and $S_{modify}$, i.e., $S_{semantic} = \\alpha*S_{preserve} + (1-\\alpha)*S_{modify}$, the results are shown in Table2. \n Among all combination, $\\alpha=0.5$ reaches the best human evaluation alignment, verifying that moderate scale between preservation and modification yield better evaluation result.\n\n|$\\alpha$ |0.1|0.3|0.5|0.7|0.9|\n|---|---|---|---|---|---|\n| Human ALignment Score| 0.647 |  0.704 |  **0.792**| 0.768 |0.732 |\n**Table 2**\n\n\n**3.Distill our rules in MLLM to evaluate in a unified model for better impact:** As suggested, we select most advanced closed-sourced MLLM GPT-4o and open-sourced MLLM qwen-vl-max-latest for our rule distillation. The results are shown in Table3, indicate\n(1) describe our design rules as prompt to guide GPT-4o via chain-of-thought (\"GPT+rule\", Line 2) improve human alignment of original GPT-4o(Line 1) by 2.7\\%, showing our rules, that separately evaluate the preservation and modification with explicit region decomposition, are essential for comprehensive evaluation and can assist to yield more trustworthy results.\n(2) Though our evaluation rules can improve GPT-4o human alignment, it still falls short of our tailored pipeline(3.0\\% performance drop, Line 2,5). This originates from that our pipeline endows LLM and segmentation tools for accurate editing region localization and explicitly evaluating preservation, modification separately, while MLLMs, even the most advanced GPT-4o, is sub-optimal for such localization task, as verified in our Fig.11,Supp. What' more, we further provide GPT-4o with our generated mask(\"GPT+rule+mask\", Line 3) as condition for evaluation, but no significant performance improvement is observed, verifying that MLLM shows weaker evaluation interpretability than our pipeline and fail to fully utilize such accurate mask guidance. \n(3) Open-sourced MLLM Qwen(Line 4) lags behind in human alignment(12.4\\% performance drop compared to BPM). Besides, there is no large-scale ideal data for such role distillation training. The data should contain paired real and edited images along with human-annotated editing masks,  and heavy human annotation on the quality judgment for modification, preservation, size and position change, so we don't do any learning like fine-tuning it. (4) Our rules witness consistent improvement in MLLMs evaluation, showing its wild application and potential for generalization towards more unseen tasks like more diverse image editing. \n|dataset |MGIE vs FT_IP2P $\\uparrow$| MGIE vs IP2P$\\uparrow$ | IP2P vs FT_IP2P $\\uparrow$ |DALLE-2 vs IP2P $\\uparrow$  |FT_IP2P vs DALLE-2 $\\uparrow$| DALLE-2 vs MGIE $\\uparrow$ | Average$\\uparrow$|\n|---|---|---|---|---|---|---|---|\nGPT-4o | 0.693 | 0.678 | 0.64 | 0.814 | 0.781 | 0.804 | 0.735 | \nGPT-4o+rule | **0.784** | **0.744** | 0.73 | 0.876 | 0.625 | **0.814** | 0.762 | \nGPT-4o+rule+mask | 0.727|0.622|0.798|**0.918**|0.688|0.763|0.752|\nQwen|0.614|0.633|0.73|0.804|0.562|0.67|0.668|\nBPM(Ours)|0.761|0.7|**0.831**|0.866|**0.823**|0.773|**0.792**|\n**Table 3**", "claims_and_evidence": "The claims in the paper are well evaluated with sufficient evidences. \nFor example, the newly proposed scores are shown to have a higher correlation with human preference.", "methods_and_evaluation_criteria": "The proposed method covers many editing cases especially object addition / replacement / removal, mostly object-oriented. So it moslty measure how the object position / size is changed. Then It leverages precomputed masks to segment edited regions / unedited regions and measure each of them, to check the preservation.\n\nHowever, the metrics might not be robust for other types of editing including image enhancement, global image style transfer, soft editing like flare removal etc. Those non-mask based editing, or non-object involved editing may not be applied for measurement.", "theoretical_claims": "No theory in the paper.", "experimental_designs_or_analyses": "The authors did extensive research on LLM selection and masking methods. The human alignment testing seems very effective. \nPreservation seems easier to measure than modification since it is a simpler task and mask is supposed to largely improve the scores. Will it be better to apply a balanced factor between preservation and modification?", "relation_to_broader_scientific_literature": "The manually-designed heuristic rules seem effective in short-term. But for a better impact, a multi-modal LLM might be more useful when we can distill the rules and the complicated pipelines involving multiple models into a unified model finetuned from a LLM. The current method has its limitations of intractable errors within the pipeline, and not very generalized to unseen tasks. Also it does not do any learning.", "essential_references_not_discussed": "The reference list is sufficient to me.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hw4NeHmmjP", "reviewer": "Reviewer_N886", "summary": "This paper introduces BPM, a metric designed for instruction-based image editing. BPM explicitly separates images into editing-relevant and irrelevant regions, enabling more precise evaluation. It conducts a two-tier assessment from both regional and semantic perspectives to provide a comprehensive measure of editing quality. Additionally, BPM’s editing region localization can be integrated into image-editing models to enhance output quality, demonstrating its broad applicability.\n\n## update after rebuttal\nThe authors’ rebuttal has clarified my previous concerns. Taking into account the other reviewers' feedback and the authors' response, I choose to maintain my original evaluation (Weak accept).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our work that \"enabling more precise evaluation\", \"broad applicability\", \"paper is well written\", \"motivation is clear and easy to understand\" and \"contributes to the broader scientific literature\". \n\n**1.More concise figure of BPM pipeline:** Thanks for your valuable suggestions, according to which we provide a new version of Figure 2 in [Fig](https://anonymous.4open.science/r/r-2/3.png).\n\n**2.Inference speed analysis:** We provide inference speed comparison in Table7, Supp.Material..\n(1) Region-Aware and Semantic-Aware Judge are independent and can conduct in parallel, and will not affect inference speed.\n(2) our BPM has comparable inference speed ($\\textless$1s per image) with other metrics like CLIP-I, and largely shrink the time cost compared with MLLM-based evluation, such as GPT-4o(11.4s per image), showing efficiency.\n\n**3.Assigning different weights to $S_{semantic}$ and $S_{region}$:** We conduct experiments with different weight factor $\\alpha$ between $S_{semantic}$ and $S_{region}$, i.e., $BPM= \\alpha*S_{semantic} + (1-\\alpha)*S_{region}$, the results are shown in Table1. \nAmong all combinations, $\\alpha=0.7$ reaches the best human evaluation alignment, verifying that higher yet moderate semantic scale would yield better evaluation results.\n\n\n|$\\alpha$ |0|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1|\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| Human ALignment Score| 0.302| 0.423| 0.598 | 0.659| 0.738| 0.792| 0.819|  **0.822**| 0.776| 0.769| 0.761|\n**Table 1**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, as there are no theoretical claims made.", "experimental_designs_or_analyses": "I reviewed the experimental results presented in Tables 1, 2, 3, and 4, and they appear to be correct.", "relation_to_broader_scientific_literature": "This paper contributes to the broader scientific literature in several ways, particularly within the context of instruction-based image editing, semantic understanding, and region-specific modifications.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Paper Strengths:**\n\nThe paper is well written. The main motivation is clear and easy to understand.\n\n**Paper Weaknesses:**\n\n1. The overall pipeline of BPM contains excessive text and needs to be more concise and clear. For example, the relationship between Object Identify, Size State, and Position State in Figure 2 is unclear. Additionally, the excessive lines and arrows make it difficult to grasp the entire pipeline.\n2. The proposed BPM utilizes Region-Aware and Semantic-Aware Judges independently. Does this mean BPM operates more slowly? Have you tested the inference time of BPM and compared it with SOTA methods?\n3. It appears that BPM-overall is obtained by summing $S_{semantic}$ and $S_{region}$. Have you explored assigning different weights to $S_{semantic}$ and $S_{region}$ to improve the assessment?", "other_comments_or_suggestions": "None.", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ljv0VusWQH", "reviewer": "Reviewer_kMZ5", "summary": "This paper proposes a novel evaluation metric for instruction-based image editing. The paper identifies the weaknesses in commonly used metrics, which can only consider the image as a whole but fail to focus on local and modified parts. It proposes to extract the edited object and intended editing effect from the editing instruction using VLLM, and use a detector and segmentor to locate these objects from original and edited images. It then proposes to decompose each perspective of the editing and compare the position, shape, and semantics of the original and edited objects separately to determine the alignment of the editing to the instruction. The paper conducts an experiment on the MagicBrush benchmark and demonstrates that the proposed evaluation metric is more aligned to the human evaluation. It also applies the object mask extracted by the segmentor to the diffusion process, improving the overall editing result.\n\n# Update after rebuttal\nI am grateful to the author's clarification on the proposed metric working under global editing with experimental results. I also appreciate that authors have taken my concern regarding the Human Alignment Test into consideration and performed additional experiments on human evaluation correlation, and promise to include these to the final version. My concerns are largely solved. Therefore, I raised my rating to \"weak accept\".", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our work that \"claim is solid\", \"metric is novel\". \n\n**Limited scope of proposed metric, i.e., global editing and object attribute changing:** 1) Thanks for recognize our proposed metric suitable for local object-oriented editing. 2) As suggested, we apply our BPM to **global instruction-based editing** including style change, environmental change. a) *visualized results* is in [Fig](https://anonymous.4open.science/r/r-2/4.pdf). b)*Quantitative verification* we use the 160 global editing samples in PIE-bench for human alignment evaluation. Results are in [Tab](https://anonymous.4open.science/r/r-2/5.pdf). BPM yield highest human alignment for such global editing cases. c) *Comparison with GPT-4o(VLLM competitors):* While applicable, GPT-4o(HQ-Edit) underperforms our BPM in global editing. d) Unlike metrics like FID(conventionally used for style transfer) that rely on paired GT before-and-after images (“after” indicates target stye image), our proposed metric operates without this constraint, offering greater practicality, our metric operates without this constraint, offering greater practicality.3) Regarding **attribute changing**(a)*Clarify object identify step* for attribute changing cases, source/target object $o_1,o_2$ would affiliate corresponding attributes, like, \"yellow mug\" as $o_1$, \"blue mug\" as $o_2$ for \"turning yellow mug into blue\" b) *Attribute understanding* Our contribution is decompose image into editing-relevant and irrelevant regions for separate evaluation, allowing any CLIP variant to be applied. In our rebuttal, we test Long-CLIP-specialized in fine-grained attribute understanding-on a subset of 80 attribute-changing samples ([Tab](https://anonymous.4open.science/r/r-2/6.png)). Results show BPM’s evaluation capability for such cases can be improved with more advanced image encoder.\n\n**Benefits of Human Alignment Test on two models comparison rather than single model:** 1) As suggested, we report metric and human score correlation for each single model in [Tab](https://anonymous.4open.science/r/r-2/2.pdf). Though our BPM yields highest correlation in this format as well, such calculation is questionable as output from distinct instruction lacks comparability, and all metrics show performance not high.2)The rationale behind our pair model preference calculation is a) *Reduce Subjectivity influence*: Scores among users exhibit variance of 1.06(out of 0-5 range), thus using absolute values for single model correlation calculation may introduce bias. In contrast, employing relative preferences, by introducing a reference, reduces subjectivity in user study to enhance the credibility of the results. b) *Reduce cross-sample bias*: our metric evaluates which model performs better for the same instruction, rather than comparing scores across different instructions (as we believe outputs from distinct instructions lack absolute comparability). By pair-wise comparison on identical instructions, we enable more accurate model-to-model comparisons explicitly than correlating scores from a single model's diverse outputs.\n\n**Details of the detector and segmentor** We use Grounding-DINO 1.5 and SAM-large as our detector and segmentor respectively. Will add this in final version.\n\n**Corner case of $f_{position}$ and $f_{size}$** In [Fig](https://anonymous.4open.science/r/r-2/1.png), for object addition type instruction like \"add dogs to the right of the surfboard\",1) regarding $f_{size}$ (Line 8 of Alg.1, main text), since original object $o_1$ is None, $S_{size}$ is set as 1.2) regarding $f_{position}$ (Line 7 of Alg.1, main text), since $pos_{st}$ is \"right\", LLM would replace $o_1$ by the reference object surfboard, and judge whether $o_2$ dog obey \"right of\" $o_1$ surfboard. \n\n**Fairness of visualized comparison with other metrics:** (1)We provide more visualizations about the metric human alignment comparison [Fig](https://anonymous.4open.science/r/r-2/7.pdf). Across diverse types of editing instructions involving local attribute editing, object addition, global style transfer, our BPM can yield higher human alignment compared to GPT-4o and other metrics.    \n(2)this mask comparison experiment in Fig 11.Supp only aims to justify the design choice of proposed route to locate mask.\n\n**Apply mask into diffusion to for edit is not new.** Agreed, 1)Our main contribution is new image editing metric that includes a novel pipeline for precise edited-region mask acquisition.2)this application is not our core contribution. It aims to show BPM design principle has broader potential as a byproduct for image editing.\n\n**Add reference to CLIP directional similarity:** Agreed, we will add reference and clarify in related work, fix overclaim point in final version. Thanks for recognize the rest of the proposed metric is novel.\n\n**Typo in Eq.2:** Thanks. We will correct this typo and incorporate all other valuable non-technical writing suggestions in the final version.", "claims_and_evidence": "The first claim is the identification of the weaknesses of commonly adopted metrics, such as CLIPSCore, L2, etc. The analysis about this part is solid because CLIP or LPIPS indeed consider the image as a whole and cannot focus on one specific region. Therefore, this claim is solid. The ground truth test in Table 2 further substantiates this point. \n\nThe second claim is the proposed metric. It uses LLM to extract the original and edited objects and compare the position, size, and semantics of them separately, and it indeed addresses the problem of most previous metrics that consider the entire image as a whole. This is a solid contribution. However, the scope of the proposed metric is worth more discussion, and I will talk about this in the **Methods And Evaluation Criteria** section of the review. Moreover, In **Semantic Instruction Compliance**, the metric for measuring modified region $S_{modify}$ is the same as the CLIP directional similarity [1]. There is no reference to it, and $S_{modify}$  is claimed as a contribution in Ln432 and Table 5. This is an overclaim. Despite this, the rest of the metric is novel. \n\nThe effectiveness of the proposed metric is shown by Tables 1, 2, 3, and 4. They show that the proposed metric aligns better to human evaluation over the baseline metrics such as CLIPScore or LPIPS. However, its benefit over its VLLM competitor HQ-Edit, labeled as GPT-4o in the main paper, is worth more discussion, which I will talk about in detail in the **Experimental Designs Or Analyses**. \n\nThe third claim is that inserting the object mask extracted by the segmentor to the classifier-free guidance of the diffusion process. The idea of locating to-be-edited objects to alleviate undesired effects is not new. Works like Prompt2prompt [2] and DiffEdit [3] use the attention map in the diffusion model to locate to-be-edited objects. In this work, object masks are obtained using an external segmentor, following the same path. The effect is demonstrated in Table 6.\n\n\n[1] Kim, Gwanghyun, Taesung Kwon, and Jong Chul Ye. \"Diffusionclip: Text-guided diffusion models for robust image manipulation.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2426-2435. 2022.\n\n[2] Hertz, Amir, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. \"Prompt-to-prompt image editing with cross attention control.\" arXiv preprint arXiv:2208.01626 (2022).\n\n[3] Couairon, Guillaume, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. \"Diffedit: Diffusion-based semantic image editing with mask guidance.\" arXiv preprint arXiv:2210.11427 (2022).", "methods_and_evaluation_criteria": "My primary concern about the proposed metric is its limited scope. I recognize that the proposed metric is particularly suitable for local object-oriented editing such as adding, removing, transforming, and removing objects. However, the proposed method may not address the editing on the global scale, such as changing the style of the image or environment of the scene. Throughout the paper, global editing is not mentioned, and it is not in the benchmark used in the experiment. Its main VLLM-based competitors, Aligmence and Coherence in HQ-Edit, instead can address the evaluation of global editing but also have a decent performance in local editing at the same time, as shown in Table 1, where GPT-4o (HQ-Edit) ranks at the second place. \n\nMoreover, even if we limit the scope to local object-oriented editing, my second concern is about the editing involving changing the object's color or texture rather than the object's category or size. According to Figure 6 Object identify, the LLM only extracts the object category from the instruction. If I have an instruction, \"Change the color of the mug to blue\", the original image and edited image will both be \"mug\", and the color change will not be reflected by $S_{modify}$ because $o_{1}$ and $o_{2}$ will be the same. In addition, CLIP is known to be good at identifying object category but not color or texture. This further limits the scope of the proposed evaluation metric.", "theoretical_claims": "This paper is an application paper with little theoretical contribution. However, I do notice an error in Equation 2, which is the standard classifier-free guidance formulation for image editing proposed in instructpix2pix [1]. The third line of the equation should be $s_{T} * (\\epsilon_{\\theta}(z_{t}, t, I_{origin}, T_{edit})-\\epsilon_{\\theta}(z_{t}, t, I_{origin}, \\varnothing)*M_{edit}$. I believe it is due to a typo. However, if it is not a typo, justification needs to be shown for the change.", "experimental_designs_or_analyses": "As discussed in the **Method and Evaluation Criteria** section, the benchmark only consists of local object-oriented editing without global editing. I am worried that this may unfairly benefit the proposed metric over other metrics, especially over the GPT-4o baseline, in the experiment because the proposed metric is specifically designed for local editing. It competes against other metrics in its strong suit. Experiments on global editing are required to establish a clearer insight into the proposed metric.\n\nRegarding the Human Alignment Test, it reports a metric's alignment to the human evaluation on choosing the better one from two given models. Why choose this format over the traditional correlation between metric score and human evaluation score on one single model? What are the benefits of the current format?\n\nThe experiment part fails to mention the details of the detector and segmentor used in the proposed method.", "relation_to_broader_scientific_literature": "The paper concerns the evaluation of instruction-based image editing, which is a longstanding challenge in generative AI. The idea of this work comes from the analysis of the prevalent metrics, realizing that they fail to focus on the local part in editing.", "essential_references_not_discussed": "The literature review contains all essential works to understand the key contributions of the paper.", "other_strengths_and_weaknesses": "All strengths and weaknesses are discussed in the previous sections.", "other_comments_or_suggestions": "There are several non-technical errors in the paper:\n\n1. There are repetitive sentences at the end of the literature review **Conventional Image Editing Quality Evaluation Metrics**. Sentences between Ln116-Ln122 and sentences between Ln123-Ln129 talk about the same thing. \n\n2. The sentence between Ln142-Ln146 is confusing with grammar mistakes.\n\n3. The citation to MGIE is incomplete. The paper title and publication details are missing.", "questions_for_authors": "I will consider upgrading my rating if my concerns can be addressed. They are summarized as:\n\n1. The limited scope of the proposed metric.\n2. Missing experiments on global editing. \n3. Benefits of the Human Alignment Test on two models over correlation to human evaluation score on one single model.\n4. Details on the detector and segmentor. \n5. The corner case on $f_{size}$ and $f_{position}$.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRoHe8n11W", "reviewer": "Reviewer_PHma", "summary": "This paper presents BPM, a new image quality assessment metric designed to address the biases of existing metrics in image editing, particularly regarding the preservation and modification areas, as well as their narrow considerations. BPM introduces a region-based approach by using masks to divide the image into preservation and modification areas, evaluating them separately across multiple dimensions, including position, size, preservation, and modification. Specifically, the calculation process of the metric minimizes the involvement of models, reducing biases introduced by the priors of other models. Experimental results show that, compared to existing metrics such as CLIPScore and LPIPS, BPM is closer to human subjective evaluations. Furthermore, incorporating BPM into model training can effectively improve the accuracy of edits.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our work that \"viewpoints are supported by experimental evidence\", \"theoretical analysis is detailed\", \"experiments is reasonable with clear logical progression and evidence\"\n\n**1.Verification on larger dataset:**(1) As suggested, we conduct experiments on a larger dataset. We curate a dataset with a total of 1000 samples, which is currently the largest for image editing metric evaluation. Images are randomly selected from MagicBrush and edited by 6 advanced editing models, which include the reviewer suggested ACE and Genartist model. \nThe results on the newly expanded dataset is shown in Table1. Our BPM yield highest human alignment score on this dataset as well, validating its effectiveness and robustness.(2)Although our BPM metric calculation is fully automated, the user study required for human alignment test hinders further dataset scale-up.\n|dataset |MGIE vs FT_IP2P $\\uparrow$| MGIE vs IP2P$\\uparrow$ | IP2P vs FT_IP2P $\\uparrow$ |DALLE-2 vs IP2P $\\uparrow$  |FT_IP2P vs DALLE-2 $\\uparrow$| DALLE-2 vs MGIE $\\uparrow$ | ACE vs GenArtist $\\uparrow$ |Average|\n|---|---|---|---|---|---|---|---|---|\nCLIP-I | 0.59 | 0.581 | 0.58 | 0.708 | 0.663 | 0.766 | 0.757 | 0.663|\nCLIP-T | 0.538 | 0.477 | 0.519 | 0.375 | 0.407 | 0.34 | 0.608 | 0.466|\nLPIPS | 0.677 | 0.605 | 0.691 | 0.711 | 0.628 | 0.766 | 0.838 | 0.709 |\nDINOScore | 0.59 | 0.581 | 0.63 | 0.781 | 0.733 | 0.798 | 0.784 | 0.699|\nL2 | **0.705** | 0.628 | 0.691 | 0.698 | 0.605 | 0.734 | 0.824 | 0.697 |\nGPT-4o |  0.667 | **0.674** |0.790 | 0.906 | 0.779 | 0.83 | 0.703 | 0.764|\nBPM(Ours) | 0.679 | 0.616 | **0.827** | **0.938** | **0.814** | **0.904** | **0.824** | **0.800** |\n**Table 1**\n\n**2.Rationality behind $f_{position}$ and $f_{size}$:**(1) Our intension is that previous metrics evaluate the entire image, thereby overlooking region-specific semantic requirements and failing to assess whether the fine-grained state changes of edited objects align with instructions. Size and position serve as typical state changes, thus we design corresponding function for their evaluation. (2) regarding how to evaluate the size and position change to make it more generalisable,  we also compare GPT-4o that directly judge whether the editing obeying corresponding state change demanding. The results are shown in  Table 2, our tailored functions $f_{size}$ and $f_{position}$ yield more human-aligned judgment, validating the trustworthiness.\n|dataset |MGIE| IP2P | DALLE-2|FT_IP2P|Average|\n|---|---|---|---|---|---|\n|$S_{size}$| | | | | |\n|GPT-4o | **0.92** |**0.90** | 0.93 | 0.90 |0.91|\n|BPM(Ours)| 0.89 | 0.89 | **0.93** | **0.92** |**0.91**|\n|$S_{position}$| | | | | |\n|GPT-4o | **0.71** | 0.69 | 0.89 | 0.75 |0.76 |\n|BPM(Ours)| 0.69 | **0.75** | **0.92** | **0.90** |**0.82**|\n**Table 2**\n\n**3.More experimental results on BPM-based editing images:** Apart from BPM evaluation, we further provide following results:\n(1)_User study:_ we conduct user preference tests, which is provided in Table12 of Supp.M. BPM-based editing yield 91\\% of the samples being rated as better or equally\ngood as the original compared to original ones, validating BPM serves as effective editing quality enhancement. \n(2) _Evaluation with other metrics:_ We also evaluate images by other metrics, which is shown in Table3. As shown, our BPM-based editing images yield the better performance for all these metrics as well, validating the robustness of BPM quality enhancement.\n|Metrics |CLIP-I $\\uparrow$|LPIPS$\\downarrow$ | DINOScore$\\downarrow$ | L2$\\downarrow$|CLIP-T$\\uparrow$|GPT-4o$\\uparrow$|\n|---|---|---|---|---|---|---|\n| IP2P  | 0.773 |0.315 |0.854 | 0.147 | 0.173 |0.559|\n|IP2P+BPM(Ours) | **0.923** | **0.103** | **0.997**|**0.047**|**0.176**|**0.667**|\nFP_IP2P |0.870 | 0.160|0.945 | 0.071 | 0.175 | 0.673 |\nFP_IP2P + BPM(Ours) | **0.911**|**0.135** | **0.951**|**0.063**| **0.175**| **0.720**|\nMGIE | 0.856 | 0.218 | 0.921 | 0.102 |0.176 | 0.613 |\nMGIE+BPM(Ours)|**0.895**|**0.124**|**0.950**|**0.065**|**0.177**|**0.701**|\n**Table 3**\n\n**4.Inclusion with more related works:**(1)Suggested works ACE and GenArtist focus on new  framework for image generation and editing, but they all use conventional metrics for evaluation. In contrast, our main contribution is the new evaluation metrics.\n(2)Experimental comparisons of testing them using our new metrics are included in  Table 1, and we will add relative discussion in related work in final version.", "claims_and_evidence": "The claims made in paper provides a thorough analysis of the issues with existing indicators, and the proposed viewpoints are supported by experimental evidence. BPM effectively maintains a balance in evaluation between the preservation and modification areas. However, the sample size used in the experiments is only 100, which lacks sufficient persuasive power.", "methods_and_evaluation_criteria": "The BPM proposed in the paper draws on the region-based concept while considering the impact of instructions on the preservation and modification areas, which is reasonable for the current problem. The diverse evaluation metrics compared on dataset A are robust, as the reference metrics include a wide range of multi-category evaluation indicators, such as CLIP-Score, GPT-4o, and LPIPS.", "theoretical_claims": "The theoretical analysis is relatively detailed, but the scoring design for position and size evaluation is unreasonable. There is no further in-depth explanation of the rationality behind the design of the metrics.", "experimental_designs_or_analyses": "The organization of the experiments is reasonable, with clear logical progression and evidence between them. The comparative and ablation experiments successfully demonstrate the advantages of BPM over other metrics. However, the limited experimental evaluation metrics in Section 4.4 fail to convincingly support the claims made in Section 3.5. In addition, the relatively small dataset in other experiments makes it difficult to provide strong persuasive power.", "relation_to_broader_scientific_literature": "The paper builds the metric design based on the existing popular region-based ideas in the field of image editing.", "essential_references_not_discussed": "[1] Han Z, Jiang Z, Pan Y, et al. ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer[J]. arXiv preprint arXiv:2410.00086, 2024.\n\n[2] Wang Z, Li A, Li Z, et al. Genartist: Multimodal llm as an agent for unified image generation and editing[J]. Advances in Neural Information Processing Systems, 2025, 37: 128374-128395.", "other_strengths_and_weaknesses": "Pros:\n\n1.\tBPM excels in maintaining and modifying the balance of regions compared to other existing metrics, and the results in Table 2 of the paper reveal the underlying reasons.\n\n2.\tBPM, when applied to model optimization, demonstrates more precise editing effects.\n\n3.\tThe impact of multi-class LLM priors on BPM is discussed, and the results in Table 8 indicate that the BPM calculation process effectively reduces the influence of LLM priors.\n\n4.\tThe writing and experimental organization are clear and make complex concepts understandable.\n\nCons:\n\n1.\tThe dataset used in the experiment is relatively small, making it difficult to provide sufficient persuasive power.\n\n2.\tThe scoring design for position struggles to handle complex orientations.", "other_comments_or_suggestions": "If the paper could conduct experiments on a larger dataset, its persuasive power would be stronger.", "questions_for_authors": "1.\tCould the authors provide experimental results on a larger dataset? \n\n2.\tCould the authors further elaborate on the design rationale behind the position and size evaluation process?\n\n3.\tCould the authors provide more experimental results in Section 4.4, including the performance of the edited images generated by the BPM-trained model on other metrics?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The final rating this paper is 4 weak accept.

Before rebuttal,

Reviewers acknowledged that 1) paper is well written and motivation is clear and easy to understand; 2) BPM excels in maintaining and modifying the balance of regions compared to other existing metrics. The main concerns from reviewers are: 1) the metrics might not be robust for other types of editing including image enhancement, global image style transfer, soft editing like flare removal etc; 2) The current method has its limitations of intractable errors within the pipeline, and not very generalized to unseen tasks; 3) proposed metric is its limited scope; 

After rebuttal, all reviewers confirmed that they had read the author's response to the review and would change review if needed. Reviewer kMZ5 raised rating to "weak accept" and all other reviewers kept their rating. Reviewer xJQS mentioned "Though not perfect and having some issues of complexity and internal errors, having it released in public will still help the community. So i will keep my original scores and encourage the authors to add these new results into the paper." Reviewer N886 suggested "The authors’ rebuttal has clarified their previous concerns"

Based on these AC decided to give weak accept as final rating for this paper. But hope authors could incorporate reviewers' comments in their final version.
Decision: Accept (poster)