VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation (2024)

Abstract: In the rapidly advancing field of conditional image generation research, challenges such as limited explainability lie in effectively evaluating the performance and capabilities of various models. This paper introduces VIESCORE, a Visual Instruction-guided Explainable metric for evaluating any conditional image generation tasks. VIESCORE leverages general knowledge from Multimodal Large Language Models (MLLMs) as the backbone and does not require training or fine-tuning. We evaluate VIESCORE on seven prominent tasks in conditional image tasks and found: (1) VIESCORE (GPT4-o) achieves a high Spearman correlation of 0.4 with human evaluations, while the human-to-human correlation is 0.45. (2) VIESCORE (with open-source MLLM) is significantly weaker than GPT-4o and GPT-4v in evaluating synthetic images. (3) VIESCORE achieves a correlation on par with human ratings in the generation tasks but struggles in editing tasks. With these results, we believe VIESCORE shows its great potential to replace human judges in evaluating image synthesis tasks.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12268–12290 August 11-16, 2024 ©2024 Association for Computational Linguistics
In the rapidly advancing field of conditional image generation research, challenges such as limited explainability lie in effectively evaluating the performance and capabilities of various models. This paper introduces VIESCORE, a Visual Instruction-guided Explainable metric for evaluating any conditional image generation tasks. VIESCORE leverages general knowledge from Multimodal Large Language Models (MLLMs) as the backbone and does not require training or fine-tuning. We evaluate VIESCORE on seven prominent tasks in conditional image tasks and found: (1) VIESCORE (GPT4-o) achieves a high Spearman correlation of 0.4 with human evaluations, while the human-to-human correlation is 0.45. (2) VIESCORE (with open-source MLLM) is significantly weaker than GPT-4o and GPT-4v in evaluating synthetic images. (3) VIESCORE achieves a correlation on par with human ratings in the generation tasks but struggles in editing tasks. With these results, we believe VIESCORE shows its great potential to replace human judges in evaluating image synthesis tasks.

heading: 1 Introduction
text: Diffusion models have become a focal point in AI research for image synthesis. Over the past year, several new models (Kumari et al., 2023; Ruiz et al., 2023; Li et al., 2023c; Zhang and Agrawala, 2023) have been introduced to enhance control over image generation. However, comprehensively evaluating AI-synthesized images remains a challenging and unresolved issue. While metrics like LPIPS (Zhang et al., 2018), CLIP-Score (Hessel et al., 2021), and DreamSim (Fu et al., 2023b) were proposed, they have certain limitations: (1) these metrics are agnostic the end task, which can fail to measure the desired aspects of the generated images, (2) the score is opaque with limited explainability. These limitations heavily restrict their effectiveness in assessing conditional image generation. Some research work (Denton et al., 2015; Isola et al., 2017; Meng et al., 2021; Chen et al., 2023; Sheynin et al., 2023) relied on human-driven evaluation methods. While humans excel at understanding and interpreting visual content, such methods in the context are facing challenges such as scalability limits and preference subjectivity issues. This reliance on
12268
human judgment highlights the need for more uniform evaluation methods in the field. To solve the mentioned issues, we formulate the problem definition with our desired properties, as presented in equation 1. The function f takes an instruction I , a synthesized image O, and C∗ which is a set of conditions (e.g. style, subject, background, cannyedge, etc). The score function should produce the intermediate rationale in the form of natural language before generating the final score according to the prompt instruction I:
fVIE(I,O,C ∗) = (rationale, score) (1)
The function f can be any Multimodal Large Language Model (MLLM) such as GPT-4 (OpenAI, 2023) and LLaVA (Liu et al., 2023a), which can take input images to generate human-like text responses. Unlike the automatic metrics, MLLM can receive human instructions and produce rationale. With such motivation, we introduce VIESCORE (Visual Instruction-guided Explainable Score), a framework to assess synthetic images in different conditional image generation tasks. VIESCORE has multiple advantages compared to auto-metrics and human evaluation. It includes: Task Awareness. Existing metrics were often designed to measure a certain aspect of generated images. For example, LPIPS measures the perceptual similarity of a pair of images, while CLIP-Score measures the text alignment of one single image. As a consequence, these metrics cannot be adapted to evaluate other tasks. VIESCORE acts as a silver bullet to tackle all conditional image generation evaluation processes due to its instruction-guiding property. It can be carefully adjusted with different instruction requirements. Explainability. The existing metrics normally output a single float-point score, which cannot offer detailed insights into the ’rationale’ behind its evaluations. Such a score makes it difficult to interpret the decisions from the metric output. Instead, VIESCORE can offer the rationale in the form of natural languages to help humans understand the reasoning process. As depicted in Figure 1, the rationale can significantly improve the trustworthiness of VIESCORE.
While the ultimate goal is to derive an MLLM that can rate images like humans, in this paper we also explore how well MLLMs can assess synthetic images compared to human evaluation and present insights and challenges on state-of-the-art MLLMs towards human evaluators, as shown in Figure 2.

heading: 2 Related Works

heading: 2.1 Conditional Image Synthesis
text: With recent advancements in Image Synthesis research (Goodfellow et al., 2016; Ho et al., 2020; Dhariwal and Nichol, 2021), researchers proposed different methods and contributed a large amount of controllable image synthesis models with conditional inputs. Conditional image synthesis can be categorized into conditional image generation and conditional image editing tasks. Prevalent tasks include Text-To-Image generation (Saharia et al., 2022; Rombach et al., 2022; stability.ai, 2023) (known as text-guided image generation), Inpainting (Avrahami et al., 2022; Lugmayr et al., 2022) (known as mask-guided image editing) and Textguided image editing (Brooks et al., 2023; Couairon et al., 2022; Wu and la Torre, 2023).
More recent works proposed new tasks such as Subject-driven image generation and editing (Gal et al., 2022; Ruiz et al., 2023; Li et al., 2023c) to inject one specific subject into a synthesized image, while Multi-concept image composition (Kumari et al., 2023; Liu et al., 2023b) allows multiple specific subjects into the synthesized image. Controlguided image generation (Zhang and Agrawala, 2023; Qin et al., 2023) allows additional conditions alongside the text prompt to guide the image synthesis. Our work uses MLLM to access all the discussed tasks on synthetic image evaluation.

heading: 2.2 Synthetic Images Evaluation
text: Various metrics are proposed to evaluate the quality of AI-generated images. Traditional measures like the Inception Score (IS) (Salimans et al., 2016) and the Frechet Inception Distance (FID) (Heusel et al., 2017) are commonly employed to measure image fidelity. On the other hand, to measure the alignment between the generated image and the text prompt, several metrics (Kim et al., 2022; Kynkäänniemi et al., 2019; Park et al., 2021; Sajjadi et al., 2018) have been introduced. The CLIP score (Hessel et al., 2021) and BLIP score (Li et al., 2022) are the most commonly used. Recently, approaches such as (Cho et al., 2023) and (Lu et al., 2023c) aim to provide a fine-grained evaluation framework, while the HEIM-benchmark (Lee et al., 2023) assesses text-to-image models across multiple aspects, such as toxicity and safety. Other methods, such as projective-geometry (Sarkar et al., 2023), evaluate images’ physical and geometric realism. However, these metrics are primarily fo-
cused on text-to-image generation and remain narrow in scope. General image generation tasks like subject-driven image generation and image editing (Ruiz et al., 2023; Li et al., 2023c) still lack effective automatic metrics. One traditional, yet effective method to evaluate AI-generated image performance is to have human annotators assess visual quality. Recent works like ImagenHub (Ku et al., 2023), and HEIM (Lee et al., 2023) attempt to standardize human evaluation across various image generation tasks, though scalability remains a challenge. Our research aims to identify the challenges in mimicking human perception in synthetic image evaluation and address these gaps by developing auto-metrics that align with human judgment across common image evaluation tasks.

heading: 2.3 Large Language Models as Evaluators
text: Large language models (LLMs) are often used to evaluate the quality of model-generated outputs. Recent works used LLMs as an evaluator demonstrating their great ability in text generation evaluation (Zheng et al., 2023; Dubois et al., 2023). This ability for evaluation naturally emerges (Fu et al., 2023a) and stems from LLM’s great reasoning ability and instruction-following ability. Recent works also tried to devise a smaller but explicitly fine-tuned LLM(Touvron et al., 2023) that achieves similar evaluation results on natural language generation (Xu et al., 2023; Jiang et al., 2023; Li et al., 2023b). Besides text evaluation, LLMs with visual features have been used as evaluators on images (Lu et al., 2023d; Huang et al., 2023; Lee et al., 2020; Inan et al., 2021; Hu et al., 2023), mainly focused on evaluating text-to-image task. GPT-4v, regarded as the state-of-the-art LLM with visual features, also reported a decent ability on image evaluation, especially in text-image alignment (Zhang et al., 2023b). However, the GPT-4v is not perfect for image evaluation. A comprehensive study on GPT-4v’s vision ability reported that GPT-4v makes mistakes on image evaluation tasks (Yang et al., 2023). For example, it failed to provide proper reasonings for spotting the difference between two similar images.

heading: 3 Preliminary

heading: 3.1 Evaluation Benchmark
text: ImagenHub (Ku et al., 2023) is a standardized benchmark for evaluating conditional image generation models with human raters. The framework
covered mainstream tasks, including image generation, editing, and several conditioned tasks. In this section, we visit how humans assess images in the ImagenHub framework. Images are rated in two aspects: (1) Semantic Consistency (SC) assesses how well the generated image aligns with the given conditions, such as prompts and subject tokens, ensuring coherence and relevance to the specified criteria according to the task. (2) Perceptual Quality (PQ) evaluates the extent to which the generated image appears visually authentic and conveys a sense of naturalness.
ImagenHub curated a human evaluation dataset for each task, in which the dataset contains around 100 to 200 conditional inputs for generating synthesized images. Then each image was rated by three human raters according to the guidelines of the defined task, and a final score in the range [0.0, 1.0] was reported for the average score in semantic consistency (SC) and perceptual quality (PQ) respectively, with another overall score (O) derived from the geometric mean of semantic consistency and perceptual quality at the instance level. ImagenHub covered 30 image synthesis models and reported 0.4 Krippendorff’s alpha on the inter-worker agreement of their human rating.

heading: 3.2 Multimodal Large Language Models
text: Multimodal Large Language Models (MLLMs) typically denote LLMs with integrated visual capabilities (Yin et al., 2023). This visual proficiency opens up the potential to perform image analysis and evaluation. However, for a comprehensive assessment of synthetic images, multiple images may be examined in one pass due to complex conditions. The prompt will also be extensive to comprehensively describe the rating process. Therefore, the MLLM candidate should possess specific capabilities: (1) The model must efficiently process and interpret multiple images simultaneously. (2) The model needs to comprehend and respond to lengthy text prompts while matching all requirements.
Recent popular open-source MLLMs, including LLaVA (Liu et al., 2023a), InstructBLIP (Dai et al., 2023), Fuyu (Bavishi et al., 2023), and CogVLM (Wang et al., 2023), can only accept a single image as input along with text instruction. To feed multiple images, a workaround is to merge and concatenate multiple images horizontally and feed as one image. More recent MLLMs such as Open-Flamingo (Awadalla et al., 2023), Kosmos2 (Peng et al., 2023), and QwenVL (Bai et al., 2023)
can accept multiple images in an interleaved imagetext format. For closed-source MLLMs, OpenAI’s GPT-4v (OpenAI, 2023), GPT-4o (OpenAI, 2023) and Google’s Gemini (Team et al., 2024) are the most popular MLLMs that perform exceptionally.

heading: 3.3 Existing Auto-metrics
text: Here we list some prominent automatic metrics: Image-Text Alignment. CLIP-Score (Hessel et al., 2021) computes the average cosine similarities between prompt and generated image CLIP embeddings. One disadvantage of CLIP-Score is that the score is biased towards the training distribution (Kim et al., 2023). Moreover, in practical evaluation, the average CLIP-Score result of a decent method will always fall in the range [0.25, 0.35] even though a single CLIP-Score is within [0, 1]. Such a narrow range may not offer enough differentiation to know which model is better. Moreover, image-text alignment is not the only considered aspect of semantic consistency. For example, it cannot examine the degree of overediting in text or mask-guided image editing tasks.
Perceptual Distance. LPIPS (Zhang et al., 2018) measures the resemblance between two images in a manner that aligns with human perception. With its sensitivity to distortions, it is an often used metric in image synthesis research such as image editing tasks and control-guided Image Generation task (Meng et al., 2021; Qin et al., 2023), to measure between the input (or ground truth) and the generated image. However, in the image editing context, the image’s naturalness (e.g. shadow, lighting, sense of distance) is often required in the human perspective of perceptual quality, which is missed in the LPIPS metric. It is also difficult to access a model’s performance by distortion level, as in the current state of research the models often can process high-quality editing without artifacts.
Subject Fidelity. CLIP-I computes the average pairwise cosine similarities between CLIP embeddings of generated and real images, first proposed in Textual-Inversion (Gal et al., 2022). However, CLIP-I cannot distinguish between different subjects that may have highly similar text descriptions, and it is less sensitive to shape consistency as it compares the semantic similarity between images. DINO metric was proposed in DreamBooth (Ruiz et al., 2023). The metric is computed by the mean cosine similarities calculated pairwise between the DINO embeddings of ViT-S/16 (Caron et al., 2021)
for both synthesized and authentic images. In contrast to CLIP-I, the DINO metric is sensitive to differences between subjects of the same class due to the self-supervised training objective of DINO. These two became popular metrics reported in research on subject-driven image generation and editing tasks (Li et al., 2023c; Lu et al., 2023a).

heading: 4 Method
text: During the experiment, we select 29 models evaluated in ImagenHub (Ku et al., 2023) to compare the correlations with human ratings. See Appendix B for the listed models. Rating instructions. In ImagenHub, each image in one rating aspect is rated by picking an option from List[0, 0.5, 1] by three human raters. While such simple rating instruction is human-friendly and offers enough granularity, the simplicity of the scale can lead to less accurate representations of opinions, as given the broad spectrum covered by the rating aspects of semantic consistency (SC) and perceptual quality (PQ). We propose a more rigorous rating instruction toward comprehensive evaluation for each type of task. We split the rating of semantic consistency (SC) and perceptual quality (PQ) into multiple sub-scores, which SC contains multiple scores according to the tasks.
For example, in the multi-concept image composition task as shown in Figure 3, two images (known as concepts) and a text prompt are provided as input, and the desired synthesized image will contain the two concept objects in the image in actions according to the text prompt. Thus SC will be split into 3 sub-scores: (1) Is the image aligning with the prompt? (2) Does the object in the image resemble the first concept? (3) Does the object in the image resemble the second concept? For PQ, the naturalness level and distortion level will be accessed separately, resulting in 2 subscores: (i) Does the image give an unnatural feeling such as a wrong sense of distance, wrong shadow, or wrong lighting? (ii) Does the image contain a portion of distortion, watermark, scratches, etc.? Our proposed rating system enhances the evaluation of tasks by dividing SC and PQ into distinct sub-scores. The details of prompt templates are available in Appendix A.
O = [min(α1, ..., αi)min(β1, ..., βi)] 1 2 (2)
Our overall score is derived as shown in equation 2. We assume each sub-score weights the same and used min operation to emphasize the importance of meeting all criteria without exception. αi is a subscore in SC and βi is a sub-score in PQ. The final rating scores of SC and PQ provided by MLLMs are on a scale of 0 to 10. The design rationale is that in ImagenHub’s human rating method, the possible results when the answers of three human
raters, each picking an option from List[0, 0.5, 1], are added together and then divided by 3, will fall into one of the options: List[0.0, 0.17, 0.33, 0.5, 0.67, 0.83, 1.0]. Thus we simply use a scale of 0 to 10 and normalized in the range [0.0, 1.0] when comparing with human ratings. Input conditions and synthetic image are fed into the MLLM together during the rating process of SC, while in the PQ rating process, only the synthetic image is fed into the MLLM. This is to avoid the model getting confused by the input conditions in the PQ rating process, as to be discussed in section 5.1.

heading: 5 Experimental Results

heading: 5.1 Correlation Study
text: For all presented correlations, we applied Fisher Z-transformation to estimate the average Spearman correlation ∈ [−1, 1] across models and tasks. Metric-to-Human (M-H) correlations. In Table 2 and 3, we first verified the reliability of ImagenHub human ratings by computing the Human-to-Human (H-H) correlation, as the correlation goes around 0.5, expected to be the highest value compared to MLLMs. Then we benchmark the MLLMs according to our designed method to compute the Metric-to-Human (M-H) correlation. We noticed only GPT-4v, GPT4o, Gemini, and LLaVA were able to follow our instructions clearly while other MLLMs were not able to produce any meaningful results according to our setup. For example,

heading: Across All 7 Tasks

heading: VIESCORE
text: BLIP-2, while able to output the correct format, the scores provided are constant zeros. Qwen-VL and InstructBLIP could only produce a portion of responses for semantic consistency but failed to generate any results for perceptual quality evaluation. From overall performance, we found that GPT-4o reports a significantly higher correlation than all other MLLMs, while LLaVA’s correlation is much less than human raters. It seems that LLaVA is less effective in these specific tasks compared to closesourced MLLMs like GPT-4v. It is worth mentioning that GPT-4o achieves a very high correlation with human raters on different image generation and editing tasks. GPT-4v and Gemini also show satisfactory performance on nearly all tasks with a difference of less than 0.2 towards human correlations, even on par with humans in text-guide image generation tasks. Both GPT-4v, Gemini, and LLaVA demonstrated the weakest performance in the text-guided editing task and subject-driven image editing task. Extra visuals resulted in a decline in performance. Numerous studies (Brown et al., 2020; Parnami and Lee, 2022; Liu et al., 2021) have highlighted that In-Context Learning (ICL) allows LLMs to tackle novel tasks effectively without requiring the traditional fine-tuning process. We applied In-Context Learning in our prompting method with the expectation of increasing the correlation

heading: VIESCORE

heading: VIESCORE

heading: VIESCORE
text: scores, but we observed the opposite. In Table 2 and 3, there is an observable general trend of diminishing correlation scores. The overall correlation score in subject-driven image generation and editing, and the multi-concept image composition task dropped significantly. Only the mask-guided image editing task and control-guided image generation task reported a subtle increase in correlation score.
Looking into the rationale, we found that the MLLMs tend to get confused by the example images, as illustrated in Figure 4. Such behavior is observed in both GPT-4v, GPT-4o, Gemini, and LLaVA rationale. Another recent work (Lu et al., 2023b) also reported a similar issue where the model attempted to consider the example when answering the visual question. This explains the deterioration of the correlation on both GPT-4v, GPT4o, Gemini, and LLaVA when the ICL prompting technique is used. This also implied the low correlation scores on image editing tasks were due to the limited capability of state-of-the-art MLLMs for multiple image understanding.
Ablation study on PQ rating setting. As providing multiple images could potentially decrease the performance, we attempt to minimize the workload of MLLM by only providing the synthetic image in the PQ rating process instead of including the input conditions. We report the correlation score

heading: VIESCORE

heading: VIESCORE
text: in the two different settings with GPT-4v in Table 5 to examine the impact. We spotted a significant improvement in correlation after taking away the inputs in the PQ rating process. Ranking image models. Besides rating score correlations, we also compared the model ranking from the ImagenHub human evaluation leaderboard and the model ranking suggested by the MLLMs, shown in Table 6. We computed Spearman’s footrule dSF (r, r∗) ∈ [0,+∞) and Spear-
man’s rho ρS(r, r∗) ∈ [−1, 1] to examine the ranking correlation. Both GPT-4v and LLaVA can align to ImagenHub rankings on the multi-concept image composition task and control-guided image generation task, and with only one model difference in the subject-driven image editing task. While the results vary significantly across other tasks, GPT-4v generally maintains a stronger alignment with the ImagenHub rankings compared to LLaVA.

heading: 5.2 Insights and Challenges on VIESCORE
text: MLLMs are weak at capturing image nuances in edited images. From Table 2, we noticed the correlation scores on editing tasks are generally
lower than generation tasks. Upon investigation, it was found that MLLMs often fail to detect minor changes made in image editing, such as small patch edits. Consequently, MLLMs might perceive two images as identical even when humans recognize the edits as successful. This issue may stem from MLLMs focusing on high-level image features while overlooking finer details like color and texture differences, as illustrated in Figure 5. This limitation is apparent in both GPT-4v, GPT4o, Gemini, and LLaVA, highlighting a challenge in synthetic image evaluation accuracy on image editing tasks. Both MLLMs and human evaluators display a broader range of views regarding perceptual quality compared to semantic consistency. From Table 2 and Table 3, we can observe that the correlation scores of PQ are generally lower than the correlation scores of SC and Overall, even on human raters. This suggests the human raters’ perspective on evaluating perceptual quality is more diverse. Possible impacting factors include the rater’s eyesight condition, screen resolution, rating leniency, etc. In the context of MLLMs, we found that MLLMs while being able to correctly recognize the naturalness and artifacts of the image, the rating scores are as diverse as human rating scores
even though we have provided a marking rubric.

heading: 5.3 VIESCORE and Auto-metrics vs Human
text: We report the human correlations in Table 2 and 3 to compare the performance between our VIESCORE and popular auto-metrics. To ensure a fair comparison, we only included automatic metrics that have been previously reported in related research for the specific tasks under consideration. DINO is an effective metric in subject-driven tasks. The DINO metric demonstrates sensitivity to variations within the same class of subjects, making it an effective metric for measuring whether the subject in the synthesized image aligns with the token subject. Our correlation result shows that DINO outperforms GPT-4v and CLIP-I on subject-driven image generation and editing tasks, suggesting that DINO highly aligns with human’s perspective on semantic consistency where subject fidelity is considered. However, multi-subject fidelity remains a challenge as DINO and CLIP-I only consider single subjects. However, GPT-4o still achieves the highest correlation to human annotations. LPIPS metric proves to be effective in controlguided tasks, but less effective in image editing tasks. As discussed in section 3.3, LPIPS has great ability in detecting distortions. Since the control-guided task is a less mature research direction compared to image editing tasks, distortions are often found in the synthetic images from the control-guided task. On the other hand, current image editing models can synthesize images with less distortions. This explains the high correlation in the control-guided task. While GPT-4o still achieves the highest correlation to human annotations, LPIPS outperforms GPT-4v, Gemini, and LLaVA on this task. CLIP-Score has a much weaker correlation with human ratings in the text-guided image generation task than GPT-4v. We also noticed none of the synthetic images achieved higher than 0.3 CLIP-Score, even though most of the images are regarded as having high semantic consistency by human raters. This can be due to different evaluation focuses, as humans tend to grab the abstract idea from the prompt to access the image, but CLIPScore considers the whole text prompt. GPT-4v outperforms other auto-metrics with its correlation to the ImagenHub leaderboard rankings. The correlation of model rankings on ImagenHub was evaluated against CLIP Score and
LPIPS metrics, as shown in Table 6, and compared with MLLMs in the VIESCORE. We found that GPT-4v can achieve a positive correlation with the model rankings on every task. This shows the sign of capability for MLLMs as evaluators for image synthesis research.

heading: 6 Conclusion
text: In this paper, we propose the VIESCORE for synthetic image evaluation across seven popular image synthesis tasks and comprehensively access the efficacy using human ratings from ImagenHub. Our experiment reported that VIESCORE with close-source MLLMs backbones like GPT-4o and GPT-4v are significantly more effective than other open-source MLLMs in assessing synthetic images, achieving a correlation of over 0.4 to human ratings most of the tasks. However, it notes a lower correlation in image editing tasks for most of the MLLMs, including GPT-4v. We also noticed Gemini has similar performance as GPT-4v, while GPT-4o stands superior. Comparing our VIESCORE to existing auto-metrics, we found that GPT-4o is more effective than auto-metrics in all tasks, while DINO is more effective in subjectdriven image generation and editing tasks than GPT-4v. GPT-4v also shows a higher ranking correlation with the ImagenHub leaderboard than other automatic metrics. This marked a milestone towards explainable metrics for conditional image synthesis evaluation. Our future research will focus on investigating the use of distillation models to replicate human-like performance in evaluating synthetic images.

heading: 7 Limitations
text: OpenAI Security and Privacy Policy. Due to ChatGPT’s security and privacy policy, AIgenerated images that resemble a real person or photograph will be refused by GPT-4v for evaluation. The model will return results similar to "I am sorry, but I cannot process these images as they contain real people.". We simply drop those results by keyword matching. OpenAI Playground vs API. While GPT-4v Playground allows the user to keep a session, the OpenAI API does not provide such a function. While we believe using GPT-4v playground might yield better performance, especially in an In-Context learning setting, we can only rely on API due to the large scale of the experiment.

heading: 8 Potential Risks
text: Multimodal models can inadvertently perpetuate or amplify biases present in their training data. The interpretation and evaluation of synthetic images depend heavily on context. A multimodal model might not fully grasp certain images’ nuances or cultural sensitivities, leading to inappropriate or offensive outputs.

heading: 9 Artifacts
text: All datasets and models are publicly accessible for academic use, and the official OpenAI API is available for academic purposes.

heading: 10 Computational Experiments
text: All open-source model experiments were conducted on an NVIDIA RTX A6000 GPU. Approximately 500 US dollars were spent on an OpenAI API call for GPT-4v and GPT-4o experiments.

heading: 11 Acknowledgement
text: We thank Kai Zhang, Yujie Lu, and Tianle Li for the discussion. We also thank Xueguang Ma and Tianle Li for sharing the quotas of GPT-4v and GPT-4o API calls from OpenAI.

heading: A Prompt Templates
text: Prompt Engineering. We found that not all MLLMs can fully understand our prompt to give a desired output format consistently. Thus we required MLLMs to output a JSON format, which is supposed to be capable for most MLLMs.
Prompt Design. The prompt is divided into two segments: the ‘context prompt’ and the ‘rating prompt’. The ultimate prompt provided to the model is a combination of these two segments.
Context
You are a professional digital artist. You will have to evaluate the effectiveness of the AI-generated image(s) based on the given rules. You will have to give your output in this way (Keep your reasoning concise and short.): { "score" : [...], "reasoning" : "..." }
PQ Rating Prompt Template (for all tasks)
RULES: The image is an AI-generated image. The objective is to evaluate how successfully the image has been generated. On a scale 0 to 10: A score from 0 to 10 will be given based on image naturalness. ( 0 indicates that the scene in the image does not look natural at all or gives an unnatural feeling such as a wrong sense of distance, wrong shadow, or wrong lighting. 10 indicates that the image looks natural. ) A second score from 0 to 10 will rate the image artifacts. ( 0 indicates that the image contains a large portion of distortion, watermarks, scratches, blurred faces, unusual body parts, or subjects not harmonized. 10 indicates the image has no artifacts. ) Put the score in a list such that output score = [naturalness, artifacts]
SC Rating Prompt Template (Text-Guided Image Generation)
RULES: The image is an AI-generated image according to the text prompt. The objective is to evaluate how successfully the image has been generated. On a scale 0 to 10: A score from 0 to 10 will be given based on the success in following the prompt. (0 indicates that the AI-generated image does not follow the prompt at all. 10 indicates the AI-generated image follows the prompt perfectly.) Put the score in a list such that output score = [score]. Text Prompt: <prompt>
SC Rating Prompt Template (Text/Mask-Guided Image Editing)
RULES: Two images will be provided: The first being the original AI-generated image and the second being an edited version of the first. The objective is to evaluate how successfully the editing instruction has been executed in the second image. Note that sometimes the two images might look identical due to the failure of the image edit. On scale of 0 to 10: A score from 0 to 10 will be given based on the success of the editing. (0 indicates that the scene in the edited image does not follow the editing instructions at all. 10 indicates that the scene in the edited image follows the editing instruction text perfectly.) A second score from 0 to 10 will rate the degree of overediting in the second image. (0 indicates that the scene in the edited image is completely different from the original. 10 indicates that the edited image can be recognized as a minimally edited yet effective version of the original.) Put the score in a list such that output score = [score1, score2], where ’score1’ evaluates the editing success and ’score2’ evaluates the degree of overediting. Editing instruction: <instruction>
SC Rating Prompt Template (Control-Guided Image Generation)
RULES: Two images will be provided: The first being a processed image (e.g. Canny edges, openpose, grayscale, etc.) and the second being an AI-generated image using the first image as guidance. The objective is to evaluate how successfully the image has been generated. On scale 0 to 10: A score from 0 to 10 will be given based on the success in following the prompt. (0 indicates that the second image does not follow the prompt at all. 10 indicates the second image follows the prompt perfectly.) A second score from 0 to 10 will rate how well the generated image is following the guidance image. (0 indicates that the second image does not follow the guidance at all. 10 indicates that the second image is following the guidance image.) Put the score in a list such that output score = [score1, score2], where ’score1’ evaluates the prompt and ’score2’ evaluates the guidance. Text Prompt: <prompt>
SC Rating Prompt Template (Subject-Driven Image Generation)
RULES: Two images will be provided: The first is a token subject image and the second is an AI-generated image using the first image as guidance. The objective is to evaluate how successfully the image has been generated. On a scale of 0 to 10: A score from 0 to 10 will be given based on the success in following the prompt. (0 indicates that the second image does not follow the prompt at all. 10 indicates the second image follows the prompt perfectly.) A second score from 0 to 10 will rate how well the subject in the generated image resembles the token subject in the first image. (0 indicates that the subject in the second image does not look like the token subject at all. 10 indicates the subject in the second image looks exactly like the token subject.) Put the score in a list such that output score = [score1, score2], where ’score1’ evaluates the prompt and ’score2’ evaluates the resemblance. Text Prompt: <prompt>
SC Rating Prompt Template (Subject-Guided Image Editing)
RULES: Three images will be provided: The first image is an input image to be edited. The second image is a token subject image. The third image is an AI-edited image from the first image. it should contain a subject that looks like the subject in the second image. The objective is to evaluate how successfully the image has been edited. On a scale 0 to 10: A score from 0 to 10 will rate how well the subject in the generated image resembles the token subject in the second image. (0 indicates that the subject in the third image does not look like the token subject at all. 10 indicates the subject in the third image looks exactly like the token subject.) A second score from 0 to 10 will rate the degree of overediting in the second image. (0 indicates that the scene in the edited image is completely different from the first image. 10 indicates that the edited image can be recognized as a minimally edited yet effective version of the original.) Put the score in a list such that output score = [score1, score2], where ’score1’ evaluates the resemblance and ’score2’ evaluates the degree of overediting. Subject: <subject>

heading: B Supplementary Information

heading: B.1 Human Correlation Study
text: In our paper content, we only reported the Spearman correlations. Here we included the Pearson and Kendall correlation in Table 8 for comparative analysis of Human-to-Human (H-H) correlation.

heading: B.2 Zero-shot vs One-shot on VIE
text: We applied only zero-shot and one-shot experiments in this paper because not even GPT-4v can produce anything with few-shot setting in our context. We report full table of GPT-4v performance in Table 7 for zero-shot vs one-shot results.

heading: B.3 Autometrics vs Human Detail results
text: See Table 9, 10, 11, 12 for detail statistics of CLIPScore, LPIPS, DINO, and CLIP-I correlation with human ratings.
B.4 ImagenHub Models used
• Text-guided Image Generation: Stable Diffusion (SD) (Rombach et al., 2022), SDXL (stability.ai, 2023), DALLE-2 (Ramesh et al., 2022), DeepFloydIF (deep floyd.ai, 2023), OpenJourney (openjourney.ai, 2023).
• Mask-guided Image Editing: SD (runwayml, 2023), SDXL (stability.ai, 2023), GLIDE, BlendedDiffusion (Avrahami et al., 2022)
• Text-guided Image Editing: MagicBrush (Zhang et al., 2023a), InstructPix2Pix (Brooks et al., 2023), Prompt-to-Prompt (Mokady et al., 2023), CycleDiffusion (Wu and la Torre, 2023), SDEdit (Meng et al., 2021), Text2Live (Bar-Tal et al., 2022), DiffEdit (Couairon et al., 2022), Pix2PixZero (Parmar et al., 2023).
• Subject-driven Image Generation: DreamBooth (Ruiz et al., 2023), DreamBooth-Lora (Hu et al., 2021), BLIP-Diffusion (Li et al., 2023a), TextualInversion (Gal et al., 2022).
• Subject-driven Image Editing: PhotoSwap (Gu et al., 2023), DreamEdit (Li et al., 2023c), BLIP-Diffusion.
• Multi-concept Image Composition: CustomDiffusion (Kumari et al., 2023), DreamBooth, TextualInversion.
• Control-guided Image Generation: ControlNet (Zhang and Agrawala, 2023), UniControl (Qin et al., 2023).
B.5 ImagenHub Human data information We showed the total human rating data we used for each task in Table 13.

heading: 154 1386

heading: 150 1800

heading: 179 4296

heading: 179 2148

heading: 197 2955

heading: C Backbone Performances

heading: C.1 Parsing MLLM outputs
text: We tried to parse the output using Regex and modify the format requirement if the MLLM fail to do so. If the output failed to pass our parsing rules, we fill random value as output to penalize the correlation.

heading: C.2 Observations of GPT-4o.
text: GPT-4o-2024-05-13 tends to be the best MLLM in this context. It can understand every task instruction in VIESCORE and produce reasonable scores and rationale.

heading: C.3 Observations of Gemini-Pro.
text: Gemini-1.5-Pro can understand every task instruction in VIESCORE and produce reasonable scores and rationale, achieving similar performance as GPT-4v.

heading: C.4 Observations of GPT-4v.
text: GPT-4-vision-preview tends to be the second best MLLM in this context. It can understand every task instruction in VIESCORE and produce reasonable scores and rationale.

heading: C.5 Observations of LLaVA.
text: LLaVA-1.5-7B can also understand every task instruction in VIESCORE and produce reasonable rationale. However, the scores produced tend to be concentrated toward certain numbers.

heading: C.6 Observations of Qwen-VL.
text: Qwen-VL-7B does not understand the meaning of delimiter. However, it was able to output a JSONlike dictionary following the instructions on both SC and PQ. The rationale produced is often not reasonable.

heading: C.7 Observations of BLIP2.
text: BLIP-2 FLAN-T5-XXL often failed to produce the result formats according to the instructions, especially in PQ. It also tend to give 0 score in SC. Prompt engineering in our context does not solve the issue.

heading: C.8 Observations of InstructBLIP.
text: InstructBLIP-T5-XL shares same observation as BLIP-2 FLAN-T5-XXL.

heading: C.9 Observations of Fuyu.
text: Fuyu-8B always output 0 and failed to follow in our instruction.

heading: C.10 Observations of CogVLM.
text: CogVLM tends to output numbers fall off the range [0, 10] and often failed to follow the required format. Prompt engineering in our context does not solve the issue.

heading: C.11 Observations of OpenFlamingo.
text: OpenFlamingo simply printing blank as the output in our context.

heading: Future Work
text: Our future research will focus on investigating the use of distillation models to replicate human-like performance in evaluating synthetic images