A-Bench: Are LMMs Masters at Evaluating AI-generated Images?

## Abstract

Abstract How to accurately and efficiently assess AI-generated images (AIGIs) remains a critical challenge for generative models. Given the high costs and extensive time commitments required for user studies, many researchers have turned towards employing large multi-modal models (LMMs) as AIGI evaluators, the precision and validity of which are still questionable. Furthermore, traditional benchmarks often utilize mostly natural-captured content rather than AIGIs to test the abilities of LMMs, leading to a noticeable gap for AIGIs. Therefore, we introduce A-Bench in this paper, a benchmark designed to diagnose whether LMMs are masters at evaluating AIGIs . Specifically, A-Bench is organized under two key principles: 1) Emphasizing both high-level semantic understanding and low-level visual quality perception to address the intricate demands of AIGIs. 2) Various generative models are utilized for AIGI creation, and various LMMs are employed for evaluation, which ensures a comprehensive validation scope. Ultimately, 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts, and tested across 18 leading LMMs. We hope that A-Bench will significantly enhance the evaluation process and promote the generation quality for AIGIs. The benchmark is available at https://github.com/Q-Future/A-Bench.

## 1 Introduction

One look is worth a thousand words. Inspired by this age-old adage, numerous researchers dedicate their efforts to developing text-to-image (T2I) models that vividly bring text to life through imagery. These T2I models, driven by free-form text prompts, aim to create images that accurately align with the text and showcase high perceptual quality. Innovations such as AlignDRAW (Mansimov et al., ) and the text-conditional GAN (Reed et al., ) have introduced differential architecture for image generation. The field continues to advance with the development of stable diffusion models (Saharia et al., ; Rombach et al., ), significantly propelling T2I technology forward. On the commercial front, major corporations leverage vast-scale data to launch stunningly effective T2I models, such as DALL-E (Ramesh et al., ), Midjourney (Holz, ), Parti (Yu et al., ), etc. However, despite their diversity and widespread adoption, all these advanced T2I models occasionally face issues of low alignment with prompts and low perceptual quality in creating AI-generated images (AIGIs), necessitating careful evaluation and improvement.

The alignment and quality evaluation of AIGIs present significant challenges that small expert models attempt to address. Although these small expert models offer some solutions, they possess inherent drawbacks and often fail to meet contemporary demands. Specifically, for alignment assessment, CLIP-based similarity models struggle with accurately judging alignment, particularly with complex text prompts (Radford et al., ). When it comes to quality evaluation, traditional image quality/aesthetic assessment methods (IQA/IAA) are not capable of identifying AIGI-generative distortions (Wu et al., , ), rendering them unsuitable for this specialized task.

Figure: Figure 2: The proposed A-Bench is designed to find out whether LMMs are reliable for T2I AIGI evaluation. Instead of directly assessing the performance of LMM-based metrics, we evaluate the LMMs themselves behind by examining whether the fundamental questions regarding semantic understanding and quality perception can be correctly answered. Based on the benchmark results, we can then ‘diagnose’ the strengths and weaknesses across various LMMs.
Refer to caption: x2.png

Many researchers are increasingly relying on large language models (LLMs) and large multi-modal models (LMMs) for their human-like processing capabilities, which are presumed to enable accurate judgments of alignment and quality in generated content. Consequently, a variety of LMM-based evaluators have been developed, including VIE-Score (Ku et al., ), Prometheus (Kim et al., ), GPT4V-Eval (Lin et al., ), TIFA (Hu et al., ), and Davidsonian Graph (Cho et al., ), among others. Despite these advancements, a fundamental question remains:

Are LMMs reliable for evaluating T2I AIGIs?

These LMM-based metrics traditionally employ evaluation criteria such as SRCC/PLCC to determine their reliability. However, this approach only reveals how well the metrics perform, without shedding light on their specific strengths and weaknesses. To address this gap, we propose conducting a detailed and comprehensive ‘diagnostic’ benchmark $\rightarrow$ A-Bench, focusing on LMMs’ capabilities in semantic understanding and quality assessment. Rather than directly evaluating these LMM-based metrics, we focus on studying the LMMs themselves behind. We move away from computing SRCC/PLCC criteria and instead examine whether the fundamental perceptual questions can be correctly answered, which is the core basis of all LMM-based evaluators. To initiate our exploration on the AIGI evaluation abilities of LMMs, we first construct the A-Bench centered on a pivotal question:

What do we expect from LMMs as AIGI evaluators?

The answer lies in the capabilities of semantic alignment and quality evaluation. Then We define two key diagnostic subsets: A-Bench^P1$\rightarrow$high-level semantic understanding, and A-Bench^P2$\rightarrow$low-level quality perception. For high-level semantic understanding, A-Bench^P1 targets three critical areas: Basic Recognition, Bag-of-Words Pitfalls Discrimination, and Outside Knowledge Realization, which are designed to progressively test the LMM’s capability in AIGI semantic understanding, moving from simple to complex prompt-related content. For low-level quality perception, A-Bench^P2 concentrates on Technical Quality Perception, Aesthetic Quality Evaluation, and Generative Distortion Assessment, which are designed to cover the common quality issues and AIGI-specific quality problems. The selection of focus areas is meticulously designed to encompass the most prevalent application scenarios. Specifically, a comprehensive dataset of 2,864 AIGIs sourced from various T2I models is compiled, including 1,408 AIGIs for A-Bench^P1 and 1,456 for A-Bench^P2. Each AIGI is paired with a question-answer set annotated by human experts. Then we test 18 prominent LMMs, including both open-source and closed-source models, on the A-Bench. From the results that the best LMM still falls behind humans by a large margin, we can derive the following conclusion:

LMMs are still not masters at evaluating AIGIs.

All LMMs lag behind even the poorest human performance on A-Bench, and there is a substantial disparity between open-source LMMs and closed-source LMMs. The performance across different subcategories fluctuates for both A-Bench^P1 and A-Bench^P2, indicating that LMMs are not yet robust for different evaluation scenarios for AIGIs. There remains a considerable gap and significant room for improvement before LMMs can be considered masters of evaluating AIGIs.

In summary, we systematically explore the capabilities of LMMs in semantic understanding and quality perception, both crucial for their role as AIGI evaluators. These two essential capabilities constitute the core of the proposed A-Bench, the first ‘diagnostic’ benchmark specifically designed for LMM assessment in AIGI evaluation. Our contributions are summarized as follows:

- •
We carry out the first ‘diagnostic’ benchmark on AIGI evaluation for LMMs, which consists of 2,864 AIGIs (sampled from various T2I models) paired with question-answer sets on both high-level semantic understanding and low-level quality perception.
- •
A detailed discussion is made about what to ‘diagnose’. Semantic understanding is subdivided into Basic Recognition, Bag-of-Words Pitfalls Discrimination, and Outside Knowledge Realization while quality perception is broken down into Technical Quality Perception, Aesthetic Quality Evaluation, and Generative Distortion Assessment.
- •
From the benchmark results, several insights are gleaned, which can enable us to diagnose various issues with different LMMs and assist in their improvement for AIGI evaluation.

## 2 Related Works

### 2.1 Large Muli-modal Models

Large language models (LLMs), such as GPT-4 (OpenAI, ), T5 (Chung et al., ), and LLaMA (Touvron et al., ), exhibit exceptional linguistic capabilities in general human knowledge domains. By integrating visual input via CLIP (Radford et al., ) and additional adaptation modules, large multi-modal models (LMMs) (Li et al., ; Gao et al., ; Liu et al., ; Dai et al., ; Zhang et al., ) are capable of addressing diverse multi-modal tasks, including image captioning, visual question answering, visual segmentation, visual classification, visual reasoning, etc. Namely, OpenFlamingo (Awadalla et al., ) initially integrates several gated cross-attention dense blocks into the pretrained language encoder layers. InstructBLIP (Dai et al., ) extends BLIP-2 (Li et al., ) by incorporating vision-language instruction tuning. To further develop open-source LMMs, many works have employed GPT-4 (OpenAI, ) to create data for vision-language tuning, such as LLaVA series (Liu et al., , , ). However, whether these LMMs are masters at evaluating T2I AIGIs is still questionable, which needs further investigation.

### 2.2 Multi-modal Benchmarks

Benchmarks such as COCO Caption (Chen et al., ) and Nocaps (Agrawal et al., ) evaluate the capability of models to generate textual descriptions for images. Subsequently, benchmarks like GQA (Hudson and Manning, ) and OK-VQA (Marino et al., ) focus on visual question answering, assessing multi-modal models’ visual perception and reasoning abilities. Further complexities are added in benchmarks such as TextVQA (Singh et al., ) and ScienceQA (Lu et al., ), which incorporate OCR tasks and commonsense reasoning, respectively. MME (Fu et al., ) and MMbench (Liu et al., ) provide comprehensive evaluations of LMMs across various subtasks. Additionally, MMMU (Yue et al., ) targets extensive multi-disciplinary tasks that require college-level knowledge and sophisticated reasoning. More recently, Q-Bench (Wu et al., ) focuses specifically on assessing the low-level visual perception capabilities of LMMs. Despite these efforts, there is still a gap in systematic benchmarks for assessing the abilities of LMMs in AIGI evaluation, prompting the development of A-Bench to address this shortfall.

Figure: Figure 3: Illustration of focused aspects and corresponding quality distributions for A-Bench. The focused aspects and the amount of AIGIs employed are shown in (a). The quality scores of AIGIs sampled for Technical Quality Perception and Aesthetic Quality Evaluation subsets are obtained from AIGIQA-20K (Li et al., ) and predicted from Q-Align (Wu et al., ) respectively.
Refer to caption: x3.png

## 3 Constructing the A-Bench

### 3.1 Key Principles

##### Covering High-level and Low-level Attributes.

The demand for generating images has become increasingly stringent, with requirements for not only accurate adherence to prompt specifications but also high visual quality of AIGIs. To ascertain whether LMMs can effectively evaluate whether AIGIs meet these criteria, it is essential to assess their capabilities in both high-level semantic understanding and low-level quality perception. High-level semantic understanding encompasses basic recognition and the integration of external knowledge, whereas low-level quality perception involves the identification of technical quality, aesthetic appeal, and generative distortions. The detailed focused aspects can be overviewed in Fig. 3 (a).

##### Ensuring Diverse AIGI Scope.

Considering the variety of current generative models and their application scenarios, we have selected a broad range of mainstream text-to-image (T2I) generation models to produce AI-generated images (AIGIs). To assess high-level semantic understanding, we design prompts rich in content to ensure diversity among the generated images. For evaluating low-level quality perception, we employ uniform sampling to encompass a wide spectrum of visual quality and the corresponding quality distributions are illustrated in Fig 3 (b) and (c). Throughout the benchmarking process, we test multiple open-source and closed-source LMMs to guarantee a comprehensive evaluation. These measures ensure that our proposed A-Bench encompasses a diverse and extensive scope.
More details about AIGIs collection can be referred to in Sec. A.1.

Figure: Figure 4: Examples of A-Bench. Each AIGI is accompanied by a question-answer pair.
Refer to caption: x4.png

### 3.2 Focused Aspects

The key evaluation aspects of T2I models involve image-text alignment and image visual quality, which correspond to high-level semantic understanding and low-level quality perception abilities. Some representative examples regarding the subcategories discussed below are exhibited in Fig. 4.

#### 3.2.1 High-level Semantic Understanding

To evaluate whether LMMs can effectively assess image-text alignment, we implement the A-Bench^P1, which consists of 1,408 challenging multi-modal question-answer pairs that focus on high-level semantic understanding for AIGIs. The high-level semantic understanding can be divided into the following subcategories, moving from simple to complex prompt-related content:

##### Basic Recognition.

This aspect concentrates on the fundamental semantic understanding of AIGIs, which can be subdivided into two distinct areas: 1) Major Object Recognition, which involves recognizing the primary objects in the image, such as humans or objects depicted in the foreground. 2) Minor Object Recognition, which pertains to the identification of less-prominent objects within the image, such as background elements or secondary characters.

##### Bag-of-Words Pitfalls Discrimination.

This dimension focuses on the discriminative semantic understanding of AIGIs crafted with Bag-of-words prompts (encompassing rich descriptive attributes or complex object relationships (Qu et al., )). This can be subdivided into the following aspects: 1) Attributes Awareness, defined as the capability to accurately identify the attributes of objects in AIGIs. 2) Additionally, given that T2I models may incorrectly interpret nouns as adjectives, resulting in the unwanted generation of objects instead of the intended attributes, we have also introduced a dimension called Nouns as Adjectives Awareness to address this issue. 3) Composition Identification, recognized as the ability to correctly comprehend the compositional relationships such as orientation, occlusion, size comparison, and spatial arrangement. 4) Number of Objects Counting, regarded as the ability to accurately count the specified objects in the image, which is crucial for assessing whether the AIGI aligns with the numerical specifications of the prompt.

##### Outside Knowledge Realization.

This aspect emphasizes the reasoning ability to utilize external knowledge not directly depicted in the images (Schwenk et al., ), and can be broken down into the following dimensions:
1) Specific Terms Recognition: This involves identifying specific scenes and objects related to distinct domains such as geography, sports, science, materials, food, everyday life, creatures, brands, and styles. 2) Contradiction Overcome, recognized as the ability to correctly interpret AIGIs even when their content contradicts established world knowledge, which is particularly crucial for evaluating AIGIs generated from controversial prompts.

#### 3.2.2 Low-level Quality Perception

Conversely, to determine the ability of LMMs on image visual quality, we conduct the A-Bench^P2, comprising 1,456 challenging multi-modal question-answer pairs centered on low-level quality perception for AIGIs, which can be categorized into the following aspects:
1) Technical Quality Perception This indicates the low-level
characteristics that directly degrade the quality of images, such as blur, noise, exposure, etc (Su et al., ; Ying et al., ). 2) Aesthetic Quality Evaluation This indicates the attributes that affect the aesthetic appeal of AIGIs and evoke varied human feelings, which include color, lighting, etc (Huang et al., ). 3) Generative Distortion Assessment This indicates the unexpected AIGI-specific distortions (Chen et al., ; Li et al., , ), such as generative blur caused by low completion, confusing geometry structure, unnaturalness, etc.

### 3.3 Question Collection

##### Question Type

In the A-Bench, two types of question formats are utilized, including Yes-or-No questions and What questions. The Yes-or-No questions (accounting for 25.9%) are used to evaluate the fundamental judgment abilities of LMMs while the What questions (accounting for 74.1%) are more complicated and require LMMs to gain a more comprehensive understanding of the AIGIs.

##### Human Expert Annotation

We have assembled a team of 15 human annotators, each with expert experience in AIGI evaluation, to develop questions for A-Bench. This annotation process is conducted in a controlled laboratory environment, ensuring consistency and reliability. Annotators are tasked with designing questions specific to the sub-categories of the AIGIs under review, utilizing their extensive knowledge to determine the content and format of each question. To ensure the highest quality and suitability, each question undergoes a rigorous review process, with at least three other expert annotators double-checking it. More details can be acquired in Sec. A.3.

##### Question Response

Specifically, the example input query to LMMs can be exemplified as:

#User: What painting style is represented in
the image? 

A. Abstract  B. Surrealism  C. Expressionism  D. Impressionism
Answer with the option’s letter from the given choices directly.

The answer candidates and correct answers are shuffled during the evaluation process. Since the
responses from LMMs can be in various forms (if the correct choice is C) such as ‘C’, ‘Expressionism’, ‘The painting style of image is expressionism’, etc., we employ a GPT-assisted choice evaluation technique proposed in (Liu et al., ; Wu et al., ) to validate the correctness of LMMs responses. More details are shown in Sec. A.4.

## 4 Experiment Results

### 4.1 Benchmark Candidates

To ensure the results are comprehensive and up-to-date, we select the latest and widely used LMMs for benchmarking. The Proprietary LMMs (closed-source) include Gemini 1.5 Pro (Reid et al., ), GPT-4v (OpenAI, ), GPT-4o (OpenAI, ), and Qwen-VL-Max (Bai et al., ). The Open-source LMMs include CogVLM2-19B (Llama3-8B) (Wang et al., ), IDEFICS-2 (Mistral-7B-Instruct-v0.2) (Huggingface, ), DeepSeek-VL-7B (Lu et al., ), InternLM-XComposer2-VL (Dong et al., ), LLaVA-NeXT (Llama3-8B), LLaVA-NeXT (Qwen-72B), LLaVA-NeXT (Qwen-110B) (Liu et al., ), mPLUG-Owl2 (LLaMA-7B) (Ye et al., ), LLaVA-v1.5 (Vicuna-v1.5-7B), LLaVA-v1.5 (Vicuna-v1.5-13B) (Liu et al., ), CogVLM-17B (Vicuna-v1.5-7B) (Wang et al., ), Qwen-VL (Qwen-7B) (Bai et al., ), BakLLava (Mistral-7B) (Liu et al., ), and Fuyu-8B (Persimmon-8B) (Adept, ). All LMMs are tested with zero-shot setting.

For human performance on A-Bench, we conduct a user-study experiment with five ordinary people in a controlled laboratory setting. Initially, participants familiarize themselves with the tasks through exposure to similar cases. Subsequently, they select the appropriate responses for the questions posed in the A-Bench. To maintain consistency with the conditions experienced by LMMs, the order of questions is randomized, and participants receive no additional information beyond the AIGIs, questions, and answer options. The best and worst performance is included for comparison.

Figure: (a) Overall results of A-Bench.
Refer to caption: x5.png

### 4.2 Findings of A-Bench

##### General Observation: Human > Proprietary LMMs > Open-source LMMs

A concise overview of the A-Bench results is provided in Fig. 5, revealing several general insights: 1) All LMMs significantly outperform the random guess, indicating their capabilities in handling AIGI evaluation, with Gemini 1.5 Pro leading, closely followed by GPT-4o and Qwen-VL-Max. Notably, among the open-source LMMs, which are preferred for AIGI evaluations due to their accessibility and modifiability, LLaVA-NeXT (Qwen-110B) stands out, though it still significantly lags behind closed-source competitors. 2) Even the lowest performance by humans surpasses that of all LMMs, with a noticeable 14.70% gap compared to the top-performing LMM, Gemini 1.5 Pro, indicating that LMMs are still far from adequately performing AIGI evaluation as humans. 3) A closer examination of the radar chart in Fig. 5 (b) shows that top LMMs exhibit varied performances across different sub-categories, suggesting a lack of robustness, while humans show more consistent and balanced performance across these categories, highlighting areas where LMMs need further improvement.

**Table 1: Benchmark results on the A-Bench^P1 subset, which reveal the high-level semantic understanding abilities across LMMs. The best performance is marked in bold and the second performance is underlined for both proprietary and open-source LMMs respectively.**
| Categories | Basic Recognition | Bag-of-Words | Outside Knowledge | Overall$\uparrow$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| \cdashline1-9<br>LMM (LLM) | Major$\uparrow$ | Minor$\uparrow$ | Attr.$\uparrow$ | N. Adj.$\uparrow$ | Comp.$\uparrow$ | Number$\uparrow$ | Term$\uparrow$ | Contra.$\uparrow$ |  |
| Human (Worst) | 95.20% | 94.27% | 96.83% | 88.64% | 85.54% | 82.50% | 81.79% | 88.89% | 92.39% |
| Human (Best) | 95.42% | 95.18% | 99.46% | 95.12% | 93.42% | 91.67% | 84.23% | 96.00% | 94.00% |
| Proprietary LMMs: |  |  |  |  |  |  |  |  |  |
| \hdashlineGemini 1.5 Pro | 93.80% | 95.17% | 94.33% | 80.31% | 72.11% | 79.31% | 73.00% | 61.76% | 84.69% |
| GPT-4v | 92.97% | 95.97% | 87.43% | 82.63% | 64.44% | 68.78% | 77.58% | 66.71% | 83.64% |
| GPT-4o | 94.33% | 95.16% | 91.96% | 79.59% | 76.34% | 73.33% | 77.53% | 68.57% | 85.42% |
| Qwen-VL-Max | 92.57% | 94.77% | 91.97% | 85.76% | 68.97% | 75.78% | 78.94% | 65.14% | 84.54% |
| Open-source LMMs: |  |  |  |  |  |  |  |  |  |
| \hdashlineCogVLM2-19B (Llama3-8B) | 93.30% | 92.74% | 89.95% | 75.51% | 64.52% | 66.67% | 75.96% | 61.43% | 82.47% |
| IDEFICS-2 (Mistral-7B-Instruct-v0.2) | 89.95% | 91.94% | 86.43% | 75.51% | 61.29% | 71.11% | 73.22% | 62.86% | 80.00% |
| DeepSeek-VL-7B | 91.49% | 91.13% | 82.41% | 83.67% | 63.44% | 70.00% | 75.41% | 60.00% | 81.24% |
| InternLM-XComposer2-VL (InternLM2) | 92.78% | 95.16% | 86.43% | 82.65% | 68.82% | 72.22% | 70.77% | 64.29% | 81.90% |
| LLaVA-NeXT (Llama3-8B) | 92.75% | 92.37% | 91.12% | 83.65% | 61.00% | 67.02% | 76.20% | 62.97% | 82.91% |
| LLaVA-NeXT (Qwen-72B) | 94.33% | 92.74% | 91.46% | 81.63% | 62.37% | 73.33% | 77.05% | 61.43% | 83.98% |
| LLaVA-NeXT (Qwen-110B) | 93.81% | 91.13% | 90.45% | 84.69% | 67.74% | 67.78% | 76.23% | 64.29% | 83.62% |
| mPLUG-Owl2 (LLaMA-7B) | 85.31% | 86.29% | 83.92% | 79.59% | 53.76% | 57.78% | 71.04% | 58.57% | 76.44% |
| LLaVA-v1.5 (Vicuna-v1.5-7B) | 87.89% | 88.71% | 83.92% | 75.51% | 61.29% | 65.56% | 74.86% | 62.86% | 79.16% |
| LLaVA-v1.5 (Vicuna-v1.5-13B) | 88.40% | 89.52% | 86.43% | 79.59% | 62.37% | 58.89% | 74.86% | 61.43% | 79.67% |
| CogVLM-17B (Vicuna-v1.5-7B) | 90.46% | 95.14% | 85.93% | 77.55% | 49.46% | 47.78% | 73.22% | 61.43% | 78.52% |
| Qwen-VL (Qwen-7B) | 86.34% | 86.29% | 81.41% | 77.55% | 52.69% | 61.11% | 71.58% | 57.14% | 76.43% |
| BakLLava (Mistral-7B) | 88.97% | 81.29% | 77.36% | 73.87% | 52.21% | 62.38% | 68.49% | 49.12% | 74.27% |
| Fuyu-8B (Persimmon-8B) | 81.24% | 68.07% | 66.67% | 57.34% | 42.14% | 48.46% | 61.06% | 29.79% | 63.05% |
| random guess | 32.28% | 37.24% | 31.00% | 42.79% | 29.81% | 29.80% | 26.46% | 32.11% | 30.87% |

##### Findings of A-Bench P1 : LMMs excel at basic recognition tasks but tend to be less effective when it comes to nuanced semantic understanding.

The performance results of LMMs on the A-Bench^P1 subset, as detailed in Table 1, reveal several key insights: 1) Almost all LMMs show good performance in Basic Recognition, suggesting that they are quite adept at fundamental semantic understanding, which includes recognizing foreground and background objects in AIGIs. 2) However, their effectiveness diminishes in more complex tasks such as Bag-of-Words, particularly in subcategories like Nouns as Adjectives Awareness, Composition Identification, and Number of Objects Counting. These areas require deeper semantic understanding and reasoning, which is critical as users often employ complex prompts that include such nuanced elements. The LMMs’ underperformance here indicates potential challenges in accurately aligning AIGIs with user prompts. 3) Additionally, Outside Knowledge poses significant challenges, with LMMs generally achieving unsatisfactory performance in the Contradiction Overcome subcategory, where AIGIs contain content that defies common sense, requiring LMMs to override their prior knowledge to respond correctly. The subcategory Specific Terms tests the knowledge base of LMMs, where proprietary LMMs generally perform better due to being trained on more recent and extensive datasets.

##### Findings of A-Bench P2 : LMMs are poor quality evaluators.

The performance results of LMMs on the A-Bench^P2 subset, as shown in Table 2, illustrate a notable disparity in capabilities: 1) There is a significant performance gap of approximately 20% between the top-performing LMMs and human evaluators, highlighting that LMMs lag considerably in quality perception and struggle to accurately assess the quality of AIGIs. 2) Furthermore, most LMMs exhibit their weakest performance in the Generative Distortion Assessment subcategory, suggesting their ineffectiveness at identifying unexpected generative distortions, such as unnatural appearances and incorrect geometric structures. 3) Interestingly, while humans generally perform better in Technical Quality Perception compared to Aesthetic Quality Evaluation, LMMs show similar performance levels in both subcategories. This difference likely stems from the more objective nature of technical quality assessments, which leads to more consistent evaluations among humans, whereas aesthetic quality, being more subjective, results in a broader range of opinions and consequently lower performance scores.

##### Human vs. Proprietary LMMs

Proprietary (closed-source) LMMs are regarded as closely mirroring human perception and demonstrate superior performance, particularly in zero-shot settings for evaluating AIGI. Therefore, here we make a finer discussion about the human and proprietary LMMs. 1) Beginning with a detailed comparison of human and proprietary LMMs, we observe that proprietary LMMs achieve human-level performance in Basic Recognition, indicating their ability to correctly assess AIGI alignment when prompts are simple. 2) Despite this, LMMs encounter difficulties in the Bag-of-Words aspect, especially in identifying composition and counting objects, which highlights their limitations in handling complex compositional relationships and specific object counts. 3) In the Outside Knowledge domain, proprietary LMMs show only a slight performance gap compared to humans on Specific Terms, demonstrating comprehensive prior knowledge about specific terms, but they notably lag behind in identifying controversial content. While humans can easily recognize contradictory elements, proprietary LMMs often struggle due to their reliance on common sense, making accurate responses challenging. To conclude, according to the results shown in Table 1, proprietary LMMs are competent as evaluators for simple prompts in AIGI, yet they require further improvements for more complex prompts related AIGI content. 4) On the other hand, Table 2 reveals that LMMs have significant shortcomings in low-level quality perception compared to humans, with an uneven performance across different quality dimensions. Surprisingly, GPT-4o shows a distinct advantage over other proprietary LMMs in recognizing generative distortions, suggesting its superior capability in this area. However, the substantial overall difference in quality perception between proprietary LMMs and humans underlines that these models are currently unsuitable for assessing the visual quality of AIGI.

**Table 2: Benchmark results on the A-Bench^P2 subset, which reflect the low-level quality perception abilities across LMMs. The best performance is marked in bold and the second performance is underlined for both proprietary and open-source LMMs respectively.**
| Categories | Technical$\uparrow$ | Aesthetic$\uparrow$ | Generative$\uparrow$ | Overall$\uparrow$ |
| --- | --- | --- | --- | --- |
| \cdashline1-1<br>LMM (LLM) |  |  |  |  |
| Human (Worst) | 94.40% | 84.41% | 86.30% | 90.53% |
| Human (Best) | 94.99% | 86.12% | 93.04% | 92.25% |
| Proprietary LMMs: |  |  |  |  |
| \hdashlineGemini 1.5 Pro | 70.97% | 77.56% | 59.02% | 69.06% |
| GPT-4v | 67.68% | 68.22% | 57.11% | 64.29% |
| GPT-4o | 70.53% | 61.65% | 67.89% | 66.76% |
| Qwen-VL-Max | 70.47% | 69.46% | 58.37% | 66.04% |
| Open-source LMMs: |  |  |  |  |
| \hdashlineCogVLM2-19B (Llama3-8B) | 64.02% | 61.44% | 56.71% | 60.71% |
| IDEFICS-2 (Mistral-7B-Instruct-v0.2) | 61.18% | 68.86% | 47.36% | 59.00% |
| DeepSeek-VL-7B | 55.89% | 53.81% | 47.56% | 52.40% |
| InternLM-XComposer2-VL (InternLM2) | 62.24% | 63.32% | 50.31% | 58.55% |
| LLaVA-NeXT (Llama3-8B) | 58.54% | 48.52% | 52.03% | 53.09% |
| LLaVA-NeXT (Qwen-72B) | 59.96% | 55.51% | 59.76% | 58.45% |
| LLaVA-NeXT (Qwen-110B) | 64.63% | 57.20% | 63.62% | 61.88% |
| mPLUG-Owl2 (LLaMA-7B) | 57.93% | 54.45% | 53.86% | 55.43% |
| LLaVA-v1.5 (Vicuna-v1.5-7B) | 45.93% | 41.31% | 54.47% | 47.32% |
| LLaVA-v1.5 (Vicuna-v1.5-13B) | 46.14% | 41.31% | 48.17% | 45.26% |
| CogVLM-17B (Vicuna-v1.5-7B) | 54.88% | 48.31% | 52.44% | 51.92% |
| Qwen-VL (Qwen-7B) | 49.59% | 34.32% | 50.41% | 44.92% |
| BakLLava (Mistral-7B) | 47.84% | 33.33% | 48.44% | 43.37% |
| Fuyu-8B (Persimmon-8B) | 44.69% | 30.28% | 45.67% | 40.27% |
| random guess | 31.89% | 32.91% | 33.11% | 32.62% |

## 5 Conclusion

In conclusion, the ambition to employ LMMs for evaluating AIGIs exposes considerable deficiencies in their capabilities, as revealed by the diagnostic benchmark A-Bench. This benchmark scrutinizes the core capabilities of LMMs themselves, focusing on their ability to accurately address fundamental questions related to high-level semantic understanding and low-level quality perception. Our findings from A-Bench serve as a stark reminder of the current limitations faced by LMMs in the realm of AIGI evaluation. The results underscore that while LMMs provide valuable insights, their evaluation capacity remains notably inferior to human performance, especially in tasks that demand deep semantic comprehension and detailed quality assessment. By identifying specific areas for enhancement and charting a course for future research, this study not only underscores the urgent need for further development but also aids in refining the application of LMMs in AIGI evaluation tasks. Future initiatives should focus on augmenting the capabilities of LMMs to reliably match or surpass human performance in these intricate evaluation scenarios.

## Appendix A Appendix

Figure: Figure 6: Overview of the AIGIs from A-Bench^P1.
Refer to caption: extracted/5645316/imgs/part1_overview.jpg

### A.1 AIGIs Collection

##### AIGI collection for A-Bench P1

To ensure that the AIGIs meet the specific subcategory requirements, we have gathered 2,000 manually-written prompts to serve as the textual foundation. Below, we provide examples of these prompts:

- 1.
Basic Recognition -> Major Object Recognition: An elaborate treehouse in a thick forest, with children playing inside, rope bridges connecting to other trees, and birds chirping around.
- 2.
Basic Recognition -> Minor Object Recognition: A magical fairy ring in a moonlit forest, with tiny glowing fairies dancing and mystical plants all around.
- 3.
Bag-of-Words -> Attributes Awareness: A delicate, frosty, crystal snowflake beside a warm, glowing, amber ember on a smooth, slate-gray stone.
- 4.
Bag-of-Words -> Nouns as Adjectives Awareness: Shark-sleek submarine exploring ocean depths.
- 5.
Bag-of-Words -> Composition Identification: A gamer’s setup with consoles and controllers on a desk, multiple screens above, and game boxes and snacks partially obscured beneath the desk.
- 6.
Bag-of-Words -> Attributes Awareness: Six logs in a woodpile, stacked so tightly that they seem to form a solid block.
- 7.
Outside Knowledge -> Specific Terms Recognition: A barometer showing a rapid decrease in pressure.
- 8.
Outside Knowledge -> Contradiction Overcome: A ship floating above the clouds, sails made of sunlight.

Afterward, we use the collected prompts to create AIGIs. 15 text-to-image generation models are selected, which include: Dreamlike [dreamlike art, ], Pixart $\alpha$ Chen et al. [], Playground v2 PlaygroundAI [], SD1.4 [Rombach et al., ], SD1.5 [Rombach et al., ], SDXL [Rombach et al., ], SSD1B [Gupta et al., ], LCM Pixart [Luo et al., ], LCM SD1.5 [Luo et al., ], LCM SDXL [Luo et al., ], SDXL Turbo Sauer et al. [] DALLE2 [Ramesh et al., ], DALLE3 [Ramesh et al., ], IF [DeepFloyd, ], Midjourney v5.2 Holz [].
Finally, a total of 15$\times$2,000 =30,000 AIGIs are collected. To guarantee diversity, we randomly select 2,000 AIGIs, choosing one AIGI per prompt. Subsequently, we conduct a manual review of these AIGIs to remove any that failed to generate correctly or are unsuitable for annotation. This process results in the final set of AIGIs for A-Bench^P1, which can be overviewed in Fig. 6.

Figure: Figure 7: Illustration of the quality distribution transformation.
Refer to caption: x7.png

Figure: Figure 8: Overview of the AIGIs from A-Bench^P2.
Refer to caption: extracted/5645316/imgs/part2_overview.jpg

##### AIGI collection for A-Bench P2

A-Bench^P2 is designed for the quality evaluation of AIGIs. Consequently, it is essential to ensure that the collected AIGIs span a wide quality range to address various practical scenarios. For Technical Quality, we sample 500 AIGIs from the AIGIQA-20K dataset [Li et al., ] using a uniform sampling strategy. Specifically, each AIGI in the AIGIQA-20K dataset is assigned a mean opinion score (MOS) for technical quality. We apply uniform sampling to create more even distributions, as illustrated in Fig. 7. For Aesthetic Quality, in the absence of provided aesthetic scores, we utilize q-align [Wu et al., ], an effective quality predictor, to infer the aesthetic values of AIGIs. Subsequently, we perform uniform sampling similarly to obtain 500 AIGIs for aesthetic evaluation. For Generative Distortion, we manually select 500 AIGIs exhibiting unexpected AIGI-specific distortions. It is important to note that there is no content overlap among the selected AIGIs, which can be overviewed in Fig. 8.

Figure: Figure 9: Some finer cases for the ‘Bag-of-Words -> Composition Identification’ and ‘Outside Knowledge -> Specific Terms’ subcategories.
Refer to caption: x8.png

### A.2 Finer Explanation for some Subcategories

For certain subcategories that require additional clarification for better understanding, we provide detailed explanations here (the corresponding cases are shown in Fig. 9):

- 1.
Bag-of-Words -> Nouns as Adjectives Awareness. The ’Noun as Adjectives’ illustrates the use of nouns as adjectives to modify objects in AIGIs. Essentially, we aim for the descriptive effect, not for the nouns themselves to be visually represented in the AIGIs. For instance, as shown in Fig.4 row 2 column 2, when we describe a submarine as ’shark-sleek,’ we do not intend to generate an image of an actual shark. This subcategory is designed to test whether LMMs can correctly identify such misunderstandings.
- 2.
Bag-of-Words -> Composition Identification. We categorize composition into four distinct types: 1) Orientation, which assesses the ability to correctly determine the relative spatial positions of objects; 2) Occlusion, which involves evaluating the accuracy in discerning the overlapping relationships between objects; 3) Size Comparison, which tests the ability to accurately judge the size relationships among objects; and 4) Spatial Arrangement, which examines the ability to accurately assess the arrangement of objects within the AIGI.
- 3.
Outside Knowledge -> Specific Terms. This subcategory covers many aspects, including geography, sports, science, materials, food, everyday life, creatures, brands, and styles. This primarily investigates whether it is possible for LMMs to infer and deduce specific knowledge within these fields based on the content of AIGIs such as identifying the exact location feature based on geographical attributes, deducing the brand from the characteristics of a product, recognizing the cooking technique of the food, etc.

### A.3 Human Expert Annotation

A total of fifteen experts, each possessing professional skills and extensive experience in photography and AIGIs, participate in the subjective labeling experiment of A-Bench. All experts are informed that their annotation data will be publicly released, and they all agree to this arrangement. The hourly wage for each expert is approximately 12 US dollars, resulting in a total expense of about 2,400 US dollars for the whole subjective experiment.

Figure: Figure 10: Illustration depicting the annotation interface, where experts are presented with the subcategory and are able to record their questions and answers.
Refer to caption: extracted/5645316/imgs/gui.png

The experiment takes place in a laboratory environment with standard indoor lighting. A Dell 4K monitor, supporting a resolution of $3840\times 2160$, is used for displaying the interfaces. Screenshots of the interfaces can be referred to in Fig. 10. Each expert annotates up to 30 AIGIs per day to avoid fatigue, with every annotation carefully reviewed by at least three other experts before acceptance. This approach ensures the highest possible accuracy and rigor of the A-Bench labels, thereby enhancing the precision and meaningfulness of the performance testing capability of A-Bench.

### A.4 GPT Evaluation for Choice Judgment

For some LMMs, the response to the question inquiry may vary. For example, given the correct answer C. Blurry’ to the question What is the most severe technical distortion of this image?’, LMMs may respond in different formats: The image is blurry’, There is blur in this image’, or ‘low clarity’. To address the impact of such variations on our evaluation, we’ve implemented a 5-round voting strategy [Wu et al., ]. Under this strategy, we pose the same prompt, as defined in the templates, five times and determine the final outcome based on the majority of GPT’s responses.

GPT Evaluation Prompt Template

#System: You are a helpful assistant that grades answers related to image perception. There are a lot of special terms or keywords related to image processing and photography.

#User: Assuming you are a grader, you will now be provided with a question [question] and a set of options [options] with option [options[0]] being the correct answer. Additionally, there will be an answer [answer] provided by a respondent. Please determine whether the respondentś answer is correct considering the context of the question. Even if the word choice is not completely the same, you can decide based on the given options and see whether the one in the answer is close enough to the given correct answer, The result is 1 if the answer is correct and else the result is 0. Please only provide the result in the following format: Result:

Example for GPT Evaluation

Question: Which is the most blurry part of this image?

Choices: [‘The house on the left’, ‘The person in the middle’, ‘The background’, ‘The tree on the left’]

LMM Answer:

The most blurry part in this image is the house to the left of the person.

5-Round GPT Answers:
[“Score: 1",“Score: 1",“Score: 1",“Score: 1",“Score: 1"]
$\to$ Final Correctness after Voting: ✓

Figure: Figure 11: Illustration of the interface for the user-study.
Refer to caption: extracted/5645316/imgs/user-study.png

### A.5 User-study on A-Bench

To provide human performance on the A-Bench, we employ five ordinary people
in a controlled laboratory setting. Initially, participants familiarize themselves with the tasks through
exposure to similar cases. Subsequently, they select the appropriate responses for the questions
posed in the A-Bench. The user-study interface is shown in Fig. 11.

### A.6 LMM Experiment Details

The LMMs undergo testing in a zero-shot setting. Proprietary LMMs are evaluated via official APIs, whereas the open-source LMMs (with the exceptions of LLaVA-NeXT Qwen-72B and LLaVA-NeXT Qwen-110B) run on an NVIDIA RTX 6000 Ada with 48 GB of memory. The LLaVA-NeXT Qwen-72B and LLaVA-NeXT Qwen-110B are operated on 4 NVIDIA H100 with 320 GB of memory. All LMMs operate with default parameters, ensuring that the A-Bench results are readily reproducible.

**Table 3: Benchmark results on the question types. The best performance is marked in bold and the second performance is underlined for both proprietary and open-source LMMs respectively.**
| Categories | A-Bench^P1 | A-Bench^P2 | Overall |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| \hdashlineLMM (LLM) | Yes-or-no$\uparrow$ | What$\uparrow$ | Yes-or-no$\uparrow$ | What$\uparrow$ | Yes-or-no$\uparrow$ | What$\uparrow$ |
| Human (Worst) | 91.21% | 92.77% | 89.45% | 91.02% | 91.23% | 91.88% |
| Human (Best) | 93.55% | 94.25% | 91.80% | 92.64% | 92.77% | 93.39% |
| Proprietary LMMs: |  |  |  |  |  |  |
| \hdashlineGemini 1.5 Pro | 81.96% | 86.91% | 74.08% | 65.57% | 76.50% | 76.82% |
| GPT-4v | 82.37% | 85.86% | 71.11% | 60.09% | 75.51% | 73.23% |
| GPT-4o | 84.39% | 85.76% | 69.76% | 65.15% | 76.28% | 75.81% |
| Qwen-VL-Max | 86.70% | 84.02% | 68.13% | 64.60% | 75.79% | 74.91% |
| Open-source LMMs: |  |  |  |  |  |  |
| \hdashlineCogVLM2-19B (Llama3-8B) | 81.77% | 83.26% | 63.70% | 58.65% | 70.55% | 71.61% |
| IDEFICS-2 (Mistral-7B-Instruct-v0.2) | 78.32% | 83.84% | 63.87% | 55.63% | 68.91% | 69.96% |
| DeepSeek-VL-7B | 80.72% | 82.00% | 60.00% | 47.15% | 66.88% | 66.48% |
| InternLM-XComposer2-VL (InternLM2) | 82.08% | 81.53% | 66.49% | 53.06% | 70.90% | 69.83% |
| LLaVA-NeXT (Llama3-8B) | 81.17% | 84.11% | 52.10% | 53.77% | 63.89% | 68.82% |
| LLaVA-NeXT (Qwen-72B) | 83.22% | 84.31% | 57.91% | 60.01% | 70.22% | 71.55% |
| LLaVA-NeXT (Qwen-110B) | 82.99% | 83.91% | 59.78% | 62.87% | 71.76% | 73.05% |
| mPLUG-Owl2 (LLaMA-7B) | 74.92% | 78.00% | 56.97% | 54.36% | 64.38% | 67.81% |
| LLaVA-v1.5 (Vicuna-v1.5-7B) | 78.27% | 82.74% | 46.39% | 47.97% | 58.85% | 66.21% |
| LLaVA-v1.5 (Vicuna-v1.5-13B) | 79.51% | 81.47% | 47.23% | 43.90% | 61.41% | 63.61% |
| CogVLM-17B (Vicuna-v1.5-7B) | 76.77% | 80.11% | 55.13% | 49.71% | 64.33% | 65.65% |
| Qwen-VL (Qwen-7B) | 72.77% | 80.95% | 46.22% | 44.02% | 56.60% | 63.39% |
| BakLLava (Mistral-7B) | 71.01% | 78.77% | 42.11% | 44.11% | 55.61% | 60.03% |
| Fuyu-8B (Persimmon-8B) | 61.56% | 64.22% | 38.76% | 41.66% | 50.06% | 52.31% |

### A.7 Question Type Performance

We assess the performance disparity between Yes-or-no and What questions among LMMs. The Yes-or-no questions gauge the fundamental judgment capabilities of LMMs, whereas What questions demand a more comprehensive understanding. According to the results in Table 3, it is observed that most LMMs perform better on What questions within A-Bench^P1, suggesting a proficiency in processing semantic content. Conversely, in A-Bench^P2, where LMMs generally show lesser performance, they exhibit limited in-depth perception, maintaining only basic evaluative capabilities without comprehensive understanding, leading to poorer performance on What questions. Interestingly, human performance consistently excels in What questions across both A-Bench^P1 and A-Bench^P2, likely due to a broader range of options facilitating easier inference. However, human performance tends to be more balanced compared to LMMs, which may exhibit significant variance, such as IDEFICS-2, where there is over a 5% accuracy difference between question types, indicating less robustness.

### A.8 Data Statement

The A-Bench dataset is released under the CC BY 4.0 license. This includes all associated AIGIs, questions, and answer candidates. However, to prevent incorporation into the training sets of any LMMs, the correct answers remain confidential. We believe this precaution will ensure that A-Bench retains its long-term value as a benchmark for assessing AIGI evaluation capabilities.

### A.9 Limitations and Social Impact

##### Limitations

While A-Bench uses a diverse set of generative models and LMMs for evaluation, the choice and number of models might still limit the generalizability of the results. The performance of untested models or newer generative approaches might differ significantly. The rapid advancement in AI and generative models may quickly outpace the current setup of A-Bench, necessitating frequent updates or redesigns of the benchmark to stay relevant.

##### Social Impact

By improving the evaluation metrics for AIGIs, A-Bench could lead to more reliable and trustworthy AI-generated content, which is crucial as these technologies increasingly intersect with areas like media, entertainment, and education. Moreover, improved benchmarks and evaluation methods can drive industry standards, potentially lowering the barrier to entry for smaller developers and promoting innovation through clearer performance targets.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Overall, the paper adds the evaluation of LMMs' high-level semantic cognition for AIGI, in addition to previous work using LMMs to assess image generation quality. However, it does not highlight the difference between AIGI tasks and conventional cognition tasks. Could the authors elaborate on this further?
2. Generally, the authors focus more on evaluating the perceptual capabilities of LMMs, but these perceptual capabilities are more inclined towards the low-level aspects for AIGI tasks. Could the authors further elaborate on the differences between the low-level perceptual aspects of AIGI and some earlier works?
soundness: 2
strengths: 1. The authors manually annotated a dataset containing 2864 image quality issues, which contributes to the development of AIGI evaluation.
2. The authors evaluate AIGI quality from high-level semantic aspects like counting and low-level aspects like distortion, providing valuable insights for subsequent general AIGI task evaluations.
3. The paper's A-Bench includes the evaluation performance of multiple LMMs, offering guidance for researchers who wish to use LMMs for AIGI quality assessment.
summary: Due to the existing evaluation models' inability to effectively assess the performance of AIGI tasks, more and more researchers are turning to LMMs for evaluating the quality of generated images. The authors question this approach and design a framework consisting of seven dimensions focused on high-level semantic understanding and low-level quality evaluation to assess the quality of AIGI. By manually annotating 2864 different image quality issues, the authors compare the evaluation performance of multiple open-source and closed-source LMMs and contrast these with human evaluation results, summarizing numerous shortcomings of LMMs in the AIGI quality assessment task.
weaknesses: 1. Although A-Bench includes multiple LMMs, it lacks some of the latest SOTA models. Better models such as QWEN-VL2 and MiniCPMv2.6 can be found from opencompass. The paper does not specify the versions of gpt4o used, such as gpt-4o-2024-08-06 or gpt-4o-2024-05-13, which is crucial for future researchers.
2. The AIGI models used to generate the dataset are somewhat outdated, lacking relatively advanced image generation models such as SD3, PixArt, Flux, etc. Currently, the more outstanding AIGI models often embed large language models, which might significantly impact the evaluation conclusions.
3. The questions are all manually generated, which is certainly good. However, this makes the evaluation dataset difficult to expand and might lose value as AIGI models rapidly evolve. It would be better if the questions could be designed based on the text prompts of T2I models.
4. Compared to previous work, The paper's main contribution, i.e., high-level semantic question answering, is not strongly related to AIGI and does not seem necessary to research specifically in the AIGI context.
5. Two-thirds of the low-level semantic question-answering data come from other datasets, reducing the paper's contribution.
6. The paper's findings are somewhat unremarkable. It is obvious that closed-source LMMs perform better than open-source ones, and some other findings, such as the LMMs' insufficient perception of distortion, have already been mentioned in works like Q-Bench.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. It would strengthen the paper if the authors could provide scientific backing for the proposed metrics, citing sources that systematically define each measure. Specific areas where additional references might be valuable include the validity of semantic reasoning components and quality perception dimensions, to help ensure that the chosen metrics align with established frameworks in the field.

2. Providing further details on the image selection process would clarify the robustness of the benchmark. Specifically, information on the criteria for image selection, the diversity of image types, and the distribution of different content categories would offer valuable context. If possible, outlining how these factors impact the benchmark's representativeness, and validity could further enhance transparency.
soundness: 2
strengths: 1. The benchmark is undoubtedly useful. Given the growing reliance on LLMs to evaluate various AI-generated content like images, having a comprehensive, quantitative benchmark that assesses the effectiveness of LLMs in evaluation is highly valuable.
2. The paper tries to objectively define the underlying metrics of evaluation.
3. The benchmark development involved a rigorous process, starting with user studies to establish a baseline, followed by testing various LLMs, which adds credibility and depth to the analysis.
4. While the findings align with expectations, quantifying the gap between human and LLM performance is a valuable contribution. It enables the research community to approach improvements in this field with a more data-driven perspective, facilitating measured, progressive advancements.
summary: The paper introduces a benchmark designed to assess the efficacy of large language models (LLMs) in evaluating AI-generated images (AIGI). As the field increasingly depends on LLMs for this evaluation—sidestepping the high costs and time commitments of traditional user studies—quantifying the quality and reliability of LLM-based assessments is essential. While it's generally accepted that LLM evaluations fall short of human assessments, this paper provides a systematic analysis of the performance gap across various LLMs, comparing open-source and closed-source models to human evaluations.

The benchmark defines several key metrics within two primary dimensions: Semantic Reasoning and Quality Perception. Using this framework, the study measures the performance of multiple LLMs, revealing a substantial disparity between human judgment and LLM performance in AIGI evaluation.
weaknesses: 1. While the metrics cover several important facets of semantic reasoning, they lack a rigorous scientific foundation, raising questions about whether they capture the full scope of semantic understanding as implicitly perceived by humans. Specific dimensions of semantic reasoning, such as cultural nuances, or emotional depth, may be missing from the current metrics, which could impact the holistic evaluation of AI-generated images. As such, while the comparisons of different LLMs using these metrics provide intriguing insights, it remains questionable whether these metrics are robust enough to serve as a truly holistic benchmark for evaluating semantic reasoning in AI-generated images.

2. The number of images used (~2,000) feels arbitrary and may be insufficient to capture the nuanced aspects of reasoning and quality perception required for a comprehensive evaluation. Expanding the dataset to around 5,000–10,000 images, with careful attention to diversity across image types and contexts, could improve the robustness of the analysis. Additionally, it would be helpful for the authors to provide a rationale for this dataset size or acknowledge any limitations they faced in scaling up.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: - I would recommend authors check out SelfEval [1] as another source of evidence that external models cannot be reliable for evaluating T2I models. Please discuss it if relevant.
- In my experience there is a huge variance to the responses provided by  LLMs/LMMs. Did the authors compute variance of the scores or perform any statistical significance studies?
- L272 controversial -> counter factual
- In the introduction (L112-L117), in my opinion, authors should provide some numbers to make the point that LMMs are still not masters at evaluating AIGIs. Right now authors state that "there remains a considerable gap and significant room for improvement". Instead providing some numbers can make it more straightforward.

[1] Sai Saketh Rambhatla, Ishan Misra, SelfEval: Leveraging the discriminative nature of generative models for evaluation
soundness: 4
strengths: - Authors address a very important problem: Are current LLM/LMMs good enough to be used as judges for generative models? This line of research can provide valuable insights to train better LMMs for understanding AIGIs.
- A-Bench along with standard LMM evaluation benchmarks provide a complete picture of an LMMs capability to understand both real and AI generated images.
- The paper is well written and very easy to follow containing all the details necessary for reproduction.
- The experimental section is exhaustive with comparisons provided for both proprietary and open-source LMMs.
summary: - Human evaluations are the gold standard for evaluating generative models especially text-to-image (T2I) models. However, they are expensive. An alternative is using automatic metrics and Large Multi-modal Models (LMMs) are a popular choice.
- LMMs are trained on real images and AI-generated Images (AIGIs) are out of domain for LMMs questioning their reliability as evaluation models. 
- This work proposes A-Bench a diagnostic benchmark for assessing the reliability of LMMs for evaluating AIGIs. 
- A-Bench consists of two subsets 1) A-Bench P1 to evaluate the text faithfulness or prompt adherence of T2I models and 2) A-Bench P2 to evaluate the quality of the generations.
- Authors samples 2864 AIGIs from 16 open and closed source T2I models. For each generation, they sourced human experts to annotate question-answer pairs and computed the accuracies of popular proprietary and open-source LMMs. 
- Authors report that 1) Proprietary LMMs are better than open-source counterparts for text faithfulness, 2) proprietary LMMs perform as well as humans on simple enough prompts and 3) LMMs are not good models to evaluate the generation quality of AIGIs.
weaknesses: - I didn't find any major weakness with this work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Q1) What is SRCC/PLCC in the introduction paragraph? It seems this abbreviation is never explained in the paper.

Q2) In section 4.1, "It’s worth noting that the instruction prompt might slightly differ for different LMMs according to the official setting." Why the instruction prompt is slight different for different LMMs? How will it impact the performance of LMMs?

Q3) For the human annotators, how are they recruited? What kind of training were they given? How many instances are labelled by each human annotator? I am also interested in the total time required to build this benchmark.
soundness: 3
strengths: S1) The paper is well-written and well-organized. All the relevant details are included in the main paper and appendix. The evaluating method also seems rigorous.

S2) A good pitch in studying LMMs directly through question answering instead of studying the effectiveness of certain LMM-based metrics. This helped to shed a light on the true capabilities of current LMM-based image evaluation metrics.
summary: This paper study the multimodal LLM's ability in the context of image evaluation. Instead of studying the effectiveness of certain LLM-based metrics, This work aims to identify whether multimodal LLM (LMM) are truly capable of evaluating AI-generated images through a question answering benchmark. Proposed a benchmark that contains 2,864 set of questions which can be categorized into 6 categories, involving semantic understanding and quality perception. After benchmarking a total of 18 LMMs and comprehensive analysis, the authors came up with a conclusion that LMMs are still not masters at evaluating AI-generated images.
weaknesses: W1) The paper would have provided more insights if the authors also studied the reasoning to verify if the LMMs truly understand how to evaluate for each categories (i.e. did the reasoning fully explain the choice made by the LMM?). This might help to explain the gap between the performance of LMMs and humans. I suggest conducting a study on small subset for each categories and see how the reasoning was aligned to the choice made.

W2)  It would also be desirable to see what kind of images LMM evaluate poorly across each categories in AIGI. A more detailed of diversity analysis on the AIGI dataset is required. E.g. for Basic Recognitions, how much portion of the questions are regarding recognitions of animal, human, or artifacts? Are these LMMs doing poorly on particularly certain type of objects?

Meta-review: Summary
The paper examines whether multimodal models can evaluate image generation models. The authors propose a benchmark, A-Bench, that can evaluate both the text alignment and the image quality of generation models. A-Bench is a diagnostic benchmark, and the authors use different multimodal models. Their key finding is that not all multimodal models can serve as evaluators, and certain close-sourced models are more suited for this task.

Strengths
1. The paper studies an important problem: easing the evaluation of generation models by using image understanding models.
2. The experiments in this paper cover a wide variety of models for both generation and understanding. This comprehensive study is valuable.
3. The conclusions drawn in the paper are novel and important for the research community.

Weaknesses
1. The size of the diagnostic benchmark seems small. While the authors do provide the comparison to MMBench, I believe MMBench is used in conjunction with multiple benchmarks. If the authors intend such a use for A-Bench, it would be good to clarify in the paper.
2. Unexplained abbreviations as also pointed out by one reviewer.

Suggestions
Maybe using a stronger Llama image understanding model can strengthen the paper ? The closed sourced systems on GPT and Gemini are hard to use at scale for many researchers.

Justification
This is a well written paper that studies an important problem. Technically sound and well executed.

Decision: Accept (Poster)