A-Bench: Are LMMs Masters at Evaluating AI-generated Images?
Authors: Zicheng Zhang
Haoning Wu
Chunyi Li
Yingjie Zhou
Wei Sun
Xiongkuo Min
Zijian Chen
Xiaohong Liu
Weisi Lin
Guangtao Zhai
Venue: ICLR 2025 Poster (2025)
Keywords: Large multi-modal models
AI-generated images
Benchmark
Primary area: datasets and benchmarks

Abstract: How to accurately and efficiently assess AI-generated images (AIGIs) remains a critical challenge for generative models. Given the high costs and extensive time commitments required for user studies, many researchers have turned towards employing large multi-modal models (LMMs) as AIGI evaluators, the precision and validity of which are still questionable. Furthermore, traditional benchmarks often utilize mostly natural-captured content rather than AIGIs to test the abilities of LMMs, leading to a noticeable gap for AIGIs. Therefore, we introduce **A-Bench** in this paper, a benchmark designed to diagnose *whether LMMs are masters at evaluating AIGIs*. Specifically, **A-Bench** is organized under two key principles: 1) Emphasizing both high-level semantic understanding and low-level visual quality perception to address the intricate demands of AIGIs. 2) Various generative models are utilized for AIGI creation, and various LMMs are employed for evaluation, which ensures a comprehensive validation scope. Ultimately, 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts. We hope that **A-Bench** will significantly enhance the evaluation process and promote the generation quality for AIGIs.

Published as a conference paper at ICLR 2025
A-BENCH: ARE LMMS MASTERS AT EVALUATING AI-GENERATED IMAGES?
Zicheng Zhang", Haoning Wu Chunyi Li', Yingjie Zhou', Wei Sun', Xiongkuo Min Zijian Chen', Xiaohong Liu', Weisi Lin?, Guangtao ? Shanghai Jiaotong University, "Nanyang Technological University
"qqual contribution. "oorresponiing authors. Project Page: httpss//githuub.com comm// Future/A Bench
From Composilion Mdnttffiction
From Contradiction Overcome
From Generative Distortion Assessment
py roctann clmber'
Viv the sold searts geoertian pewrztieeooottee dooutee
Rodi bebenc
hameray
Rully sevss
Oir dus kob compusos
dan voroeee
00744 Resecus doci irad up salind X
61954 Sespeuu-A X
G0T744 RaggorAA X
X
Figure 1: Error cases from the A-Bench.
ABSTRACT
How to accurately and efficiently assess Al-generated images (AIGIs) remains a critical challenge for generative models. Given the high costs and extensive time commitments of user studies, many researchers have turned towards employing large multi-modal models (LMMss as AIGI evaluators, the precision and validity of which are still questionable. Furthermore, traditional benchmarks often utilize mostly natural-captured content rather than AIGIs to test the abilities of LMMs, leading to noticeable gap for AIGIs. Therefore, we introduce A-Bench in this paper, a benchmark designed to diagnose whether LMMs are masters at evalu- ating AIGIs. Specifically, A-Bench is organized under two key principles: I) Emphasizing both high-level semantic understanding and low-level visual qual- ity perception address the intricate demands of AIGIs. 2) Various generative models are utilized for AIGI creation, and various LMMs are employed for eval- uation, which ensures comprehensive validation scope. Ultimately, 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers an- notated by human experts. We hope that A-Bench will significantly enhance the evaluation process and promote the generation quality for AIGIs.
INTRODUCTION
One look is worth a thousand words. Inspired by this age-old adage, numerous researchers dedicate their efforts to developing text-to-image (T2I) models that vividly bring text to life through imagery. These T2I models, driven by free-form text prompts, aim to create images that accurately align with the text and showcase high perceptual quality. Innovations such as AlignDRAW (Man- simov al.] 2015) and the text-conditional GAN (Reed al., 2016) have introduced differential architecture for image generation. The field continues to advance with the development of stable diffusion models (Saharia et al.) 2022, Rombach et al.) 2022b), significantly propelling T2I tech- nology forward. On the commercial front, major corporations leverage vast-scale data to launch stunningly effective T2I models, such as DALL-E (Ramesh et al., 2022). ). Midjourney (Holz 2023), Parti YYu et al.) 2022). etc. However, despite their diversity and widespread adoption, all these advanced T2I models occasionally face issues of low alignment with prompts and low perceptual quality in creating Al-generated images (AIGIs), necessitating careful evaluation and improvement.

---

Published as a conference paper at ICLR 2025
Increasing Usage of LMM as AIGI Evaluators
A-Bench Diagnosing LMMs on AIGI Evaluation
Text prompt
AIGIs
Diognosis
Port2 Diognasls
High-lewei Semontic Understonding
Low-level Quelity Perception
GOO
T21 Models
LMM Evaluation
Reliable
What the most severe technic distortton la the Iaage?
Expectations for LMMS AIGI Evaluators
growing on: trees? d. LLea [ B.Woeccy C, Flower
Undgerpesore
Principles: Semontic Alignment vs. Perceptual Quality
Overegesere P. Bar
A-Bench
Analyze
Report
Annotote
Construct
GPT-4V
Find Strengths Wiolmesses
Experts
AIGis
A-Bench
Gemini
Figure 2: The proposed A-Bench is designed to find out whether LMMs are reliable for T2I AIGI evaluation. Instead of directly assessing the performance of LMM-based metrics, we evaluate the LMMs themselves behind by examining whether the fundamental questions regarding semantic understanding and quality perception can be correctly answered. Based on the benchmark results, we can then 'diagnose' the strengths and weaknesses across various LMMs.
The alignment and quality evaluation of AIGIS present significant challenges that small expert mod- els attempt to address. Although these small expert models offer some solutions, they possess in- herent drawbacks and often fail to meet contemporary demands. Specifically, for alignment assess- ment, CLIP-based similarity models struggle with accurately judging alignment, particularly with complex text prompts (Radford et al.) 2021b). When it comes to quality evaluation, traditional im- age quality/aesthetic assessment methods (IQA/IAA) are not capable of identifying AIGI- -eGerrenee distortions (Wu et al. 2023a b), rendering them unsuitable for this specialized task.
Many researchers are increasingly relying on large language models (LLMs) and large multi-modal models (LMMss for their human-like processing capabilities, which are presumed to enable aceu- rate judgments of alignment and quality in generated content. Consequently. many LMM-based evaluators have been developed, including VIE-Score (Ku et al. 2023), Prometheus (Kim et al.)
2023), VQAScore (Lin et al., 2024), GPT4V-Eval (Zhang et al. (003bb) TIFA (Hu et al. 2023). and Davidsonian Graph (Cho et al. 2023). etc. However, a fundamental question remains:
Are LMMs reliable for evaluating T2IAIGIS?
These LMM-based metries traditionally employ evaluation criteria such as SRCC/PLCC to deter- mine their reliability. However, this approach only reveals how well the metrics perform, without shedding light on their specific strengths and weaknesses. To address this gap. we propose con- ducting comprehensive 'diagnostic' benchmark - A-Bench, focusing on LMMs' capabilities in semantic understanding and quality assessment. Rather than directly evaluating these LMM-based metrics, focus on the L.MMs themselves behind. We move away from computing SRCC/PLCC criteria and instead examine whether the fundamental perceptual questions can be correctly an- swered, which the core basis of all LMM-based evaluators. To initiate our exploration on the AIGI evaluation abilities of LMMs, we first construct the A-Bench centered on a pivotal question:
What do we expect from LMMs as AIGI evaluators?
The answer lies in the capabilities of semantic alignment and quality evaluation. We then define two key diagnostic subsets: A-Bench P1 -highh-e--- semantic understanding, and A- Bench"2 "2 +loow--vll quality perception. For high-level semantic understanding, A-Bench"" targets three critical areas: Basic Recognition, Bag-of- Words Pitfalls Discrimination, and Outside Knowl- edge Realization, which are designed to progressively test the LMM's capability in AIGI semantic understanding, moving from simple to complex prompt-related content. For low-level quality per- ception, A- A-Bench P2 concentrates on Technical Quality Perception, Aesthetic Quality Evaluation, and Generative Distortion Assessment, which are designed to cover the common and AIGI-specific quality problems. The aspect selection is meticulously designed to encompass the most prevalent ap- plication scenarios. Specifically, a comprehensive dataset of 2,864 AIGIS sourced from various T2I models is compiled, including 1,408 AIGIs for A-Bench and 1, 1,456 for A- A-Bench P2, Each AIGI is
2

---

Published as a conference paper at ICLR 2025
paired with question- answer set annotated by human experts. We then test 23 prominent LMMs, including both open- ppen-s and closed-soue source models, on the A-Bench. From the results that the best LMM still falls behind humans by a large margin, can derive the following conclusion:
LMMs are still not masters at evaluating AIGIs.
All LMMs lag behind even the poorest human performance A-Bench, and there is a substantial disparity between open-source LMMs and closed-source LMMs. The performance across different subcategories fluctuates for both A-Bench" P1 and A-Bench "2 indicating that LMMs are not yet robust for different evaluation scenarios for AIGIs. There remains a considerable gap and significant room for improvement before LMMs be considered masters of evaluating AIGIs.
In summary, we systematically explore the capabilities of LMMs in semantic understanding and quality perception, both crucial for their role as AIGI evaluators. These two essential capabilities constitute the core of the proposed A-Bench, the first 'diagnostic" benchmark specifically designed for LMM assessment in AIGI evaluation. Our contributions are summarized as follows:
We carry out the first 'diagnostic' benchmark on AIGI evaluation for LMMs, which consists of 2,864 AIGIs (sampled from various T2I models) paired with question-answer sets on both high-level semantic understanding and low-level quality perception.
A detailed discussion is made about what to 'diagnose'. Semantic understanding is sub- divided into Basic Recognition, Bag-of- Words Pitfalls Discrimination, and Outside Knowl- edge Realization while quality perception is broken down into Technical Quality Percep- tion, Aesthetic Quality Evaluation, and Generative Distortion Assessment.
From the benchmark results, several insights are gleaned, which enable to diagnose various issues with different LMMs and assist in their improvement for AIGI evaluation.
RELATED WORKS
2.1 LARGE MULI-MODAL. MODELS
Large language models (LLMs), such as GPT-4 (OpenAl 2023), ) T5 (Chung et al., 2022). and ILaMA (Touvron et al. 2023). exhibit exceptional linguistic capabilities in general human knowl- edge domains. By integrating visual input via CLIP (Radford et al. (021a) and additional adaptation modules, large multi-modal models (LMMs) (Li et al. [023a Gao al. 2023 Liu al. 2023b] Dai et al., 2023: Zhang et al. 2023a) are capable of addressing diverse multi-modal tasks, including image captioning, visual question answering, visual segmentation, visual classification, visual rea- soning, etc. Namely, OpenFlamingo (Awadalla et al.] (202)) initially integrates several gated cross- attention dense blocks into the pretrained language encoder layers. InstructBLIP (Dai et al. 2023) extends BI.IP- (Li et al.) 2023c) by incorporating vision-language instruction tuning. To further develop open-source LMMs, many works have employed GPT-4 (OpenAl 2023) to create data for vision-language tuning, such as LLaVA series (Liu et al. 2023b (2024). However, whether these LMMs are masters at evaluating T2I AIGIS is still questionable, which needs further investigation.
2.2 MULTI-MODAL BENCHMARKS
Benchmarks such COCO Caption (Chn et al. 2015) and Nocaps (Agrawal et al. 2019) evalu- ate the capability of models to generate textual descriptions for images. Subsequently, benchmarks like GQA (Hudson & Manning 2019) and OK-VQA (Marino et 2019) focus on visual question answering, assessing multi-modal models' visual perception and reasoning abilities. Further com- plexities are added in benchmarks such as TextVQA (Singh al. 2019) and ScienceQA (Lu al. 2022). which incorporate OCR tasks and commonsense reasoning, respectively. MME (Fu et al., 2 and MMbench (Liu et al. 2023c) provide comprehensive evaluations of LMMs across vari- ous subtasks. Additionally, MMMU (uue et al. 2023) targets extensive multi-disciplinary tasks that require college-level knowledge and sophisticated reasoning. More recently, Q- Q-Bench (Wu et al. 2023a) focuses specifically assessing the low-level visual perception capabilities of LMMs. De- spite these efforts, there is still gap in systematic benchmarks for assessing the abilities of LMMs in AIGI evaluation, prompting the development of A-Bench to address this shortfall.
3

---

Published as a conference paper at ICLR 2025
Technical quality distribucion for Technical Quality Perception subset
Nunder
Recesntiin +12
Speaift Ferms Racaget 1m4-3866
Bay-of wlards
Viee Opeier
majo Object Rucsnn Reconiiie -388
Bosk eecommaion
Diserimination
4880
Realisation
Anovitage
Outside
Gontrar -Joclnn Overcone eeercaee
-512
0-1 bad
1-2 poor
2-3
3-4
4-5 excellent
Semantic
-416
fair
good
Understanding
(b)
P1-1408
Quality
Aesthetic quality distribution for Aesthetic Quality Evaluation subset
Perception
Technical
P2-1456
Generative Distortion
Quality Perception
Assessment
492
Aesthetic Quality Evaluation
0-1
1-2
2-3
3-4
4-5
bad
fair
good
excellent
(a)
(c)
Figure 3: Illustration of focused aspects and corresponding quality distributions for A-Bench. The focused aspects and the amount of AIGIs employed are shown in (a). The quality scores of AIGIs sampled for Technical Quality Perception and Aesthetic Quality Evaluation subsets are obtained from AIGIQA-20K (Li et al. 2024) and predicted from Q-Align (Wu et al. (023c) respectively.
CONSTRUCTING THE A-BENCH
3.1 KEY PRINCIPLES
Covering High-level and Low-level Attributes. The demand for generating images has become increasingly stringent, with requirements for only accurate adherence prompt specifications but also high visual quality of AIGIs. To ascertain whether LMMs can effectively evaluate whether AIGIs meet these criteria, it essential assess their capabilities in both high-level semantic understanding and low-level quality perception. High-level semantic understanding encompasses basic recognition and the integration of external knowledge, whereas low-level quality perception involves the identification of technical quality, aesthetic appeal, and generative distortions. The detailed focused aspects can be overviewed in Fig. (a).
Ensuring Diverse AIGI Scope. Considering the variety of current generative models and their application scenarios, we have selected broad range of mainstream text-to-image (T2I) generation models to produce Al-generated images (AIGIs). To assess high-level semantic understanding, we design prompts rich content to ensure diversity among the generated images. For evaluating low-level quality perception, we employ uniform sampling to encompass a wide spectrum of visual quality and the corresponding quality distributions are illustrated in Fig 3 (b) and (c). Throughout the benchmarking process, we test multiple open-source and closed-source L.Mss to guarantee a comprehensive evaluation. These measures ensure that our proposed A-Bench encompasses a
diverse and extensive scope. More details about AIGIs collection can be referred to in Sec. A.I
3.2 FOCUSED ASPECTS
The key evaluation aspects of T2I models involve image-text alignment and image visual quality, which correspond to high-level semantic understanding and low-level quality perception abilities. Some representative examples regarding the subcategories discussed below are exhibited in Fig.
3.2.1 HIGH-LEVEL SEMANTIC UNDERSTANDING
To evaluate whether LMMs can effectively assess image-text alignment, we implement the BenchPl which consists of 1,408 challenging multi-modal question-ansee answer pairs that focus high-level semantic understanding for AIGIs. The high-level semantic understanding can be divided into the following subcategories, moving from simple complex prompt-related content:

---

Published as a conference paper at ICLR 2025
Major Object Recogrition
Major Objjcct Recognition
Basic Recognition
Quertion:
Question:
What the hair color of the persen the far left in the picture?
the decorstion en the back wall is the piceure moynificen??
Ansser condidobes:
Answar condidatee:: A Yes
Black
B: Siler
Yallww
Correct
Attributes Awareness
Nouns as Adjectives Awareness
Question:
Question:
Whet is the first. baéble made of from left to right?
there shark-sleck submarine the deep oceer?
Bag of Wor-Words Pittailss Discrimination
Answer candidates:
Answer candidates::
plastic Blue Sheel glass
A: Yes
BBBnchhP
plass
choice:
Composition Identification
Objects Counting
Question:
wechosologiies comp?
How are bants arronged in the
many on ele well? Answer condidsbes:
hanging
Answer condidotes:
At Arronged in aircle, around fire the middle of the
A: 1
8:
3
D- E
space camp
Correct choice:
choice:
Outside Knowledge Realization
Specific Recognition
Overcome
Question:
palrtiig style represenced in the imagee?
the meon larger bhon the sun in the imoge?
flnswer condiáotes:
Answer candidates:
A: Abstract
8. Surreslism
Yes
Expressionism Impretsionism
Correct choica:
Technical Quality Perception
Aesthetic Quality Evaluation
Generative Distortion Assessment
Question:
aestlutic distertions are presene in tiis imaga?
A B--Behhh2
Answer condidotes: A- Lock of clear guidence style Colors are manotonous and unsoordinated D- None of above Correct shoice: P
Question:
Guestioe:
WWat technical distortien moinly exits in this picture?
zenerabive
Answer candidates:
candidatec:
A- Overesposure Spine blur
B. #lomotched left and right ers iitong Ragere
aptions
Underespasure Out facus
choiss:
Corverk duscc: A
Figure 4: Examples of A-Bench. Each AIGI is accompanied by question-answer pair.
Basic Recognition. This aspect concentrates on the fundamental semantic understanding of AIGIs (Nichol et al. 2021 Saharia et al. (022). which can be subdivided into two distinct areas based on the objects of interest: 1) Major Object Recognition, which involves recognizing the pri- mary objects in the image, such humans objects depicted in the foreground. 2) Minor Object Recognition, which pertains to the identification of less-prominent objects within the image, such background elements secondary characters.
Bag-of-Words Pitfalls Discrimination. This dimension focuses on the discriminative semantic understanding of AIGIs crafted with Bag-of-words prompts (encompassing rich descriptive attributes or complex object relationships (Qu et al., 2024)). This can be subdivided into the following aspects related the crucial points of T2I generation alignment: 1) Attributes Awareness, defined as the capability to accurately identify the attributes of objects in AIGIs (Xu et al. 2023 Liu et al., (203cc) 2) Additionally, given that T2I models may incorrectly interpret nouns as adjectives, resulting the unwanted generation of objects instead of the intended attributes (Chatterjee et al.) 2025) Motamed jet al. 2023). we have also introduced dimension called Nouns as Adjectives Awareness to ad- dress this issue. 3) Composition Identification. recognized the ability to correctly comprehend the compositional relationships such as orientation, occlusion, size comparison, and spatial arrange- ment (Wang al. 2024b; Zhang et al., 2024). 4) Number of Objects Counting, regarded as the ability to accurately count the specified objects in the image, which is crucial for assessing whether the AIGI aligns with the numerical specifications of the prompt (Binyamin al. 2024

---

Published as a conference paper at ICLR 2025
Outside Knowledge Realization. This aspect emphasizes the reasoning ability to utilize external knowledge not directly depicted in the images (Schwenk et al. 2022), and can be broken down into the following dimensions: 1) Specifie Terms Recognition: This involves identifying specific scenes and objects related to distinct domains such as geography, sports, science, materials, food, everyday life, creatures, brands, and styles. 2) Contradiction Overcome, recognized as the ability to correctly interpret AIGIs even when their content contradicts established world knowledge, which particularly crucial for evaluating AIGIS generated from controversial prompts (Hou al. 2024).
3.2.2 LOW-LEVEL QUALITY PERCEPTION
Conversely, determine the ability of LMMs on image visual quality, we conduct the A-Bench comprising 1,456 challenging multi-modal question-answer pairs centered on low-level quality per- ception for AIGIs, which can be categorized into the following aspects: 1) Technical Quality Per- ception This indicates the low-level characteristics that directly degrade the quality of images, such as blur, noise, exposure, etc (Su et al.] [02T]] Ying et al. 2020). 2) Aesthetic Quality Evaluation This indicates the attributes that affect the aesthetic appeal of AIGIs and evoke varied human feel- ings, which include color, lighting, etc (Huang et al. (024). 3) Generative Distortion Assessment This indicates the unexpected AIGI-specific distortions (Chen et al., 2023b Li et al., 2023b (2024), such as generative blur caused by low completion, confusing geometry structure, unnaturalness, etc.
3.3 QUESTION COLLECTION
Question Type In the A-Bench, two types of question formats are utilized, including Yes-or-No questions and What questions. The Yes-or-No questions (accounting for 25.9%) are used to evaluate the fundamental judgment abilities of LMMs while the What questions (accounting for 74.1% 1) are more complicated and require LMMs to gain a more comprehensive understanding of the AIGIs.
Human Expert Annotation We have assembled a team of 15 human annotators, each with ex- pert experience in AIGI evaluation, to develop questions for A-Bench. This annotation process is conducted in a controlled laboratory environment, ensuring consistency and reliability. Annotators tasked with designing questions specific to the sub-categories of the AIGIs under review, utiliz- ing their extensive knowledge to determine the content and format of each question. To ensure the highest quality and suitability, each question undergoes rigorous review process, with least three other expert annotators double-checking it. More details can be acquired in Sec. A.3
Question Response Specifically, the example input query to LMMs can be exemplified as:
#User: What painting style is represented in the image? [TM AGE_TOKEN A. Abstract
B. Surrealism C. Expressionism D. Impressionism Answer with the option's letter from the given choices directly.
The answer candidates and correct answers are shuffled during the evaluation process. Since the responses from LMMs can be in various forms (if the correct choice is C) such 'C", "Expres- sionism', 'The painting style of image is expressionism' etc., we employ a GPT-assisted choice evaluation technique proposed in (Liu al. 2023c Wu al. 2023a) validate the correctness of LMMs responses. More details are shown in Sec. A.4
4 EXPERIMENT RESULTS
4.1 BENCHMARK CANDIDATES
To ensure the results are comprehensive and up-to-date, we select the widely used LMMs for bench-
marking. The Proprietary LMMs (closed-source) include Gemini 1.5 Pro (Reid et al. 2024). GPT-
4v (OpenAl 2023), GPT-4o (2024-05-13) (OpenAl 2024). and Qwen-VL-May (Bai et al. 2023)
The Open-source LMMs include VL-72B (Qwen2-72B) (Wang et al. 2024a), MiniCPM-
V2.6 (Qwen2-7B) Yao et al. 2024).. VL2--M
Heirs-Herress-2Y--- Ovis1.5 (Llama3-8B), LLa VAVA-Onn OneA-OOeiiiio Vision (Qwen2-7B) (Lu et al. 2024b), LLM2-1999 (Llama3-
(Chen et al. 2024
8B) (Wang et al.
2023). IDEFICS-2 (Mistral-7B--nat 1n-srrutucc--0
(Hugginglace 2023). DeepSeek-
VL-7B (Lu al.
2024a Internl M-rmmlMMCComii M-XXCMposse22-VVV VI. (Dong et al.
2024), L.L. VA-NeX (Llama3-
6

---

Published as a conference paper at ICLR 2025
Human (Veret
21.44%
53.1%
NAd
Owee2
7638%
Gomin 1.5
7674%
GP744
7338%
IntenVi 400
75.22%
Ovad Uamuu
75.00%
Lawk 1200
72.775
WE BU DA
Cogstai
71.4%%
Lowh 22
11.00%
70.35%
(4.22%
LLak
61.25%
Gencratie
mPsLo 2.10
Contra
12.18%
62.97%
Technkckl
Owcrvv
63.4115
DastLave-Th m
28.00%
(Reed)
(Wkrr))
Queet-Vi
Riy--n on
51.0%
Gomin 1.5 Pro
GPT 4g
Intervi2 408
randem pues
33.70%
Quan-VLMMM Max
Ovida S-Limma ab
LLakk Dww.-onn///ainn
(a) Overall results of A-Bench.
(b) Detailed results of A-Bench.
Figure 5: A Quick Look of the A-Bench outcomes. (a) showcases comparative analysis of the overall accuracy between humans, 23 selected LMMs (both closed-source and open-source), and random guess. (b) displays radar chart that details the accuracy performance (subtracting the accuracy of random guess) of the top-7 LMMs across various subcategories of A-Bench.
8B), LLaVA-NeXX (Owen-728) LLaVA-NeXT (owen-110B (Luu et al. 2024), OW12 (LLaMA-78 Ye al. 2023), LLaVA-v1.5 1.5 (Vicuna-vl.5-2) 5-7BB, LLaVA-vI. 1.5 (.cuma-v1.5-13B) (Liu ct al. 2023a). CogVLM-17P (Vicuna-vl, v1.5-7B) (Wang et al. 2023). Qwen VL (Qwen-7B) (Bai ct al.
2023 Lava (Mistral-7B) (Liu et al. (023b)), and Fuyu-8B (Persimmon-88) (Adept 2023). All
LMMs are tested with zero-shot setting. It's worth noting that the instruction prompt might slightly
differ for different LMMs according to the official setting.
4.2 HUMAN PERFORMANCE
For human performance A-Bench, conduct user-study experiment with five ordinary people in a controlled laboratory setting. Initially, participants familiarize themselves with the tasks through exposure similar cases. Subsequently, they select the appropriate responses for the questions posed in the A-Bench. To maintain consistency with the conditions experienced by LMMs, the order of questions is randomized, and participants receive additional information beyond the AIGIs, questions, and answer options. The best and worst performance is included for comparison. More details about acquiring human performance can be referred to in Sec A.5.
4.3 FINDINGS A-BENCH
General Observation:
Proprictary LMMs deliver performance comparable to the best open-
source LMMs. concise overview of the A-Bench results provided Fig. 5 revealing several general insights: 1) All LMMs significantly outperform the random guess. indicating their capa- bilities in handling AIGI evaluation, with Qwen2-VL-72B leading, closely followed by Gemini 1.5 Pro and GPT-4o. Notably, among the open-source LMMs, which preferred for AIGI evalua- tions due to their accessibility and modifiability, Qwen2-VL. VL-72B stands out, even outperforming the best closed-source competitors. 2) Even the lowest performance by humans surpasses that of all LMMs, with a noticeable 14.62% gap compared to the top-performing LMM, Qwen2-VL-72B, indicating that LMMs are still far from adequately performing AIGI evaluation as humans. 3) A closer examination of the radar chart in Fig. 5 (b) shows that top LMMs exhibit varied perfor- mances across different sub-categories, suggesting a lack of robustness, while humans show more consistent and balanced performance across these categories, highlighting areas where LMMs need further improvement.
Findings of A- A-BenchPI LMMs excel at basic recognition tasks but tend to be less effective when it comes to nuanced semantic understanding. The performance results of LMMs on the A- A-Bench Pi subset, as detailed in Table | reveal several key insights: 1) Almost all LMMs show
7

---

Published as a conference paper at ICLR 2025
Table I: Benchmark results on the A-Bench subset, which reveal the high-level semantic un- derstanding abilities across LMMs. The best performance is marked in bold and the second perfor- mance is underlined for both proprietary and open-source LMMs respectively.
Categories (MMM)
Basic Recognition Mujorf
Tog-of-Worrd
Outside Knowledge Toou
Ovenall
Mitior
N. Adj.
Coup.1
Number?
Contaa."
HUMAN (WORST)
95.18%
94. 24.
96.78%
88.20%
85.49%
82.46%
81.76%
88.91%
92.40%
HUMAN (BEST)
95.40%
95.21%
99.42%
95.17%
93.34%
91.73%
84.29%
96.05%
94.02%
Proprictury LMMs:
GEMINI 1.5 PRo
93.82%
95.18%
94.38%
80.27%
22.14%
79.35%
72.88%
61.56%
84,70%
GPT-4V
92.95%
96.00%
87.10%
52.67%
64.39%
68.84%
77.60%
66.73%
83.60%
GPT-40 (2024-05-13)
94.34%
95. 14%
91.9996
79. 79.54%%
76.40%
73.30 10.30%%
77. 77.47%
68.59%
85.44%
QWEN-VL-MAX
92.56%
94.75%
91. 19.99%
85.78% %
68.94%
75.85%
78.94%
65.05%
84.47%
Open-source LMMs:
Qwen2-VL-T7 VL--BB (Qwn2-7288)
95.15%
94.61%
92.31
83.66%
71.37%
78.20%
79.12%
68.99%
86.02%
MiniCPM-V2.6 (Qwen2-78)
93.01%
93. 221
933 93.44%
81.21%
78.31%
77 0616
79.32%
67 86%
84.98%
InterVL2--400 (Nous-Hemmes:2 2-B-34B
94.86%
93.87%
93.56 %
80.32%
79.88%
78.01%
77. 47.444%
69.54%
15.
1.is (Llama3-88)
92.79%
92. 262
92.12%
80.55%
78.61%
78.59%
78. 78.34%
87%
85.08%
VA-VV- Vision (Qwam1-7H)
92.53%
92.01%
92.077%
81.125
79.334
77.98%
79.02 79.02%%
69.91% %
84.88%
Cogy M2-198 (1lama3-88))
93.31%
92.70%
89.97%
75.41%
64.63%
66.631
88%
51%
82.55%
IDEFICS-2 (Minmall77 ht-../77ttttttr.rr........
89.92%
91.87%
86.50%
75.45%
61.36%
71.04%
73.31%
62.91%
10.14%
DeepSock-VL-Y
91.48%
91.15%
82.44%
83.73%
63.38%
75.40%
60.32%
81.42%
Internl M4-XCompose2-N11 (Interal M2)
92.79%
95.21%
86.38%
62.645%
68.87%
72.22%
70.77%
64. 35%
81.89%
LLXVA-NeXT (Elama3-5B)
92.72%
92.40%
91.15%%
83.62%
61.04%
67.07%
76.23%
62.94%
$2.88%
LLaVA-NeXT (Qwen-728)
94,37%
92.72%
91. 49%
81.61%
62.40%
73.39%
77.15%
61. .44%
83, 83,99%
LLAVA-NeXT (QweneJ -108)
93.83%
91.10%
90.43%%
84.71 %
67.76%
67.70%
76.25%
64.288%
83.66%
mPLUG-Ow12 (LLaMA-7B)
85.29%
86.26%
83.87%
79.66%
53.73%
57.85%
71.14%
58.47%
76.10%
LLaVA-v1.5 (Vicunavel 5-78)
87.82%
88 635
83.86%
71%.41%
61.39%
65 65..671
774,76%
62. 69%
78.86%
LLaVA-v1.5 (Vicuaavll.5. 1333B
88.60%
89.57%
86.48%
79.52%
62.334%
58.82%
74.81%
61.556%
79.72%
CogVLM-1778 (Vicoma-11.5-7)
90.38%
95.177%
85.8 85.89%
77.47%
49.56%
47.882%
73.34%
61.34%
78.61%
86.14%
86.
81.38%
777.47%
52.7256
61.225%
71.61%%
57.32%
76.399%
Lak (Miatrll-B)
88.91%
81.311%
77.42%
73.81%
52.18%)
62.32%
68.37%
49.02%%
74.33%
RByu-8B (Persimmon-58)
81.41%
68. 27%
66.72%
57.45%
42.24%
48.322%
61.16%
29.651%
63.12%
rundom guess
52,27%
37.22%
31.03%
42.82%
29.85%
29.78%
26.51%
32.13%
30.80%
good performance in Basic Recognition, suggesting that they are quite adept at fundamental mantic understanding, which includes recognizing foreground and background objects in AIGIs. 2) However, their effectiveness diminishes in more complex tasks such as Bag-of- Words. particularly in subcategories like Nouns as Adjectives Awareness, Composition Identification, and Number of Ob- jects Counting. These areas require deeper semantic understanding and reasoning, which critical as users often employ complex prompts that include such nuanced elements. The LMMs' under- performance here indicates potential challenges in accurately aligning AIGIs with user prompts. 3) Additionally, Outside Knowledge poses significant challenges, with LMMs generally achieving un- satisfactory performance in the Contradiction Overcome subcategory, where AIGIs contain content that defies common sense, requiring LMMs to override their prior knowledge to respond correctly. The subcategory Specific Terms tests the knowledge base of LMMs, where proprietary LMMs gen- erally perform better due to being trained on more recent and extensive datasets. 4) Therefore, to improve the evaluation of text alignment in AIGIs using LMM, it is recommended to simplify overly complex prompts. By employing divide-and-on aedee-ndd-oouuuee conquer approach to break down complex prompts into shorter ones, sequential judgment can effectively enhance accuracy.
BaggoofWordds
Findings of A- A-Bench P2; LMMs are poor quality evaluators. The performance results of LMMs on the A-Bench" subset, as shown in Table illustrate a notable disparity in capabilitiess 1) There is a significant performance gap of approximately 23.10% between the top-performing LMMs and human evaluators, highlighting that LMMs lag considerably in quality perception and struggle to ac- curately assess the quality of AIGIs. 2) Furthermore, most LMMs exhibit their weakest performance in the Generative Distortion Assessment subcategory (except effectiveness at identifying unexpected generative distortions, such as unnatural appearances and incorrect geometric structures. 3) Interestingly, while humans generally perform better in Techni- cal Quality Perception compared to Aesthetic Quality Evaluation. LMMs show similar performance levels in both subcategories (except Qwen2-VL-72BB VL-72B and MiniCPM-V2.. This difference likely stems from the more objective nature of technical quality assessments, which leads to more con-
VL-2BB, suggesting their in-
8

---

Published as a conference paper at ICLR 2025
subset, which reflect the low-level quality perception abilities across LMMs. The best performance is marked in bold and the second performance is underlined for both proprietary and open source LMMs respectively.
Table 2: Benchmark results on the A-Bench "2
Categories
Technical
Aesthetic?
Generative
Overall
(TJM)
HUMAN (WORST)
94.32%
84.49%
86.25%
90.50%
HUMAN (BEST)
94.6918
86.01%
93.00%
92.22%
Proprictary LMVs:
GEMMNI 1.5 PRe
71.22%
77.61%
59.07%
69.12%
GPT-4v
67. 7..%
68.34%
58.02%
64.31%
GPT-4o (2024-05-13)
70.59%
61.61%
67.92%
66.88%
QWEN-VI-MAX
71.31%
69.77%
58.56%
66.21%
Open-suurce
Owen2-V1--729 (Qwen2-728)
74.22
60.31
70.23%
68.99
MiniCPM-V24 (200PMM222000022222
69.10%
60.14%
60.47%
64.01%
LL2cm8 Honn--Hmmes-2-----ee
66.28%
63.21%
50.10%
59, 59.22%
Ovis15 (Llama3-88)
70.83%
67.82%
55.39%
64.50%
LLaVA-One (Qwen2-78)
68. 84.8%%%
79%
54.27%
63,78
Cog VLM2-199 (Liamu3-88)
64.21%
61.33%
56.75%
60.73%
IDEFICS-2 (Mistrol-7B Miszral-7B-lastmmcr--0.... Instrucct-00.2)
62.00%
68.76%
47. 47.12%
59.11%
DeepSeck-VLL VL-BBB
55.91%
53,791
47.54S
52.36%
MiernLLM-XComposee222 (InterLLMM))))
62.29%
63.37%
50.26%%
58.58%
LLaVA-NeXT (Llamu3-SB)
58.59%
48.57%
52, 0009
53.13%
LLaVA-NeXT (@wenn
59.91%
55.51%
59.809
58.42%
LLaVA-NeXT (Qwenn- 1100)
64.69%
57.20%
63.645
61.89%
mPLUGG-Ww2
53.811
LLaVA-v1.5 Vicuna-x1.5.72
45,90S
41.33%
54.59%
47.12%
LLaVA-v1.5 Vicama-vl.5-1330
46.08%
41,22%
48. 10%
45.54%
CogVI.M. M-178 (Viemae-r].5. (-5mm
54.76%
48.45%
52.47%
$1.36%
Qwen (Lweeeen 7B)
49.46%
34.34%
50.499
Baki.Lava (Mirral-7B)
47,88%
33.37%
48.46%
43.39%
Fuyu-8B (Persimonn-88)
44.61%
30,2396
45.65%
40,20%
random guess
31,87%
32.92%
33.14%
32.63%
sistent evaluations among humans, whereas aesthetic quality, being more subjective, results in broader range of opinions and consequently lower performance scores.
Human vs. Proprietary LMMs Proprictary (closed-source) LMMs are regarded as closely mir- roring human perception and demonstrate superior performance, particularly in zero-shot settings for evaluating AIGI. Therefore, here we make finer discussion about the human and proprietary LMMs. 1) Beginning with a detailed comparison of human and proprietary LMMs, we observe that proprietary LMMs achieve human-level performance Basic Recognition, indicating their ability to correctly assess AIGI alignment when prompts are simple. 2) Despite this, LMMs encounter difficulties in the Bag-of- Words aspect, especially in identifying composition and counting objects, which highlights their limitations in handling complex compositional relationships and specific object counts. 3) the Outside Knowledge domain, proprietary LMMs show only a slight perfor- mance gap compared to humans Specific Terms, demonstrating comprehensive prior knowledge about specific terms, but they notably lag behind in identifying controversial content. While hu- mans can easily recognize contradictory elements, proprietary LMMs often struggle due to their reliance on common sense, making accurate responses challenging. To conclude, according to the results shown [ proprietary LMMs are competent as evaluators for simple prompts in AIGI, yet they require further improvements for more complex prompts related AIGI content. 4) On the other hand, reveass that LMMs have significant shortcomings low-level quality perception compared to humans, with uneven performance across different quality dimensions. Surprisingly, GPT-4o shows a distinct advantage over other proprictary LMMs recognizing generative distor- tions, suggesting its superior capability in this area. However, the substantial overall difference in quality perception between proprictary LMMs and humans underlines that these models are cur- rently unsuitable for assessing the visual quality of AIGI.
CONCLUSION
In conclusion, the ambition to employ LMMs for evaluating AIGIs exposes considerable deficiencies in their capabilities, revealed by the diagnostic benchmark A-Bench. This benchmark scrutinizes

---

Published as a conference paper at ICLR 2025
the capabilities of LMMs themselves, focusing on their ability accurately address funda- mental questions related to high-level semantic understanding and low-level quality perception. Our findings from A-Bench serve as stark reminder of the current limitations faced by LMMs in the realm of AIGI evaluation. The results underscore that while L.Mss provide valuable insights, their evaluation capacity remains notably inferior to human performance, especially in tasks that de- mand deep semantic comprehension and detailed quality assessment. By identifying specific areas for enhancement and charting course for future research, this study not only underscores the urgent need for further development but also aids in refining the application of LMMs in AIGI evaluation tasks. Future initiatives should focus on augmenting the capabilities of LMMs to reliably match or surpass human performance in these intricate evaluation scenarios.
6 ETHICS STATEMENT
This submission complies fully with the ethical guidelines set by ICLR 2025. We follow ICLR's principles for responsible AI development, ensuring that research avoids any potential harm, bias, or discrimination. The data utilized in this work is sourced exclusively from publicly available open-source datasets and models. Furthermore, our methods prioritize fairness, accountability, and transparency in the evaluation of Al-generated images.
7 LIMITATIONS
Timeliness Concern Creating a benchmark involves generating images, collecting data, training evaluators, and verifying data quality, making the process both time-consuming and costly. As result, it is inevitable that AIGI benchmarks may not always keep pace with the latest technologies models. However, the insights provided by the benchmark in evaluating AIGI remain valuable and offer useful guidance. We are committed to ongoing updates and expansions to ensure the benchmark remains current.
Scale-up Concern Since the A-Bench dataset is fully manually annotated and requires validation by at least three other humans, the annotation process is both costly and time-consuming. As such, it is quite challenging scale up.
8 LEKNOWLEDGEMENT
This work is supported in part by the National Natural Science Foundation of China (623B2073, 62101326, 62225112, & 62301316).
REFERENCES
Adept. Fuyu-8b: A multimodal architecture for ai agents, 2023. URL httph:ttp///// htt::://wwwww.... adept. ai ai/blog/ fuyu-82
Accessed: 2024-05-13.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning scale. In ICCV, 2019.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 01390, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vasion-language model for understanding, local- ization, text reading, and beyond. arXiv preprint a308v:230. 2ttiv:233../2900 /2966, 2023.
Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. Make it count: Text-to-image generation with an accurate number of objects. arXiv preprint arXiv:2406 10210, 2024.
10

---

Published as a conference paper at ICLR 2025
Agneet Chatterjee, Gabriela Ben Melech Stan, Estelle Aflalo, Sayak Paul, Dhruba Ghosh, Tejas Gokhale, Ludwig Schmidt, Hannaneh Hajishirzi, Vasudev Lal, Chitta Baral, et al. Getting it right: Improving spatial consisteney in text-to- -iitgg models. In European Conference Computer Vision, pp. 204-222. Springer, 2025.
Junsong Chen, Jincheng Yu. Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. 2310.00426, 2023a.
Xinlci Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma. et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1682 2024.
Zijian Chen, Wei Sun, Haoning Wu, Zicheng Zhang, Jun Jia, Xiongkuo Min, Guangtao Zhai, and Wenjun Zhang. Exploring the naturalness of ai-generated images. arXiv preprint arXiv:2312.05476 2023b.
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset. and Su Wang. Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-image generation. arXiv preprint arXiv:2310.18235, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction- finetuned language models, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purposse vision-language models with instruction tuning, 2023.
DeepFloyd. If-i-xl-v1.0.. hetps: www. www..ddeppppp...o.ooo 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He. Xingeheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Intemln-xccomposer2. Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401 16420, 2024.
dreamlike art. dreamlike- htt.:l..............
https::/ddae httpp::://dreamiike.eeo art
2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint Xiv:2334.115100, 2023.
Yatharth Gupta, Vishnu V. Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen. Progres- sive knowledge distillation of stable diffusion xl using layer level loss. 2401.02677, 2024.
David Holz. Midjourney.
http://wwwwmcoooueee...... com 2023.
Yufang Hou, Alessandra Pascale, Javier Carnerero-Came Tigran Tchrakian, Radu Marinescu, Eliz- abeth Daly, Inkit Padhi, and Prasanna Sattigeri. Wikicontradict: A benchmark for evaluating 1lms real-world knowledge conflicts from wikipedia. arXiv preprint arXiv:2406. 13805, 2024.
11

---

Published as a conference paper at ICLR 2025
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp-
20406-20417, 2023.
Yipo Huang, Quan Yuan, Xiangfei Sheng, Zhichao Yang. Haoning Wu, Pengfei Chen, Yuzhe Yang, Leida Li, and Weisi Lin. Aesbench: An expert benchmark for multimodal large language models on image aesthetics perception. arXiv preprint arXiv:2401.0822
2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Huggingface. Introducing idefics: An open reproduction of tfat-oof-the-ar visual language model, 2023. URL/sttts://huoinooofaaa
co ctt.eccccllllloooo co/blog/idefics.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. arXiv preprint arXiv:2310.0849/ 2023.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312. 14867, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.
Chunyi Li, Zicheng Zhang, Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. Agiqa-3k: An open database for ai-generated image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 2023b.
Chunyi Li, Tengchuan Kou, Yixuan Gao, Yuqin Cao, Wei Sun, Zicheng Zhang, Yingjie Zhou, Zhichao Zhang, Weixia Zhang, Haoning Wu, al. Aigiqa-20k: A large database for ai-generated image quality assessment. arXiv preprint 24Xiv:2404.03407, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023c.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint 0tt.::20000012991 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved basclines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
Nava-vl.gr thub. c002244//2222-111-1111
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: your multi-modal model an all-around player?, 2023c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng. Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseck-vil vl: Towards real-world vision- language understanding, 2024a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
12

---

Published as a conference paper at ICLR 2025
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXi::2405.20977 2024b.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang. Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module, 2023.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint arXiv:1511.02790 2015.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Saman Motamed, Danda Pani Paudel, and Luc Van Gool. Lego: Learning to disentangle and invert concepts beyond object appearance in text-to-image diffusion models. arXiv e-prints, pp. arXiv- 2311, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10744 2021.
OpenAl. Gpt-4 technical report, 2023.
OpenAI. Hello gpt-4o, 2024. URL http://p./opeaaaaaccoo////en//// cessed: 2024-05-13.
Ac-
PlaygroundAI, 2oyground-2211044px-aa
htttps::/playgrouuee com 2023.
Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, and Tat-Seng Chua. Dis- criminative probing and tuning for text-to-image generation. arXiv preprint arXiv:2403.0432,, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8748-8763. PMLR, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. 2204.06125, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022h.
13

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- sion processing systems, 35:36479-36494, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation, 2023.
Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022.
Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Shaolin Su, Vlad Hosu, Hanhe Lin, Yanning Zhang, and Dietmar Saupe. Koni++ Boosting no- reference image quality assessment in the wild by jointly predicting image quality and defects. In The British Machine Vision Conference (BMVC), pp. 1-12, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing saiiunla model's perception of the world at any resolution. arXiv preprint wrXiv:2409.12199 2024a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.
Yunnan Wang, Ziqiang Li, Zequn Zhang, Wenyao Zhang. Baao Xie, Xihui Liu, Wenjun Zeng, and Xin Jin. Scene graph disentanglement and composition for generalizable complex image generation. arXiv preprint arXiv: 2oXiv:24400.00 24..0044 2024b.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-bench: benehmark for general- purpose foundation models on low-level vision. 2023a.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. Q-instruct: Improving low-level visual abilities for muuti-modalliyy foundation models. arXiv preprint arXiv:231 1.06773,110.,,.. 2023b.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.1 170900 2023c.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-chub: A comprehensive evaluation benchmark for large vision- models. arXiv preprint arXiv:2306.0922 230.09266. 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.00 01000, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan. Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:23/ 1.04257, 2023.
Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. From patches to pictures (paq-2-piq): Mapping the perceptual space picture quality. In CVPR, 2020.
14

---

Published as a conference paper at ICLR 2025
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint ar arXiv:2206. 10789, 2(3):5, 2022.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023.
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim-xcome xcomposer: A vision-lanuuaee large model for advanced text-image comprehension and composition, 2023a.
Xinchen Zhang, Ling Yang, Guohao Li, Yaqi Cai, Jiake Xie, Yong Tang, Yujuu Yang, Mengdi Wang, and Bin Cui. Itercomp: Iterative composition--wwre feedback learning from model gallery for text-to-image generation. arXiv preprint arxiv:2410.07171, 2024.
Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) as generalist evaluator for vision- language tasks. arXiv preprint arX1v:2311.0136,, 2023b.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Figure 6: Overview of the AIGIs from A-Bench" P1
A.1 AIGIS COLLECTION
AIGI collection for A-Bench P1 To ensure that the AIGIS meet the specific subcategory require- ments, we have gathered 2,000 manually-written prompts to serve as the textual foundation. Below, provide examples of these prompts:
1. Basic Recognition ->Major Object Recognition: An elaborate treehouse in thick forest, with children playing inside, rope bridges connecting to other trees, and birds chirping around.
2. Basic Recognition ->Minor Object Recognition: magical fairy ring in a moonlit forest, with tiny glowing fairies dancing and mystical plants all around.
3. Bag-of-Words -> Attributes Awareness: A delicate, frosty, crystal snowflake beside a warm, glowing, amber ember on smooth, slate-gray stone.
4. Bag-of-Words - as Adjectives Awareness: Shark-sleek submarine exploring ocean depths.
5. Bag-of- Words Composiiton Identification: A gamer's setup with consoles and con- trollers on a desk, multiple screens above, and game boxes and snacks partially obscured beneath the desk.
6. Bag-of-Words -> Number of Objects Counting: Six logs in woodpile, stacked so tightly that they seem to form a solid block.
7. Outside Knowledge Specific Terms Recognition: A burometer showing a rapid decrease in pressure.
8. Outside Knowledge Contradiition Overcome: A ship floating above the clouds, sails made of sunlight.
Afterward, we use the collected prompts to create AIGIs, 15 text-to-image generation models are se- lected, which include: Dreamlike (dreamlike art. 2023). Pixart a Chen et al. (2023a). Playground v2
PlaygroundAI (2023). SD1.4 (Rombach (022a), SD1.5 (Rombach et al, (202aa). SDXL. (Rom-
bach et al. 2022a). SSDIB (Gupta et al. 2024). LCM Pixart (Luo et al. 2023). LCM SD1.5 (Luo
ct al. 2023 LCM SDXL (Luo et al. 2023). SDXL Turbo Sauer et al. (0023) DALLE2 Ramesh
et al. 2022). DALLE3 (Ramesh et al. 2022 IF (DeepFloyd 2023). Midjourney v5.2 Holz (023).
Finally, a total of 15 x,000 =30,000 AIGIS are collected. To guarantee diversity, we randomly se- AIGI per prompt. Subsequently,
conduct manual review of these AIGIs to remove any that failed to generate correctly or are unsuitable for annotation. This
lect 2,000 AIGIs, choosing
process results in the final of AIGIs for A-Bench which can be overviewed in Fig. 6
16

---

Published as a conference paper at ICLR 2025
Technical quality distribution for AIGIQA-2OK
Technical quality distribution
Technical Quality Perception subset
Uniform Sampling
Nean Cyanco
Maxt
20K
0:
1-2
2 3
3 good
4.3 excellent
3:
1:2
2.3
4
4.5
bad
fair
bad
excellent
(a)
(b)
Aesthetic quality distribution for AIGIQA-20K
Aesthetic quality distribution for Aesthetic Quality Evaluation subset
Poteo dy Carel
Unifom Sampling
nesthetic from Allgn
Irom Q Align
0-1
1-2
2-3 fair (c)
3-4
4-5 excellent
0-1
1-2
2-3 fair
3-4
4-5 excellent
bad
poor
good
bad
poor
good
(d)
Figure 7: Illustration of the quality distribution transformation.
MWHL WLTD
Figure 8: Overview of the AIGIs from A- A-Bench P2
AIGI collection for A-Bench P2 A-Bench P2 is designed for the quality evaluation of AIGIs. Con- sequently, essential ensure that the collected AIGIs span wide quality range address various practical scenarios. For Technical Quality,
sample 500 AIGIs from the AIGIQA-20K dataset (Li et al. 2024) using a uniform sampling strategy. Specifically, each AIGI in the AIGIQA- 20K dataset is assigned
opinion score (MOS) for technical quality. We apply uniform sam- pling to create more even distributions, as illustrated in Fig. For Aesthetic Quality, the absence of provided aesthetic scores, utilize q-align (Wu al. 2023c). effective quality predictor, to infer the aesthetic values AIGIs. Subsequently. we perform uniform sampling similarly to obtain manually select 500 AIGIs hibiting unexpected AIGI-specific distortions. It is important to note that there is no content overlap among the selected AIGIs, which be overviewed in Fig. g
500 AIGIs for aesthetic evaluation. For Generative Distortion,
17

---

Published as a conference paper at ICLR 2025
Composition Identification Orientation
Composition Identification Orientation
What located to the left of the desks the classroom?
What partially covered by cloaks hanging the background of the magic workshop?
Educalional posters the walls (correct)
A. Shelves (correct)
teacher's desk
Tricks
C. blackboard
C. A magic wand
D. bookcase
Cards spread out
Composition Identification Size Comparison
Composition Identification Spatial Arrangement
In the paleontologist's dig site, seems be largest?
strategy boards placed in all four corners?
A. Human the right
(correct) B. Yes
Human in the middle
The field journal (corret))
Pens
Specific Terms Geography
Specific Terms > Brand
What geography feature is depicied here?
Which brand is famous for
item?
Sandy beach
A. Shell
B. Mangrove forest
Sony
Coral
Nike (correcf)
D. Rugged coastline (correct)
D. Amazon
Specific Terms > Food
Specific Terms
Style
What is the main cooking technique in this dish?
What painting style represented the image?
Baroque
A. Poaching
Rococo (correct)
B. Frying
Neoclassicism
C. Grilling (corrett)
D. Roasting
D. Art Nouveau
Figure 9: Some finer cases for the Baag-of Knowledge-> Specific Terms' subcategories.
of Words - Composition Identification' and 'Outside
A.2 FINER EXPLANATION FOR SOME SUBCATEGORIES
For certain subcategories that require additional clarification for better understanding, we provide detailed explanations here (the corresponding cases are shown Fig.
Bag-of-Words Words -> Nouns as Adjectives Awareness. The 'Noun as Adjectives' illustrates the use of nouns as adjectives to modify objects in AIGIs. Essentially, we aim for the descriptive effect, not for the nouns themselves to visually represented the AIGIs. For instance, shown in Fig row 2 column 2, when describe submarine as "shark-sleck." we do not intend to generate an image of an actual shark. This subcategory designed to test whether LMMs correctly identify such misunderstandings.
2. Bag-of-Words Commpositoon Identification. We categorize composition into four distinct types: 1) Orientation, which assesses the ability to correctly determine the relative spatial positions of objects; 2) Occlusion, which involves evaluating the accuracy in discerning the overlapping relationships between objects; 3) Size Comparison, which tests the ability to accurately judge the size relationships among objects; and 4) Spatial Arrangement, which examines the ability to accurately assess the arrangement of objects within the AIGI.
3. Outside Knowledge SSSeeiifcc Terms. This subcategory covers many aspects, including geography, sports, science, materials, food, everyday life, creatures, brands, and styles. This primarily investigates whether it is possible for LMMs to infer and deduce specific knowledge within these fields based the content of AIGIs such identifying the exact location feature based on geographical attributes, deducing the brand from the characteris- ties of product, recognizing the cooking technique of the food, etc.
18

---

Published as a conference paper at ICLR 2025
Cil Tuom
Ont Tresout bubon reden ort tery
dagure houmiisdge
Neow
Tce
Finm
Prac
Figure 10: Illustration depicting the annotation interface, where experts are presented with the sub- category and are able to record their questions and answers.
A.3 HUMAN EXPERT ANNOTATION
A total of fifteen experts, each possessing professional skills and extensive experience in photog- raphy and AIGIs, participate in the subjective labeling experiment of A-Bench. All experts are informed that their annotation data will be publicly released, and they all agree to this arrangement. The hourly wage for each expert is approximately 12 US dollars, resulting a total expense of about 2,400 US dollars for the whole subjective experiment.
The experiment takes place in a laboratory environment with standard indoor lighting. A Dell 4K monitor, supporting resolution of 3840 x 2160, is used for displaying the interfaces. Sereenshots of the interfaces can be referred to in Fig. TO Each expert annotates up to 30 AIGIs per day avoid fatigue, with every annotation carefully reviewed by least three other experts before acceptance. This approach ensures the highest possible accuracy and rigor of the A-Bench labels, thereby enhancing the precision and meaningfulness the performance testing capability of A- Bench.
A.4 GPT EVALUATION FOR CHOICE JUDGMENT
For some LMMs, the response to the question inquiry may vary. For example, given the correct answer C. Blurry' to the question What the most severe technical distortion of this image?". LMMs may respond in different formats: The image is blurry', There is blur in this image', or 'low clarity'. address the impact of such variations on our evaluation, we' ve implemented a 5- round voting strategy (Wu et al. 2023a). Under this strategy, pose the same prompt, as defined in the templates, five times and determine the final outcome based on the majority of GPT's responses.
GPT Evaluation Prompt Template
#System: You are a helpful assistant that grades answers related to image perception. There are a lot of special terms or keywords related image processing and photography.
#User: Assuming you are a grader, you will now be provided with a question [question] and a set of options Additionally, there will be
[options] with option [options[O]] being the correct provided by respondent. Please determine whether the respondents answer is correct considering the context the question. Even if the word choice is completely the same, you decide based the given options and see whether the one in the onswer is close enough to the given correct answer, The result is I if the answer correct and else the result is Please only provide the result the following format: Result:
[answer]
Example for GPT Evaluation
Question: Which is the most blurry part of this image?
Choices: [Teee house on the left', 'The person in the middle', 'The background', 'The tree on the left']
19

---

Published as a conference paper at ICLR 2025
What color the coral the top left comnr of the picture? red yellow pink blue
CA G
C
C
Figure 11: Illustration of the interface for the user-study.
LMM Answer:
The most blurry part in this image is the house to the left of the person.
5-Round GPT Answers:
["Score: 1", "Score: 1", "Score: 1", "Score: 1", "Score: 1"]
Final Correctness after Voting:
A.5 USER-STUDY ON A-BENCH
To provide human performance the A-Bench, employ five ordinary people in controlled laboratory setting. Initially, participants familiarize themselves with the tasks through exposure to similar cases. Subsequently, they select the appropriate responses for the questions posed in the A-Bench. The user-study interface is shown in Fig. I1
A.6 LMM EXPERIMENT DETAILS
The LMMs undergo testing a zero-shot setting. Proprietary LMMs are evaluated via official APIs, whereas the open-source LMMs (with the exceptions of LLaVA-NeXT Qwen-72B and LLaVA- NeXT Qwen-110B) run on an NVIDIA RTX 6000 Ada with 48 GB of memory. The LLaVA- NeXT Qwen-72B and LLaVA- Qwen-110B are operated NVIDIA H100 with 320 GB of memory. All LMMs operate with default parameters, ensuring that the A-Bench results are readily reproducible.
20

---

Published as a conference paper at ICLR 2025
Table Benchmark results the question types. The best performance is marked in bold and the second performance is underlined for both proprietary and open-source LMMs respectively.
Categories LMM (LLM)
A-Bench")
A-Bench
Overall
Yes-or-not
Whart
Yes-or-nof
Whor"
Yes-or-not
Whart
HUMAN (WORST)
91.21%
92.77% 94.25%
89.45%
91.02%
91.23%
91.88%
HUMAN (BEST)
93.55%
91.80%
92.64%
92.77%
93.39%
Proprietary LMMs:
GEMINI T.5 PRO
81.96%
86.91 %
74.08%
65.57%
76.50%
76.82%
GPT-4v
82.37%
85.86%
71.
60.09%
75.51%
73.23%
GPT-40
84.39%
85.76%
69.76%
65.15%
76.28%
75.81%
VL-MAX
86.70% %
84.02%
68.13%
64.60%
75.79%
74.91%
Open-source LMMs:
CogVi.M2-T99 (IJama3-88)
81.77%
83.26%
63.70%
58.65%
70.55%
71.61%
IDEFICS-2
78.32%
83.84%
63.87%
55.63%
68.91%
69.96%
DeepSeek-VEr VV-7BB
80.72%
82.00%
60.00%
47.15%
66.88 88%
66.48%
Internl Www.LMM-Xoooooo222---1 (2)
82.08%
81.53%
66.49%
53.06%
20.90%
69.83%
LLaVA-NeXT (Llama3-8B)
81.17%
84.1 14.
50%
53.77%
63.89%
68.82%
LLaVA-NeXT (Qwen-728)
83.22%
84.31
57.91%
60.01 %
70.22%
71.55%
LLaVA-NeXT (Qwen-1100
82.99%
83.91%
59.78%
62.87 %
71.76%
73.05 %
mPLUG-Owl2 (LLaMA-7B)
74.92%
78.00%
56.97%
54.36%
64.3 64.38%
67.81%
LLaVA-v1.5 (Vicuma-vl.5-7)
78.27%
82.74%
46.39%
47.97%
58.85%
66.21%
LLaVA-v1.5 (Vicuma-vI.5-123
79.51%
81.47%
47. 47.23%
43.90%
61.41%
63.61%
CogVLM-11B (Vccmnovv..5-8)
76.77%
80.11%
13%
49.71%
64.33%
65.65%
Qwen- VL (Qwen-7B)
72.77%
80.95%
46.22%
44.02%
56.60%
63.39%
BakLLava (Mistrul-7B)
71.01%
78.77%
42.
44.11%
55.61%
60.03%
Fuyu-8B (Persinmmon-85
61.56%
64.22%
38.76%
41.66%
50.06%
52.31%
A.7 QUESTION TYPE PERFORMANCE
We assess the performance disparity between Yes-or-no and What questions among LMMs. The Yes- or-no questions gauge the fundamental judgment capabilities of LMMs, whereas What questions demand more comprehensive understanding. According to the results in Table [3, observed that most LMMs perform better on What questions within A-Bench P1 suggesting a proficiency in processing semantic content. Conversely, A-Bench P2, where LMMs generally show lesser per- formance, they exhibit limited in-depth perception, maintaining only basic evaluative capabilities without comprehensive understanding. leading to poorer performance on What questions. Inter- estingly. human performance consistently excels in What questions across both A-Bench and A-Bench P?, likely due to a broader range of options facilitating easier inference. However, human performance tends to be more balanced compared to LMMs, which may exhibit significant variance, such as IDEFICS-2 2, where there over a 5% accuracy difference between question types, indicating less robustness.
A.8 RESPONSE VARIANCE FOR LMMS
Considering that the accuracy and stability of the benchmark directly affect the quality of the evalu- ation, therefore we conduct the response variance experiment here. First, we use a consistent prompt instruction format to minimize any misunderstanding by LMMs and standardize the output. Addi- tionally, we set the model's temperature parameter to 0, meaning
LMM's output will no longer be affected by randomness. As result, the model will give the same response to the same question each time, eliminating variance.
It's also worth noting that increasing the model's temperature to encourage more diverse and ploratory answers indeed an interesting consideration. To further address the concern about the statistical significance of the experiment, ferent temperature settings across several popular 7B-8B LMMs. The performance is listed in the table below, with the results presented the mean accuracy standard error.
repeat the A-Bench experiment for 5 rounds with dif-
Based on the results, we can observe that when the temperature set to zero, the accuracy results for all LMMs remain consistent across all 5 rounds. As the temperature increases, the average
21

---

Published as a conference paper at ICLR 2025
Temperature
DeepSeck-VI--71
LLaVA-NeXT-SB
LLaVA-v1.5-7B
Qwen-VL-7B
0.0
66.58a0.00
67.7540.00
62.97a0.00
60.41
0.5
65.11=1772
66.4342.09
60.61+2.23
18.17
1.0
62.04=4.51
63.7743.88 33.6
59.2224.01
55.22+6.04
Table 4: Performance comparison at different temperatures for various LMM models.
performance declines and the results become more unstable, with higher standard errors. Therefore, ensure reproducibility and performance stability, we prefer the zero-temperature setting, it more accurately and reliably reflects the performance of LMMs, making it more suitable for practical applications.
A.9 DATA STATEMENT
The A-Bench dataset is released under the CC BY 4.0 license. This includes all associated AIGIs, questions, and answer candidates. However, to prevent incorporation into the training sets any LMMs, the correct answers remain confidential. We believe this precaution will ensure that A- Bench retains its long-term value as benchmark for assessing AIGI evaluation capabilities.
A.10 FURTHER LIMITATIONS AND SOCIAL IMPACT
Limitations While A-Bench uses diverse set generative models and LMMs for evaluation, the choice and number of models might still limit the generalizability of the results. The performance of untested models or newer generative approaches might differ significantly. The rapid advance- ment in AI and generative models may quickly outpace the current setup of A-Bench, necessitating frequent updates or redesigns the benchmark to stay relevant.
Social Impact By improving the evaluation metrics for AIGIS, A-Bench could lead to liable and trustworthy Al-generated content, which is crucial as these technologies increasingly intersect with areas like media, entertainment, and education. Moreover, improved benchmarks and evaluation methods can drive industry standards, potentially lowering the barrier to entry for smaller developers and promoting innovation through clearer performance targets.
22


Reviews:
[{"review_id": "qly712uEDo", "reviewer": "Reviewer_8kKW", "summary": "Due to the existing evaluation models' inability to effectively assess the performance of AIGI tasks, more and more researchers are turning to LMMs for evaluating the quality of generated images. The authors question this approach and design a framework consisting of seven dimensions focused on high-level semantic understanding and low-level quality evaluation to assess the quality of AIGI. By manually annotating 2864 different image quality issues, the authors compare the evaluation performance of multiple open-source and closed-source LMMs and contrast these with human evaluation results, summarizing numerous shortcomings of LMMs in the AIGI quality assessment task.", "questions": "1. Overall, the paper adds the evaluation of LMMs' high-level semantic cognition for AIGI, in addition to previous work using LMMs to assess image generation quality. However, it does not highlight the difference between AIGI tasks and conventional cognition tasks. Could the authors elaborate on this further?\n2. Generally, the authors focus more on evaluating the perceptual capabilities of LMMs, but these perceptual capabilities are more inclined towards the low-level aspects for AIGI tasks. Could the authors further elaborate on the differences between the low-level perceptual aspects of AIGI and some earlier works?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The authors manually annotated a dataset containing 2864 image quality issues, which contributes to the development of AIGI evaluation.\n2. The authors evaluate AIGI quality from high-level semantic aspects like counting and low-level aspects like distortion, providing valuable insights for subsequent general AIGI task evaluations.\n3. The paper's A-Bench includes the evaluation performance of multiple LMMs, offering guidance for researchers who wish to use LMMs for AIGI quality assessment.", "weaknesses": "1. Although A-Bench includes multiple LMMs, it lacks some of the latest SOTA models. Better models such as QWEN-VL2 and MiniCPMv2.6 can be found from opencompass. The paper does not specify the versions of gpt4o used, such as gpt-4o-2024-08-06 or gpt-4o-2024-05-13, which is crucial for future researchers.\n2. The AIGI models used to generate the dataset are somewhat outdated, lacking relatively advanced image generation models such as SD3, PixArt, Flux, etc. Currently, the more outstanding AIGI models often embed large language models, which might significantly impact the evaluation conclusions.\n3. The questions are all manually generated, which is certainly good. However, this makes the evaluation dataset difficult to expand and might lose value as AIGI models rapidly evolve. It would be better if the questions could be designed based on the text prompts of T2I models.\n4. Compared to previous work, The paper's main contribution, i.e., high-level semantic question answering, is not strongly related to AIGI and does not seem necessary to research specifically in the AIGI context.\n5. Two-thirds of the low-level semantic question-answering data come from other datasets, reducing the paper's contribution.\n6. The paper's findings are somewhat unremarkable. It is obvious that closed-source LMMs perform better than open-source ones, and some other findings, such as the LMMs' insufficient perception of distortion, have already been mentioned in works like Q-Bench.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhceFAee74", "reviewer": "Reviewer_WKMw", "summary": "The paper introduces a benchmark designed to assess the efficacy of large language models (LLMs) in evaluating AI-generated images (AIGI). As the field increasingly depends on LLMs for this evaluation—sidestepping the high costs and time commitments of traditional user studies—quantifying the quality and reliability of LLM-based assessments is essential. While it's generally accepted that LLM evaluations fall short of human assessments, this paper provides a systematic analysis of the performance gap across various LLMs, comparing open-source and closed-source models to human evaluations.\n\nThe benchmark defines several key metrics within two primary dimensions: Semantic Reasoning and Quality Perception. Using this framework, the study measures the performance of multiple LLMs, revealing a substantial disparity between human judgment and LLM performance in AIGI evaluation.", "questions": "1. It would strengthen the paper if the authors could provide scientific backing for the proposed metrics, citing sources that systematically define each measure. Specific areas where additional references might be valuable include the validity of semantic reasoning components and quality perception dimensions, to help ensure that the chosen metrics align with established frameworks in the field.\n\n2. Providing further details on the image selection process would clarify the robustness of the benchmark. Specifically, information on the criteria for image selection, the diversity of image types, and the distribution of different content categories would offer valuable context. If possible, outlining how these factors impact the benchmark's representativeness, and validity could further enhance transparency.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The benchmark is undoubtedly useful. Given the growing reliance on LLMs to evaluate various AI-generated content like images, having a comprehensive, quantitative benchmark that assesses the effectiveness of LLMs in evaluation is highly valuable.\n2. The paper tries to objectively define the underlying metrics of evaluation.\n3. The benchmark development involved a rigorous process, starting with user studies to establish a baseline, followed by testing various LLMs, which adds credibility and depth to the analysis.\n4. While the findings align with expectations, quantifying the gap between human and LLM performance is a valuable contribution. It enables the research community to approach improvements in this field with a more data-driven perspective, facilitating measured, progressive advancements.", "weaknesses": "1. While the metrics cover several important facets of semantic reasoning, they lack a rigorous scientific foundation, raising questions about whether they capture the full scope of semantic understanding as implicitly perceived by humans. Specific dimensions of semantic reasoning, such as cultural nuances, or emotional depth, may be missing from the current metrics, which could impact the holistic evaluation of AI-generated images. As such, while the comparisons of different LLMs using these metrics provide intriguing insights, it remains questionable whether these metrics are robust enough to serve as a truly holistic benchmark for evaluating semantic reasoning in AI-generated images.\n\n2. The number of images used (~2,000) feels arbitrary and may be insufficient to capture the nuanced aspects of reasoning and quality perception required for a comprehensive evaluation. Expanding the dataset to around 5,000–10,000 images, with careful attention to diversity across image types and contexts, could improve the robustness of the analysis. Additionally, it would be helpful for the authors to provide a rationale for this dataset size or acknowledge any limitations they faced in scaling up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "toOjLoU0gV", "reviewer": "Reviewer_vuEQ", "summary": "- Human evaluations are the gold standard for evaluating generative models especially text-to-image (T2I) models. However, they are expensive. An alternative is using automatic metrics and Large Multi-modal Models (LMMs) are a popular choice.\n- LMMs are trained on real images and AI-generated Images (AIGIs) are out of domain for LMMs questioning their reliability as evaluation models. \n- This work proposes A-Bench a diagnostic benchmark for assessing the reliability of LMMs for evaluating AIGIs. \n- A-Bench consists of two subsets 1) A-Bench P1 to evaluate the text faithfulness or prompt adherence of T2I models and 2) A-Bench P2 to evaluate the quality of the generations.\n- Authors samples 2864 AIGIs from 16 open and closed source T2I models. For each generation, they sourced human experts to annotate question-answer pairs and computed the accuracies of popular proprietary and open-source LMMs. \n- Authors report that 1) Proprietary LMMs are better than open-source counterparts for text faithfulness, 2) proprietary LMMs perform as well as humans on simple enough prompts and 3) LMMs are not good models to evaluate the generation quality of AIGIs.", "questions": "- I would recommend authors check out SelfEval [1] as another source of evidence that external models cannot be reliable for evaluating T2I models. Please discuss it if relevant.\n- In my experience there is a huge variance to the responses provided by  LLMs/LMMs. Did the authors compute variance of the scores or perform any statistical significance studies?\n- L272 controversial -> counter factual\n- In the introduction (L112-L117), in my opinion, authors should provide some numbers to make the point that LMMs are still not masters at evaluating AIGIs. Right now authors state that \"there remains a considerable gap and significant room for improvement\". Instead providing some numbers can make it more straightforward.\n\n[1] Sai Saketh Rambhatla, Ishan Misra, SelfEval: Leveraging the discriminative nature of generative models for evaluation", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Authors address a very important problem: Are current LLM/LMMs good enough to be used as judges for generative models? This line of research can provide valuable insights to train better LMMs for understanding AIGIs.\n- A-Bench along with standard LMM evaluation benchmarks provide a complete picture of an LMMs capability to understand both real and AI generated images.\n- The paper is well written and very easy to follow containing all the details necessary for reproduction.\n- The experimental section is exhaustive with comparisons provided for both proprietary and open-source LMMs.", "weaknesses": "- I didn't find any major weakness with this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCQQO4G89N", "reviewer": "Reviewer_AWPs", "summary": "This paper study the multimodal LLM's ability in the context of image evaluation. Instead of studying the effectiveness of certain LLM-based metrics, This work aims to identify whether multimodal LLM (LMM) are truly capable of evaluating AI-generated images through a question answering benchmark. Proposed a benchmark that contains 2,864 set of questions which can be categorized into 6 categories, involving semantic understanding and quality perception. After benchmarking a total of 18 LMMs and comprehensive analysis, the authors came up with a conclusion that LMMs are still not masters at evaluating AI-generated images.", "questions": "Q1) What is SRCC/PLCC in the introduction paragraph? It seems this abbreviation is never explained in the paper.\n\nQ2) In section 4.1, \"It’s worth noting that the instruction prompt might slightly differ for different LMMs according to the official setting.\" Why the instruction prompt is slight different for different LMMs? How will it impact the performance of LMMs?\n\nQ3) For the human annotators, how are they recruited? What kind of training were they given? How many instances are labelled by each human annotator? I am also interested in the total time required to build this benchmark.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1) The paper is well-written and well-organized. All the relevant details are included in the main paper and appendix. The evaluating method also seems rigorous.\n\nS2) A good pitch in studying LMMs directly through question answering instead of studying the effectiveness of certain LMM-based metrics. This helped to shed a light on the true capabilities of current LMM-based image evaluation metrics.", "weaknesses": "W1) The paper would have provided more insights if the authors also studied the reasoning to verify if the LMMs truly understand how to evaluate for each categories (i.e. did the reasoning fully explain the choice made by the LMM?). This might help to explain the gap between the performance of LMMs and humans. I suggest conducting a study on small subset for each categories and see how the reasoning was aligned to the choice made.\n\nW2)  It would also be desirable to see what kind of images LMM evaluate poorly across each categories in AIGI. A more detailed of diversity analysis on the AIGI dataset is required. E.g. for Basic Recognitions, how much portion of the questions are regarding recognitions of animal, human, or artifacts? Are these LMMs doing poorly on particularly certain type of objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)