Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
Authors: Haoning Wu
Zicheng Zhang
Erli Zhang
Chaofeng Chen
Liang Liao
Annan Wang
Chunyi Li
Wenxiu Sun
Qiong Yan
Guangtao Zhai
Weisi Lin
Venue: ICLR 2024 spotlight (2024)
Keywords: Benchmark
Vision-Language
Large Language Models
Low-level Vision
Image Quality Assessment
Primary area: datasets and benchmarks
TL;DR: We propose the first systematical benchmark for multi-modality LLMs (MLLMs) on low-level computer vision.

Abstract: The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs on **low-level visual perception and understanding**. To address this gap, we present **Q-Bench**, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment. **_a)_** To evaluate the low-level **_perception_** ability, we construct the **LLVisionQA** dataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions. **_b)_** To examine the **_description_** ability of MLLMs on low-level information, we propose the **LLDescribe** dataset consisting of long expert-labelled *golden* low-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and the *golden* descriptions. **_c)_** Besides these two tasks, we further measure their visual quality **_assessment_** ability to align with human opinion scores. Specifically, we design a softmax-based strategy that enables MLLMs to predict *quantifiable* quality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MLLMs possess preliminary low-level visual skills. However, these skills are still unstable and relatively imprecise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper to discover and enhance these untapped potentials of MLLMs.

Published as a conference paper at ICLR 2024
Q- Q-BENCH: A BENCHMARK FOR GENERAL-PURPOSS FOUNDATION MODELS ON LOW-LEVEL VISION
Haoning Wu' ZZcheng Zhang², Erli Zhang' Chaofeng Chen', Liang Liao', Annan Wang', Chunyi Li", Wenxiu Sun Qiong Yan², Guangtao Zhai? Weisi Lin S-Lab, Nanyang Technological University. Shanghai Jiaotong University, SSensttmme "Equal contribution. 'Corresponding author. Project Page: hutps://qqfuurre http:///.uuuurr.ggihhu.uo Caaiuuuiio//-Beeee Bench
ABSTRACT
The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs on low-level visual perception and understanding. To ad- dress this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs three realms: low-level visual percep- tion, low-level visual description, and overall visual quality assessment. a) To evaluate the low-level perception ability, we construct the L.Visiono dataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions. b) To examine the description ability of MLLMs on low-level information, we propose the LLDescribe dataset consist- ing of long expert-labelled golden low-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and the golden descriptions. c) Besides these two tasks, we further measure their visual quality assessment ability to align with human opinion scores. Specifically, we design a softmax-base based strategy that enables MLLMs to predict quantifiable quality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MILMS possess preliminary low-level visual skills. However, these skills are still unstable and relatively im- precise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper discover and enhance these untapped potentials of MLLMs.
INTRODUCTION
The emergent large language models (LLMMs) such as ChatGPT and Bard, as well as their excellent open-source counterparts (e.g., LLaMA (Touvron et al., 2023). MPT (Team] (023)). have served as powerful general-purpose assistants, which opens targeting specific tasks towards general intelligence. Following the advancements of LLMs, multi- modality large language models (MLLMs), represented by LLaVA (Liu et al. 2023b). MiniGPT-
era for artificial intelligence (AI) from
4 (Zhu et al.) 2023). InstructBLIP Daa et al. 2023). and Otter (Li et al., 2023a). have brought exciting progresses on the vision field as well. They are capable of providing robust general-level and can even seamlessly dialog and interact with hu- mans through natural language. While such abilities of MLLMs have been explored and validated on several Languge tasks such as image captioning (Chen et al. 2015), visual question an- swering (Attll et al.] 2015), cross-modality grounding (Peng et al.) 2023), and traditional vision tasks such as image classification or segmentation (Lai et al. (023)) most attention is paid to the high-level perception and understanding of visual contents. Meanwhile, the ability of MLLMs re- mains not clear low-level visual perception and understanding, which play significant roles in image quality assessment (IQA) (Hosu et al. 2020; Fang et al.) 2020) and its associated tasks on perceiving visual distortions (noises. blurs) (Su et al. [022]] Wu et al. 2023d) and other low-level attributes (color, lighting. composition, style, etc) (Kong et al.) 2016) that may relate to aesthetics and emotions of natural photos (Murray al. 2012 and human preferences on emerging computer-
abilities on visual perceptionn/undestaandii

---

Published as a conference paper at ICLR 2024
Questions:
Answers: No. 2. High.
1. is this image clea??
2. How the brightness?
Perception
This picture is underrxxpoaed with people in the middle completely black and the fish the background b urred. Therefore. this is a very low-quality image.
Please describe the quality the image decails.
(b) Description
Emerging Language Abilities
Low-level
Construct
Visual Abilities
Retrieve
Fvaauatee
Q-Bench (propoead)
Human User
MLLMs
Human Assistant
The quality of the image is SOCORETOKENT
Rate the quality of the image.
Ability on Traditional Tasks
Assessment
Softrax between [cood] and (poor)
81.2
Figure 1: In the proposed Q-Bench, we build the first benchmark on emerging abilities of MLLMS on low-level vision, including perception of low-level attributes (by correctly answering diverse queries) and description of low-level quality-related information via natural language. Furthermore, the Q-bench also evaluates the quantitative assessment ability of MLLMs on traditional IQA tasks.
graphics generated (Zhang et al. 2023b) or AI Al-gererteed images (Li et al. 2023c; Xu et al. 2023).
These low-level visual abilities are strongly associated with wide range of applications, such as recommendation (Wu et al., (023c), guidance on camera systems (Zhang et al., (022), or visual quality cnhancement (Zhang et al. 2018). Henceforth, it is crucial to evaluate the current abilitics of these ceneral-purpose foundation models in low-level visual perception and understanding, to ideally relieve extensive human resources to give feedback on every specific low-level task.
In our work, we propose the first systematic benchmark to measure the low-level visual perception and understanding abilities of MLLMs. Our benchmark is constructed around a key question:
How do MLLMs emulate human ability related to low-level visual perception and understanding?
simple answer is language, which is the fundamental property of MLLMs. Specifically, we define two emerging language abilities of MLLMs on low-level vision follows:
Ability (AI): Perception of Low-level Attributes. As shown in Fig. [a), like a human, MLLM should be able respond accurately simple questions related to low-level attributes, e.g answering 'No' for a blurry image when queried with 'Is this image clear?'
Ability 2 (A2): Description via Natural Language. As shown in Fig. [[b). like human, an MLLM should be able to describe the quality and other low-level information for an image with natural language. The descriptions should be both complete and accurate.
To systematically evaluate the low-level perception ability (A1) on various low-level attributes un- der diverse circumstances, we construct the LLVisionQA dataset, including 2,990 images from 10 diverse sources. Aligned with existing practices (Liu et al. 2023c: Lu et al.) 2023), each image in LLVisionQA is equipped with a question, alongside a correct answer and false candidate answers. In LLVisionQA, design three diverse types of questions: Yes-or-No questions. What questions, and How questions. Moreover, we divide low-level concerns into four quadrants, via two axes: (1) dis- tortions (blur, noises, etc) vs other low-level attributes (color, lighting, composition, etc) (Guha et al.
2

---

Published as a conference paper at ICLR 2024
2020). (2) global perception (e.g., sharpness of the whole picture) local content-related in-context perception (e.g., whether the red flower is in focus) (Li et al. 2019). With three types of questions and four quadrants of concerns. the proposed LLVision dataset provides holistic, diverse, and balanced benchmark for the perception ability on low-level visual attributes of MI.LMs.
For the description ability (A2), given that the output description is expected to be complex (without fixed formats), we propose the LLDescribe dataset by inviting experts to write long golden low-level descriptions (average 58 words per description) for 499 images, which serve as the reference texts for the single-modal GPT to evaluate MLLM output descriptions. The quality of MLLM descrip- tions evaluated through three dimensions: completeness (punish missing information), preciseness (punish outputs controversial with reference), as well as relevance (pumish outputs irrelevant to low- level attributes). With golden descriptions and the multi-dimensional evaluation process participated by GPT, we comprehensively evaluate the low-level description ability of MLLMs.
Besides the two emerging language abilities, we also evaluate MILMs on the traditional IQA task, a more abstract task that requires understanding human opinions of low-level attributes, as follows:
* Ability 3 (A3): Precise Assessment Aligned with Human Opinions. As depicted in Fig. [c] (c), an MILLM should be able to predict quantifiable quality scores for images, which can be aligned with the human-rated mean opinion scores (MOS) low-level visual appearances.
For the assessment ability (A3), we utilize plenty of existing IQA databases (Hosu et al. 2020 Lin al., 2019 Li et al. 2023c) that focus on various low-level appearances of images, to benchmark MLLMs within conventional IQA settings. Specifically, we notice that MILMs encounter difficul- ties in providing sufficiently quantifiable outputs, whether instructed to directly rate with texts or provide numerical outputs. To solve this challenge, propose to extract the softmax pooling result on the logits of the two most frequent tokens (good and poor) under the response template of MLLMs (Fig [e)) as their quality predictions. Our studies prove that the proposed roftmax-based strategy is generally better correlated with human perception than direct token outputs of MLLMs (via aggax), which bridges between these emergent MLLMs and the traditional IQA task settings. Under this strategy, we evaluate all MLLMs on their precise assessment ability by measuring the correlations between their predictions and human opinion scores various IQA databases.
In summary, we systematically explore the potential of MLLMs on three low-level visual abilities: perception, description, and assessment. The three realms compose into the proposed Q-Bench, a MLLM benchmark on low-level visual tasks. Our contributions can be summarized as three-fold:
We build benchmark for MLLMs on low-level perception ability. To achieve this, we construct first-of-its-kind balanced and comprehensive LLVisionQA dataset with 2,990 images with one low-level-related question-an answer pair for each image. The LLVisionQA includes three question types and four quadrants of low-level concerns to ensure diversity.
We define a benchmark process to evaluate the low-level description ability of MLLMs, including L.Description dataset of 499 images with lxpert-labelled long golden quality descriptions, and a GPT-assisted evaluation to rate diss.iddesissssons in terms of com- pleteness, preciseness, and relevance compared with golden descriptions.
To evaluate precise quality assessment ability, we propose a unified softmax-based quality prediction strategy for all MI.LMs based on their probability outputs. With its effective- ness validated in our experiments, the proposed strategy sets up a bridge between general- purpose MLLMs and traditional IQA tasks that requires quantifiable scores as outputs.
CONSTRUCTING THE Q-BENCH
2.1 GENERAL PRINCIPLES
Focusing on Low-level Visual Abilities of MLLMs. Unlike existing MLLM benchmarks (Li et al. 2023b Liu et al. 2023c Lu et al. (023) that aim at all-round abilities, the tasks in Q-Bench are constrained with two basic principles: (1) Requiring perception and/or understanding on low- level attributes of images; (2) Not requiring reasoning (i.e. why) outside knowledge (Marino et al.) 2019). We adhere to the principles in designing the perception, description, and assessment tasks, making the proposed Q-hench focused reflection the low-level visual abilities of MS..Ms.
3

---

Published as a conference paper at ICLR 2024
Table 1: Overview of the 10 diverse image source datasets in the Q-Bench, and the respective benchmark dataset size for each low-level ability among perception. descrption and assessment. The Corrupted COCO denotes COCO-Captions images corrupted by Michaelis et al. (2019).
Sampled Size
Sumpled Size in LLDeseribe
Full Dataset Size
Type
Image Source Dataset
in LLVisionQA
for Assessment Task
KONIO-1OK (Hosu et al.
2020
600
100
10,073
In-the-wild
SPAQ FFng et all 2020
800
130
11,125
LIVE-FB Ying al 2020
300
50
39,810
LIVE-itw Ghadiyaram & Bovik
2016
300
50
1,169
CGIQA-6K (haang et all 20241
200
30
6,000
Generated
AGIQA-3K Li al 80250
198
30
2,982
ImageRewandDH eX et 2023
194
29
not included in (A3)
KADID-TOK (Lin et all k019)
81
20
10,
Artificially-
LIVEMuliiDisottion (Jayaamann et al. 2012
15
not incladed in (A3)
distorted
Corrupted COCO (Chen et al., 2015)
302
50
not incladed in (A3)
Corresoondngg Ahility/ lask in Q-Bench
(A1) Perception
(A2) Description
(A3) Assessment
Total Benehmark Size for Respoctive Task
2,990
499
81,284
Covering Diverse Low-level Appearances. To cover diverse low-level appearances, we collect multi-sourced images for each task, as depicted in Tab. | Among all images in the perception and description tasks, two-thirds are tn-th images directly collected from social media posts, smartphones or professional photography. The rest one-third images are collected after various artificial distortions, or via generative processes (CGI, AIGC). Furthermore, we employ k-means clustering for the low-level attribute indicators to certify that the sub-sampled images retain high diversity. In the assessment task, full images of 7 IQA datasets within all three source types are evaluated through traditional IQA metries. The diverse and multiple sources of images morph the Q-bench into a holistic and balanced benchmark to fairly evaluate low-level-related abilities.
2.2 BENCHMARK ON LOW-LEVEL PERCEPTION ABILITY
In the first task of Q-Bench, we evaluate the low-level perception ability of MLLMs to examine whether they can answer simple natural queries related to low-level attributes. For this purpose, first collect 2,990 images (I) from multiple sources (see Table [I with diverse low-level concerns. Then, we collect one low-level-related question (2), one correct answer the question (C), and 1-3 candidate false answers (F) for each image. The 2,990 (I, Q, C, F) tuples compose into the LLVi- sionQA dataset (as illustrated in Fig. (2). the first visual question answering (VQA) dataset in the low-level computer vision field. Specifically, the questions in LLVisionQA cover four quadrants of distinct low-level concerns (in Sec.2.2.1 and three question types (in Sec. 2.2.2.. After constructing the dataset, the (I, Q, C, F) are together fed into MLLMs for evaluation, while their outputs are further examined by GPT to judge correctness (in Sec. 2.2.3 The details are elaborated as follows.
2.2.1 QUADRANTS FOR LOW-LEVEL VISUAL CONCERNS
Axis 1: Distortions vs Other Low-level Attributes. The primary axis differentiates two categories of low-level perceptual attributes: 1) technical distortions (Su al. 2021)) seen as the low-level characteristics that directly degrade the quality of images (Ying et al. 2020). and 2) aesthetic-related other low-level attributes (Kong et al. 2016 Hou al.) 2023) which are discernible perception and evoke varied emotions. Several studies (Talebi & Milanfar 2018 Ying et al. 2020 Guha et al. 2020 follow this paradigm and categorize them through a relative golden standard, that whether the attributes directly improve or degrade picture quality (Yes Vestttsstos;ons;; No-+Others). Despite this standard, we also enumerate common types of distortions vs other low-level attributes as extra guidance for constructing the V.Viiion dataset, as listed in Sec. A.1.2
human
Axis 2: Global Perception vs Local In-context Perception. In recent research on low-level vi-
sion, it is observed that human perceptions of low-level visuals often intertwine with higher-level contextual comprehension LLi et al., 2019 Wang et al., 2021 Wu et al.) 2023a). For instance, a clear sky might lack complex textures yet display exceptional clarity. Furthermore, localized low- level appearances can deviate from their overall counterparts, as observed by Wu et al. (2022): Ying et al. (2021). Acknowledging these differences, we curate local in-context perception (Fig. right) questions, that require MLLMs to grasp the content or other context to answer correctly, while other questions categorized as global perception (Fig. 2 left). (More analysis in Sec. A.1.2
4

---

Published as a conference paper at ICLR 2024
Global Perception (1885)
Locaa In-context Perception (1105)
Distontions (1 (34)
Other Attributes (851)
In-context Distortions (195) plea FJ" in-contest Other Autributes (209)
Yes-or-oo 11000111000 (1100)
Queetion (O)
Doos 1-s dicu/
Image 0f:
Question
Are the sEata in
pkcture seer?
Question (G):
It:
Doct this
Ile
malizz?
Correct
Correct Answer (C)- You
Correct Answer (C):
Cornect Answer [CH
No
False Answers [Nl
Faloe Answers (FI:
hho)
Palax Answers (F): [[we]
Image W::
Duestion I0g
Question
Image W:
Question IS]:
What the wortl
Wi platigpeplt techtigue non used in this mage??
Whst
image escedally
What/8111 (811
dedtrrtinn thi trsepe?
Correct Anwwer (CP
Gornect Answer Masic blr
Conrect (C)
image less Vidoow
Image 0::
False Answers IFr Nosa, Ovansepoiun undereaposumt
Palss Answers (FI:
False Anewars [F):
Baorg contrati Ractggrand
False Answers IF) Midkeuqpoare, Ekr
Tooolihh.
How (979))
Image 01:
Question (Q):
How in the theicvedd
Darity of the image?
Question (Q): How kyring this Image?
How the staeprasoc Iew marle bece?
Question (Q):
Image W
the pat in this mage?
How in the lightngg of
Question (Q):
Correct Answer (CI:
Correct Answer (C):
Correct Arrower (CI:
Correct Answer (C)
False Answers (F): Loud
Falso Amewwers (F):
False Amewers (F): |Mddum,
[ioo Too
Far Good]
Figure 2: A dataset card of LLVisionQA that evaluates the low-level perception ability of MLLMs. 2,990 (I, Q, C, F) tuples are collected to cover three question types and four quadrants of low-level visual concerns, providing all-around evaluation of low-level visual perception for MLLMs.
2.2.2 QUESTION TYPES
In the LLVisionQA dataset, we curate three question types, Yes-or-No, What, and How to simulate multiple query forms from humans. The details of the three question types are defined as follows.
Type 1: Yes-or-No Questions. The fundamental type of questions is Yes-or-No, i.e,, judgments. Specifically, we notice that some MLLMs especially prefer to respond with yes rather than no. To reduce such biases in our benchmark, though designing questions with answers yes is easier, we ensure that around 40% of all judgments are with correct answers as no, via querying on contrastive low-level attributes non-existing lowelevel attributes. We further measure the bias levels of dif- ferent MILMs and present a further de- biased evaluation among them, as discussed in Sec. A.3.2
Type 2: What Questions. Despite Yes-or-No judgments, the what questions are also common type of queries in recent MILMM benchmarks such as Lu et al. (2023). In Q-bench, they classify low-level attributes in pictures (e.g., What distortion occurs in the image?), ?), or associated context given specific low-level appearances (for in-context perception questions, egg., Which object in the image is under- exposed?). Unlike Yes-or-No questions, the What questions examine more comprehensive low-level attribute understanding of MLLMs, by requiring correct perception on multiple attributes.
Type How Questions. Despite the two common types, we also include a special type, the How questions, to cover non-extreme appearances (Wu et al.) (023d) of low-level attribute dimensions into our benchmark, as an extension to Yes-or-No questions. As shown in Fig. 2 we can query How is the clarity of the image? for the image with both clear and blurry areas, and answer with Medium. With this special question type, we broaden the Q-bench into finer-grained low-level perception.
2.2.3 GPT-ASSISTED EVALUATION PROCESS
After constructing the LLVision dataset, we feed it to multiple MLLMs to evaluate their abilities on low-level visual perception. The input format to query MLLMs is exemplified as follows:
#User: How is the clarity of the image? (Question) on) /IMAG_JTOKEN (Image)
Choose between one of the following options: A. High (Correct) Medium (Wrong) C. Low (Wrong)
The correct and wrong answers are shuffled during the actual evaluation. Moreover, while traditional visual question answering (Antll et al. 2015 Marino et al. 2019) tasks typically employ traditional language metrics (BLEU-4, CIDEr) to compare performance, as observed by recent studies (Ye et al. 2023) and validated by us, most MLLMs cannot consistently provide outputs formats. Given the question above, different MLLMs may reply "A.". "High", ", "The clarity of the image is high.", "The image is of high clarity." (all correct), which are difficult to be exhaustively-
instructed
5

---

Published as a conference paper at ICLR 2024
Aurincaal slettottd
In-ihe-wll
uurrspoees on bu hurtet und bolk not Iu solul, lhe sas comen and prear deal out Erunciis in the hadici ai maing a umeoogrosen The renaal only of mmggei good.
of Bariow's hox pnourdat enypanyy undltr Amtunno ain good licwoor the ooraet auo shrah tos fedin Put the woord quality of propgee uny
pnour aud moll und terkkaat and heu to use alo phow fol. not surmare by mos noveret of wolte cosch the move Theetto, Peope iy of mago peecily por
Wn-the-wkkd
Inusttkk labar undendiaa cnunlll
Fight and und can ceadbe Macndd Mowero ton andima dersis workinly Thanttre,
Herreeeee Bruphddo mnby out (kilul
siund.. Boal arroodgeeo ddlats
achal anond faams Poreoo up bur here cosospiiii boutbaggguuun sullic rasommbe 24 with abeer coRTy courtior.
losua offee suject Thon cwertd is in the oe comar Therthi,
Figure 3: A dataset card of LLDescribe that evaluates the low-level description ability of MILMs. 499 images from 10 diverse sources are labeled with golden descriptions, to serve as text references for single-modal GPT to evaluate the completeness, preciseness, and relevance of MLLM outputs.
included under traditional metrics. To solve this problem, we design, validate, and employ a 5-round GPT-assisted evaluation process inspired by Liu et al. (2023c). Under this process, the question, correct answers, and MILLM replies are fed into GPT for evaluation (See Sec. A.2. for its details).
2.3 BENCHMARK ON LOW-LEVEL DESCRIPTION ABILITY
In the second task of Q-Bench, we evaluate the language description ability of MILMS on low-level information. This task is sibling task of image captioning (Chen et al.) 2015 Young et al. 2014] Agrawal al. 2019) that describes image content with natural language, with a specific concern on the low-level appearance of images. To evaluate this ability automatically, we first derive a golden low-level description dataset, denoted as LLDescribe (Sec. 2.3.Ip. including one long (average 40 words) golden description provided by experts for each of 499 images. With these golden text descriptions, we are able to measure the quality of output low-level descriptions from MLLMs with single-modal GPT, under the three dimensions: completeness, preciseness, as well as relevance The discussions of the golden descriptions and the evaluation process are as follows.
(Sec 2.3.2
2.3.1
DEFINING Golden LOW-LEVEL DESCRIPTIONS FOR IMAGES
For the description ability, MLLMs should accurately and completely describe low-level visual in- formation of images. Thus, the ground truths for these MI.MMs are also built within basic principle to cover as many low-level concerns as possible, so long as they are enumerated in Sec. 2.2.1 and occur in images. The resulting golden descriptions in LLDescribe have an average duration of 58 words, notably longer than common high-level image caption datasets (11 for Agrawal et al. (2019). 10 for Chen et al. (2015)). Similar to the LLVisionQA dataset for the perception task, the 499 im- ages in L.Describe dataset also include all 10 sources (as in Tab. IT to cover images with diverse low-level appearances. The golden descriptions different sources of images are depicted in Fig.
2.3.2 EVALUATION WITH SINGLE-MODAL GPT
Recent studies (Zheng et al. 2023) have proved single-modal GPT (OpenAl 2023) to be reliable evaluation tool for pure language tasks. Via the LLDescribe dataset, we convert the multi-modality problem into text-only setting, by matching the MILM outputs with the golden descriptions with single-modal GPT under three dimensions: (1) Completeness. More matched information with the golden description is encouraged. (2) Preciseness. The controversial information with the golden description is punished. (3) Relevance. More proportions MLLM outputs should be related to low-level information, instead of others. Each dimension is scored among [0.1,2]. Similar as Sec. 2.2.3] we repeat rounds for each single evaluation and collect the weighted average as the final score. The detailed settings for GPT to evaluate the three dimensions Sec. A.2.2

---

Published as a conference paper at ICLR 2024
Votual tamplases offerrnt grity
Raw Logits Tokens [SCORE_ TOKEN
Softmux
"(SCORE, TOKEN
0.2276
0.1903
O. 0.1875
0.1 130
0. 1.1481
0,1392
MLLM-Pradicted Score
81.2
good
poor
excellent
high
great
bad
maago
[
quality
Imagc
Large Language Models (LLaMA, LLaMA2, Flan-T5, T5, MPT)
User
Figure 4: The proposed softmax- based quality assessment strategy for MILMMs Instead directly decoding tokens from the ISCORE_TOKEN] position, the strategy extracts log probabilities (logits) of good and poor, and predicts quantifiable score via a softmax pooling between the two logits.
2.4 BENCHMARK ON PRECISE QUALITY ASSESSMENT ABILITY
In the third task, benchmark the ability of MLLMs provide quantitariiv assessment the overall low-level appearance of images. Unlike the two tasks above, we utilize existing IQA datasets that collected across variety of low-level appearances to evaluate how MLLMS can predict quantitative quality scores aligned with human opinions. All the three types of IQA datasets (in- the-wild, generated, creficialll-distorteed as mentioned in Bec.2.IJare evaluated, provide broad range measurement of the assessment ability of MLLMs. Nevertheless, how to collect quantifiable quality scores from MILMs remains challenging as their outputs only have weak measurability (Sec. 1.4.1. Noticing that MLLMs can provide probabilities of tokens, we employ softmax pooling on the logits good and poor under a simple and direct prompt template, deriving into quantifiable quality score predicted by MLLMs (Sec. 2.4.2). as illustrated in Fig. 4 Details are follows.
2.4.1 WEAK ARASURABILLTYY OF MILM OUTPUTS
In Q-Bench, we aim to fairly compare the assessment ability between different MLLMs on di- verse low-level appearances. Henceforth, our principle is to define unified, simplest instruction that is applicable for all MLLMs on all IQA datasets. Under this principle, we conduct toy ex- periments on LLVisionQA on Shikra and LLaVA-vI, with two simple instruction strategies: (A) Direct Instruction, in which the prompt is designed as simple as "Rate the quality of the image". The top-free top-frequency answers are good (78%), and poor (20%), with other outputs almost negligible. (B) Numerical Instruction, in which we specifically instruct numerical ratings, with the prompt: "Score the quality of the image from 1 5, with as lowest and 5 as highesz." ". Under the numerical strategy, the frefrreuency answers are 5 (84%), 1 (9%), and 3 (5%); though within the score range, the frequencies of scores 2 and 4 are both less than 1%. The toy experiments imply the weak mea- statistically 1) biased towards positive, 2)
surability of MLLM outputs, given that the answers biased towards extreme, and 3) with only two effective seales. Therefore, it is necessary to explore extended strategies for MLLMs to provide truly quantifiable outputs for low-level assessment.
2.4.2 A SOFTMAX-BASED EVALUATION STRATEGY
Given the above observations, we design the softmax- based evaluation strategy (Fig. to reduce the negative impacts of the biases and lack of scales. To start with, we design our strategy within the Direct Instruction, which is more general and less biased than the Numerical Instruction. The strategy is based on the observation that two top-frequency outputs, good and poor, can be considered anchors for better and worse human perception, and the Direct Strategy can be ap- proximated into a binary classification problem on the /SCORE TOKENJ position, or technically, an a argmax between the logits of good (CSCORE TOKEV) and poor (xSSOOR TOKeN) on this position. In our revised strategy, we modify the a argmax into softmax collect better quantifiable scores:
SCORF
Spred
200d
This simple and generally cenerally-apllicable applicable strategy enables us to collect quantifiable outputs (sprea) from MLLMMs with higher correlation to human ratings, as verified in experimental analysis (Tab.

---

Published as a conference paper at ICLR 2024
Table Results on the test subset for the low-Ievel Perception ability of MLLMs. MLLMS with top-3 performance in each sub-category and the overall LLVisionQA is emphasized with boldface.
Sub-categories
Question Types
Quadrants Low-level Concerns
Model (variant)
Ves-or-Not
Whart
Howt
Distortion
Othert
Iotteee Distoritont
In-conteer Othert
Overall
random guezz
50.00%
28.48%
33.30%
37.24%
38.500
39.13%
37.10%
37.94%
3 1.cumi--1...775
64.60%
59.222%
53.766 76%%
44798%%
67. 30%
58. .0.90%
73,7695
60.0%%
LLaVA-v1.5 (Vicmn-vi1.-1338
64.96%
64.86%
54.12%
53.55%
66.59%
58.90%
71.48%
61.40%
Deemm_M-XComposer-- (foteroLM)
68.43%
62.04%
61.93%
56.81%
70.41%
57.53%
77.19%
64.35%
IDEFICS Instruct (MA78))
60.04%
46.42%
46.71%
40.38
59.90%
47.26%
64.77%
51.51%
Owen VL (QweeLMM
65.33%
60.74%
58.44%
54.13%
66.35%
56.22%
73.00%
61.67%
Shikma (ic
69.09%
47.93%
47.31%
60.86%
53.08%
64.77%
35%
Otiervil (MPT-7B)
57.66%
39.70%
42.59%
42.12%
48.93%
47.60%
54.17%
47.22%
InstrecBLIP (Flon-T5-XL)
69.53%
59.00%
56. 17%
57.31%
65 65.63%
56.51%
71.21%
61, 94%
InstractBI (Vicuna-78;
70.99%
51.41%
45.00%
63.01%
57.19%
64.39%
85%
Visua (GLM-68)
61.31%
53.58%
44. 44.03%
48.56%
$4,89%
55.48%
57.79%
33.31%
mPL (7.)
72.45%
54.88%
47.535
49.624%
63.01%
62.67%
66.67%
58, 933%
LLaMA-Adapten-V
66.61%
54.66%
51.65%
56.15%
59.25%
54.55%
0666
LLaVA-v1 (Vacuna- (3B)
57.12%
54.88%
51. 85%
45. 58%
58.00%
59.19%
64.77%
54, 72%
MiniGPT-4 (Vicuna-133 13B1
60.77%
50.33%
43.00%
45.58%
52.51
53.42%
60,981
51.77%
GPT-4V (Close-Soure) Model)
77.92%
79.18%
62. 68%
70.58%
73.03%
74.66%
77.95%
73.36%
Junior-eewe Human
82.48%
79.39%
60.29%
75.62%
72.08%
76.37%
73.0099
74.31
Senior-level Human
84.31%
88.94%
72.02%
79.65%
79.47%
83.90%
87.07%
81.74%
3 RESULTS ON Q-BENCH
In Q-Bench, we evaluate 15 variants on 13 up-to-date popular and competitive open-souree MLLMs, together with GPT-4V, under zero-shot settings. More results and analyses are appended in Sec. A.3
3.1 RESULTS AND ORSERVATIONS ON PERCEPTION
Open-Source MILMMs For a holistic examination on the perception ability of MLLMs, we eval- uate the multi-choice correctness of MILMs on different sub-categories of the LLVision dataset, which equally divided as dev (Tab. [ will be released) and test (Tab. 2 will keep private) subsets. We are glad that the majority of MLLMs can significantly outperform random guess all sub-categories. Considering that all participating MLLMs are without any explicit training on low- level visual attributes, these results show strong potentials for these general-purpos. models when further fine-tuned with respective low-level datasets. Among all MLLMs, the recently-released Inteml.--CComooser
reaches the best accuracy this question-answering task, followed by LLaVA-v1.5. QW and InstructBLIP (Flan-T5)) which show rather close results. By achieving more than 60% % accuracy on both subsets, these models show exciting potentials as robust low-level visual assistants in the future. Another key observation is that almost all methods perceive worse on distortions than other low-level attributes. One exception is LLaMA-Adapter- V2, which is the only MLLM that adopts multi-scale features as visual inputs. We also notice that all MLLMs prefer yes than no among Yes-or-No questions, as analyzed in Tab. X qualitative comparisons are illustrated in Fig. 10 For Kosmos-2, we specially adopt close-set inference for it, as discussed in Sec. A.2.
GPT-4V vs Human. To evaluate the low-level perception abilities of the commercial MLLM, GPT-4V, we gauge its accuracy against human using the test subset of LLVision dataset. GPT-4V exhibits competitive performance and outperforms open-source MLLMs by a large margin (+9%), ), and on par accuracy with the Junior-level Human. Despite its prowess, there is still way to go for GPT-4V before it can match the overall proficiency of the Senior-level Human (with experiences on low-level visual tasks, 8% better than GPT-4V). Furthermore, across all categories, the results show that GPT-4V, much like its open-source counterparts, faces challenges in recognizing distortions.
3.2 RESULTS AND OBSERVATIONS ON DESCRIPTION
For the description ability, DeemmL--XCompossee VL reaches best proficiency again, especially in terms of the relevance dimension. Nevertheless, in the perspective of the completeness and precision of the descriptions, even the best of all MLLMs cannot obtain an excellent score; on the contrary, almost all MILMs reach acceptable standard (0.8/2.0). In general, all MLLMs at present are only with relatively limited and primary ability to provide low-level visual descriptions. We also conduct qualitative comparison for MLLM descriptions in Sec. A.3.3
8

---

Published as a conference paper at ICLR 2024
Table 3: Results on the low-level Description ability of MLLMs. P. denotes frequency for score i.
Dimensions
Completeness
Precision
Relevance
Sinnc T
Mudel (varian))
7%
acare
P3
score?
7%
Score]
LL (1.3-7-
27.48%
54.74%
17.78%
0.500
30.511
26.04%
43.45%
1.13
10.855%
60.34%
25.815
1.15
3.21
LLaVA-v1.5 v1. (Vicama v1.5 1330))
27.68%
53.76%
18.55%
0.91
25.45%
21.47%
53.0815
6.31%
58.75%
34,9415
3.47
IntemLMM-X InemLL.-XCooppeee XComppsze VL (M)tenLM))
19.54%
51.82%
28.24%
1.06
22.59%
1999999
48.421%
1.26
1.015%
10.62%
H8.5258
1.87
4.21
IDERCCS Instrctt (LLaMd 79)
28.91%
19.1
11.93%
34.68%
27.36%
37.4615
39.66%
36.4455
3.15
Qwen-VE (QQeenLM)
20.34%
49.13%%
24.53%
0.58
50.6216
25.44%
25.9458
0.77
0.7255
35. 50.50%
63.72%
1.63
3.35
Shiku (Vicnn
21.14%
68.33%
10.52%
30.33%%
30%
41.371
61.36%
34.50%
3.34
Oner-v1 (MPT--78)
22.3855
59,301%
18.25%
0.56
40.6655
9999
23.333%
0185
1.95%
13.20%
54.251
1.85
3.61
Kosmos
20.33%
29.45%
34.75%
35.811%
14.77%
85.06%
4,03
Ins IP.I T5-XX)
23.166
66.445 4494
10.40%
0.57
34.85%
26 0.0%
39 121
1.04
14.71%
59.87%
25.425
1,11
3.02
InstuetBLLPP /Viaaa
29.73%
61.47%
8.80%
0.79
27.84%
23.52%
1.21
27.40%
11.215%
2,84
VouulGLM-SBB (GLM-6B)
30.75%
65.64%
12.61%
0.52
38.641%
26. 18%
35.18%
0.97
6.14%
67.15%
26.71%
1.21
2.99
mPLUG Owl (LLaMA
28.28%
37.69%%
34.03%%
26.75%
18.19%
55.07%
33.82%
63.15%
3.94
L.MA-AAdppee---
30.4655
33.99%
15.57%
0.85
25. 29.41%
44.80%
1.15
1.50%
52.75%
45.75%
1.44
3.45
LLaVA v1 (Niunn
34.00%
40.52%
25.39%
0.91
30.02%
15.15%
54.83%
38.02%
60.915
3.76
MiniGPT-4 (Nicmmo-1380 138)
34.01%
52,
33 85%
1.00
39.20%
15.27%
55.53%
1.26
6.88%
45.65%
47.48%
1.41
3,67
Table 4: Main evaluation results on the zero-shot Assessment ability of MLLMs, in comparison with NIQE and CLIP- CLIP-ViT-Large-14, 14, the visual backbone of most MLLMs. Metrics are SRCC/PLCC.
Datasel Type
In-thx-wild
Generated
Artificial
Aueragee
Model/ / Dataset
KONIO-JOR
SPAQ
LIVE-FB
LIVE-iNw
CGROA-6K
AGIOA-3K
KADID 10k
NIQE Mittal at 2015
0.31600377
0.69330.602
0.211A0:258
0.40000451
0/075001866
0.56281517
0.37480.228
0.38700 298
CLIP.VIT ango-
D. 0468805505
0.38550 389
0.21600237
0,50770 308
2.05000.00
0.4360 458
0.376/0 388
0. 0.354/0. 368
[LaVAvv1.5 (H (H.2max).3.78
6.44530.467
0.36BA0.52
0.545.6.88
022100333
0.67341.33
041100240
0:454
LIaVA-v1.5
(Voune-v).5-133
D. 0.44801.400
0.56330-55 584
0,5100.339
0.45004881
0.285/0.27
0.6641 754
0.39000.400 400
0.444/0.477 474
MaennlMMXCooppseet VL. (nterndM
0,56480.615
0.730/9.750
0,36000416
0.61290.76
0.243/0226
0.73200775
0.5469.572
0.541/0.581
IDEFICS- (LLaMA--7R)
0:37581.401
0,47480 484
0,235/1240
0.40900-478
0,24/002277
0.56281622
0.37000373
0.381/0. 796
Qwen-VL Owen-YL-(QwenLMM
0:47000.46
0.676/0 669
0.29800338
0.50400.532
0.273/0.284
0.617/0.686
0.486004866
0.47500.006
Shikra (Veume-78)
0.314003077
0,328N0 357
0,257/1.241
0,322/0
0,19/00200 201
066401666 661
0.32481.332
0.336/0 345
Olierv1 (MPT-7B)
0:40600406
0.436/0.441
0.143/0.142
00008/00008
0.25-00.261
0.475004881
0.557/0.577
0.323/0 333
2 2
0.255/02281
0,644/0.641
0,1960 195
0,358/0.368
0.210/0.225
0.489/0.491
0.359/0.365
0.359/0.367
InstrccBLIP TFXL))
0,334803362
0.582/0 599
0.248/02677
0.113/0.113
0.167/0.188
0
0.211/0.179
0,29000.30 301
InstruciBLIP /Vicana 7B)
0.359/0.437
0.68340 689
0,200/02283
0,253/0.367
0.263/0.304
0.629/0.663
0.337793322
0.389/0.466
ViualGLM-6B (GLM-68) 68)
0,24780.234
0,496/0 507
0.146/0114
0.1100.116
0.209001185
0.34200.349
0.127/0.131
0,240/0.22 239
mPLUG-Owl (LLaMAA 7B)
0.409/0.427
0,63410.644
0,241/0.271
0,437/2.477
0,148/0,180
0.68700711
0.16699.866
0.432/0.458
LLaMA-Adapter-VV
0.35490363
0.46440.500 506
0.275/03329
0.298/0.360
0.257/0.271
0.604/0.666
0.412K0.425
0.38119.417
LLaVA-vi (Vicuna-130)
0.462/0.457
0.442/0.462
0,266/02200
0.404/0.417
0.208/0.237
0.626/06684
0.349/0.372
0.394/0.416
MiniGPT-4 -338
0,239002257
0.238.0223
0.17000183
0.330/03300
0.252/0.246
0.57241.591
0.239/0223
0.293/03300
3.3 RESULTS AND OBSERVATIONS ON ASSESSMENT
To measure the assessment ability, we evaluate the performance of 15 ML.MM on 7 IQA datasets that with at least 1,000 images and 15 human ratings per image (itu 2000). Primarily, we notice that the majority of MLLMs are notably better than NIQE on non-natural circumstances (CGI, AIGC, artificial distortions), showing their potential towards general-purposs evaluators on broader range of low-level appearances. We also notice that without explicit alignment with human opinions during training, the most excellent MILM, which is again IntemLM--Xooppnn XComposer- VL, can already outperform CLTP--VT-Large- 14 by large margin (20%), marking the dawn of MILMs as robust quality evaluators. Furthermore, we also design a synonym ensemble (see Sec. A.2.3) strategy which can further generally improve IQA accuracy of MILMs, whose results are analyzed in Sec. A.3.5 Despite their proficiency, current MLLMS are still less accurate in finer- mrar-grained situations (LIVE-FB, CGIQA-6K) for the assessment task, which could be enhanced in the future.
4 CONCLUSION
In this study, we construct the Q-Bench, a benchmark to examine the progresses of MLLMS on low- level visual abilities. Anticipating these large foundation models to be general-purpos intelligence that can ultimately relieve human efforts, we propose that MLLMs should achieve three important and distinct abilities: accurate perception on low-level visual attributes, precise and complete lan- guage description on low-level visual information, as well as quantitative assessment on image quality. To evaluate the abilities, we collect two multi-modality benchmark datasets for low-level vision, and propose unified softmax-based quantitative IQA strategy on MLLMs. Our evaluation proves that even without any low-Level-specifi training, several extraordinary MLLMs still have de- cent lowelevel abilities. Nevertheless, there is still long way to go for MLLMs to be truly-reliable general low-level visual assistants. We sincerely hope that the observations found in the Q-Bench can inspire future MLLMS to enhance the low-level perception and understanding abilities.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
100% of the annotated labels in the LLVisionQA and LLDescribe datasets (question- answers and long golden descriptions) are conducted by human experts. We sincerely thank their efforts.
We thank the anonymous reviewers on ICLR2024 Conference on providing valuable and construc- tive suggestions for us to improve this paper.
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
REFERENCES
Recommendation 500-10:
Methodology for the subjective assessment of the quality of television pictures. ITU-R Rec. BT,500,, 2000.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning, 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Yuming Fang, Hanwei Zhu, Yan Zeng. Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In CVPR, 2020.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient 2023.
visual instruction model. arXiv preprint 2334
Deepti Ghadiyaram and Alan C. Bovik. Massive online crowdsourced study subjective and ob- jective picture quality. IEEE, 25(1):372-387, 2016.
Tanaya Guha, Vlad Hosu, Dietmar Saupe, Bastian Goldlücke, Naveen Kumar, Weisi Lin, Victor Martinez, Krishna Somandepalli, Shrikanth Narayanan, Wen-Huang Cheng, Kree McLaughlin, Hartwig Adam, John See, and Lai-Kuan Wong. Atqam/mast Joint workshop on aesthetic and technical quality assessment of multimedia and media analyties for societal trends. ACM MM, pp. 4758 4760, 2020.
Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Koniq- 10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE TIP, 29:4041 29:4041-4056, 2020.
10

---

Published as a conference paper at ICLR 2024
Jingwen Hou, Weisi Lin, Yuming Fang, Haoning Wu, Chaofeng Chen, Liang Liao, and Weide Liu. Towards transparent deep image aesthetics assessment with fag-based content descriptors. IEEE TIP. 2023.
Huggingface. Introducing idefies: An open reproduction of state-of-the-art visual language model, ace. cw/bloc///ooooooo.ooo.. wwwpo///g/efffff
2023. hRttts::://hugg
Dinesh Jayaraman, Anish Mittal, Anush K. Moorthy, and Alan C. Bovik. Objective quality assess- ment of multiply distorted images. ASILOMAR, pp. 1693-1697, 2012.
Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In ECCV, 2016.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. arXiv preprint arXiv:2308.0000 00692, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension, 2023b.
Chunyi Li, Zicheng Zhang. Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. Agiqa-3k: An open database for ai-generated image quality assessment, 2023c.
Dingquan Li, Tingting Jiang, Weisi Lin, and Ming Jiang. Which has better visual quality: The clear blue sky or blurry animal? IEEE TMM, 21(5):1221-1133,, 2019.
Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-1Ok: A large-scale artificially distorted iqa database. In QuMEX, pp. 1-3, 2019.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2023c.
Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. Evaluation and mitigation of agnosia in multimodal large language models, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Claudio Michaelis, Benjamin Mitzkus. Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness object detec- tion: Autonomous driving when winter is coming. arXiv preprint arXiv: htt.//907779555 2019.
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making "completely blind" image quality analyzer. IEEE Signal Processing Letters, 2003)220 2013.
Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aesthetic visual analysis. In CVPR, pp. 2408-2411 2012.
OpenAI. Gpt-4 technical report, 2023.
Zhiliang Peng, Wenhui Wang, Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

---

Published as a conference paper at ICLR 2024
Shaolin Su, Vlad Hosu, Hanhe Lin, Yanning Zhang, and Dietmar Saupe. Konig++ Boosting no- reference image quality assessment in the wild by jointly predicting image quality and defects. In The British Machine Vision Conference (BMVC), pp. 1-12. 2021.
Hossein Talebi and Peyman Milanfar. Nima: Neural image assessment. IEEE TIP, 2018.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Accessed: 2023-05-05.
1lms, 2023. WRL.ww..W... mosa cml com cwm/blog/mpt--111 7b]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Jianyi Wang, Kelvin C. K. Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images, 2022.
Yilin Wang, Junjie Ke, Hossein Talebi, Joong Gon Yim, Neil Birkbeck, Balu Adsumilli, Peyman Milanfar, and Feng Yang. Rich features for perceptual quality assessment of uge videos. In CVPR, pp. 13435-13444, June 2021.
Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling. In ECCV, 2022.
Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, Jinwei Gu, and Weisi Lin. Neighbourhood representative sampling for efficient end-to-end video quality assessment, 2023a.
Haoning Wu, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Erli Zhang, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring opinion-unaware video quality assessment with se- mantic affinity criterion. In International Conference Multimedia and Expo (ICME), 2023b.
Haoning Wu, Erli Zhang. Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In ICCV, 2023c.
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Towards explainable video quality assessment: A database and language-prompted approach. In ACM MM, 2023d.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
Zhenqiang Ying. Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality. In CVPR, 2020.
Zhenqiang Ying, Maniratnam Mandal, Deepti Ghadiyaram, and Alan Bovik. Patch-vq: 'patching up' the video quality problem. In CVPR, 2021.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Cheng Zhang, Shaolin Su, Yu Zhu, Qingsen Yan, Jinqiu Sun, and Yanning Zhang. Exploring and evaluating image restoration potential in dynamic scenes. In CVPR, pp. 2057-2066, 2022.
12

---

Published as a conference paper at ICLR 2024
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan. Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingeheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Interolm-xcomposer:: A vision-language large model for advanced text-image comprehension and composition, 2023a.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, pp. 586-595, 2018.
Zicheng Zhang. Wei Sun, Tao Wang, Wei Lu, Quan Zhou, Qiyuan Wang, Xiongkuo Min, Guangtao Zhai, et al. Subjective and objective quality assessment for in-the-wild computer graphics images. ACM TOMM, 2023b.
Zicheng Zhang, Wei Sun, Yingjie Zhou, Haoning Wu, Chunyi Li, Xiongkuo Min, and Xiaohong Liu. Advancing zero-shot digital human quality assessment through text- eext-prompted evaluation, 2023c.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
A APPENDIX
A.1 MORE INFORMATION ON BENCHMARK DATASETS
A.1.1 SUBJECTIVE EXPERIMENT
A total cleven experts, each with professional skills and extensive experience in photography, are invited to participate in the subjective labeling experiment of Q-Bench. The subjective experiment takes place in a laboratory environment with standard indoor lighting. A Dell-4K monitor, which supports resolution of 3840 x 2160, is used for displaying the interfaces. The screenshots of interfaces can be referred to in Fig. 5 Each expert annotates up to 30 images day to avoid fatigue, and every annotation is carefully reviewed by least three other experts before acceptance. In this way, we ensure the accuracy and rigor of the Q-Bench labels to the greatest extent possible. This, turn, makes the performance testing capability of Q-Bench more precise and meaningful.
A.1.2 MORE DETAILS ON I.LVISIONAA
The Enumeration on Distortions and Other Low-level Attributes:
Blurs [lens blur (out-of-focur) focus,,,) motion blur, blur, gaussian blur, glass blur), Noises [gaus- sian noise, speckle noise, pepper noise), Artifacts [compression artifact, transmission error), Exposure Issues [under-exposure, over-exposure). Miscellaneous Artificial Distortions [pixelate, color-diffusion, jitter, etc] Other low-level attributes: Color [color style, color vividity]. Lighting [bright, dim]. Composition [Sym- metrical, (ile-oofThirds], Visual Styles [animation, realism, computer-generatte Al-generatedJ, Photographic Methods [hackground bokeh (shallow DOF), high contrast, motion blur (on fast-moving objects), etc)
Distortions:
Relationship between In-context Questions and Global Questions:
Distortions: Is this image blurred? + In-context Distortions:
Is the tree in the image blurred?
Distortions: How is the clarity of the image? In-context Distortions: How the clarity of the man's face?
Other low-level: Is this image colorful? - In-context Other: Is the house colorful?
Other low-level: How brightness of the image? In-context Other: Which the darkest object?
13

---

Published as a conference paper at ICLR 2024
Guiéelnez:
Clsk SSer and Nest button proced the cisk "reeioue bunco recem the last bern click "Iave to case your resubt
Click "cet ciese the application
Inorr Queetion liene
this away. rulleciing From, orarrcagreeee
Answer ie
Ansoer 1:
Answer 2:
Arower 3:
TAe
Rt:
Interface for the LLVisionQA dataset (Perception)
Cuidel
Srvaieec modis the
Send Cire bulf in cose application
your
evesall claucty Cass Laage vayy Eae sace
Send and Nove
Proilous
Selz
EdHt
(b) Interface for the LLDescribe dataset (Description)
The illustration of the annotation interfaces for the LLVisionQA dataset (questions, an- swers) on Peception ability, and the LLDescribe dataset (text description) on Description ability.
Figure 5:
A.2 DETAILS ON BENCHMARK EVALUATION SETTINGS
A.2.1 EVALUATION DETAILS FOR PERCEPTION ABILITY
[Special Note] Multi-choice Question vs Close-Set Inference for Kosmos-2:
While Kosmos-2 performs generally well on the description and assessment tasks, we notice that is hardly capable of answering a multi-choice question with the general prompt form applicable for other methods, as follows:
How the clarity of the image? (Question ion) [IMAGE TOKEN] (Image)
Choose between one of the following options: A. High (Correct) B. Medium (Wrong) C. Low (Wrong)
For most situations (86%) our primary sample test with the prompts above, Kosmos-2 will di- rectly append new candidate (e.g., D. Excellent or D. Very Low) answer instead of choosing one option among them, denoted as prompt failure. This might be because the language model of Kosmos-2 has smaller capacity (IB) than other MLLMs that are based LLaMA/MTT (7B/13B).
Considering that the prompt failure is actually not directly related with low-level perception, we try different prompt engineering techniques to reduce the prompt failure rate, and finalize with simple modification which can limit the prompt failure to less than 10% in our sample set, as follows:
How the clarity of the image? (Question) IMAGE.TOKENT (Image) Choose between one of the following options: A. High (Correct) B. Medium (Wrong) C. Low (Wrong) #Answer:
Nevertheless, we are still not able to eliminate the prompt failures for Kosmos-2. Henceforth, to systematically remove the negative effect of prompt failures multi-choice questions for Kosmos-
14

---

Published as a conference paper at ICLR 2024
Table 5: Perplexity-based close-ser evaluation compared with normal evaluation on LLVisionQA; after eliminating the prompt failures, the results of Kosmos-2 significantly improved.
Sub-categories
Question Types
Quadrants of Low-level Concerns
In-context
In-context Othert
Overall
Model (variant)
Yes-or-Not
Whart
Howt
Distortiont
Othert
Distortion
random guess
50.00%
28.18%
33.30%
37.54%
38.49%
38.70%
36.50%
37.87%
Kosmos-2 2 (normal)
58.20%
29.13%
34.22%
38. 38%
44.30%
40.93%
44.20%
41.47%
x
Kusmoss2 (clowe-set)
61.48%
37.13%
40.76%
40.04%
50.88%
45.30%
58.15%
47.26%
S L
ST. 185
Figure 6: Image of example (1). Figure 7: Image of example (2). Figure 8: Image of example (3).
2, we conduct a choice-free special setting for it, ie. close-set inference, via ranking the perplexity of different answers and choose the answer with minimum generative loss:
How is the clarity of the image? /IMAGE. TOKEN] #Answer: High + loss: 7. + Choose this.
How the clarity of the image? [IMAGE_TOKEN] #Answer: Medium + loss: 7. 56 + X
How is the clarity of the image? [IMAGE.TOKEN #Answer: Low + :.oss:7.92 + X
As shown in Tab. [ perplexity- close-set inference notably improve results of Kosmos-2. Considering that it is still the MLLM with fewest parameters among the ten models, its results are decent at its model size. More importantly, they validate that observation the prompt failure reasonable, and we will further delve deeper into this problem of MLLMs in our extended works.
Settings for GPT Evaluation:
Given GPT's inherent variability, identical prompts can yield non-definitive responses. To address the impact of such situations
evaluation, we've implemented 5-round voting strategy. Under this approach, we pose the same prompt as defined in the following templates five times, taking the popular votes of GPT's answers to determine the final outcome. Our human analysis sample set confirms that the 5-round voting strategy improves GPT evaluation accuracy from 93.2% % to 98.4% %. reducing errors to only 1/4 compared with the single-round evaluation.
Prompt Templates for GPT Evaluation:
#System: You are a helpful assistant that grades answers related to image quality and aesthetics. There are lot of special terms or keywords related to image processing and photography. You will pay attention to the context of qqaality evaluation' when grading.
#User: Assuming are grader. will now be provided with a question [question] and a set of options [options] with option [options[O]] being the correct answer. Additionally, there will an answer [answer] provided by respondent. Please determine whether the respondentó answer is correct considering the context of the question. Even if the word choice is not completely
same, you can decide based on the given options and see whether the one in the answer is close enough to the given correct answer, The result is I if the answer correct and else the result is Please only provide the result in the following format: Result:
Examples for GPT Evaluation:
(1) "Rephrased" Answers. (Fig.
Question: Which is the brightest part in this image?
Choices: [Caapital letters E and S', 'ST', '18', '56']
MLLM Answer:
15

---

Published as a conference paper at ICLR 2024
The brightest part in this image is the neon sign with the letters "E" and "S" in capital letters.
5-Round GPT Answers:
[ ('Coore: 1", "Score: 1", "Score: I"." "Score: 1", "Score: I"]
+ Final Correctness after Voting: /
(2) "Distracted" Answers. (Fig.
Question: How's the level of blur in the image?
Choices: ['Some blur', 'oot blurry at all', "Very blurry']
MLLM Answer:
The image has some blur, but it is not very blurry.
R-Round GPT Answers:
[ ("Score: 1", "Score: 1", "Score 1", "Score: 1", "Score: ["]
+ Final Correctness after Voting: //
(3) "Both" Answers. (Fig. 8
Question: Is the little boy emphasized in the center of the composition of the image?
Choices: ('No', "Yes"]
MILM Answer:
A. No, the little boy is not emphasized in the center of the composition of the image. B. Yes,
5- S-Round GPT Answers:
[ ("Score: 1", "Score: 0", "Score: 0", "Score: 1", "Score: 0"]
+ Final Correctness after Voting: X
A.2.2 EVALUATION DETAILS FOR DESCRIPTION ABILITY
General Description Prompt for MLLMs:
#User: Describe the quality, aesthetics and other low-level appearance the image in details.
Settings for GPT Evaluation:
Given GPT's inherent variability, identical prompts yield non-definitive responses. To address the impact of such situations on our evaluation, we've implemented a 5-round average pooling pose the prompt as defined in the following templates
strategy. Under this approach, five times, taking the mean result GPT's answers to determine the final outcome. This method effectively mitigates the unpredictability associated with GPT, ensuring a more accurate score.
Prompt Templates for GPT Evaluation:
#System: You helpful assistant.
Completeness. #User: Evaluate whether the description [MLLM_DESC] completely includes the low-level visual information in the reference description [GOLDEN DESC].
Please rate score 2 for completely or almost completely including reference information, 0fr not including at all, for including part of the information similar description.
Please only provide the result in the following format: Score:
Preciseness. #The precision metric punishes controversial low-level descriptions output description contrasts with the referencce, e.g., blur for clear, high quality for low quality, colorful for monotonous, noisy for clean, bright for dark.
Evaluate whether output [MLLM [DEC] precisely reflects reference [GOLDEN_DESC,
Please rate score 2 for totally no controversial low-level description, for less controversial low-level description than matched descrpition, and 0 for more controversial low-level description than matched description. Please only provide the result in the following format: Score:
Relevance. #User: Evaluate whether the description [MLLM_DESC] is relevant the low-level visual information, which may include blur, noise, exposure, artifact, color, lighting, focus, composition, etc. Please rate 2 for completely relevant, for partly relevant, and O for totally irrelevont.
16

---

Published as a conference paper at ICLR 2024
Please only provide the result in the fallowwnggformatt Score:
In the prompt template, the [MLLM-DESC] denotes the output description from MLLMs, and [GOLDEN DESCJ denotes the golden description in the LLDescribe dataset.
Examples for GPT Evaluation:
(A) Completeness.
User Input:
#User: Evaluate whether the description [ The image is a large, clear, and detailed picture of a white airplane flying in the sky. The airplane appears be JAL. (Japan Airlines) passenger jet, and it is flying at relatively low altitude. The plane's 's white color and its distinctive red logo are clearly visible, making it easily discernable as JAL aireraft. Overall, the image captures clear, beautiful scene of the airplane in flight, with a blue sky as the backdrop.] completely includes the low-level visual information in the reference description (This image has poor lighting. The blue sky has vivid colors, but the overall tone is dark. The focus is on the main subject. the aircraft. Apart from the backlighting on the wings, the details of the aircraft are clear. The composition is slightly incomplete, and the clarity of the background sky is average. Overall, the image quality is good. Please rate score for completely or almost completely including reference information, 0 for not including at all, 1 for including part of the information or similar description. Please only provide the result in the following format: Score:
5- 5-Round GPT Answers:
1 ("Soree: 1", "Score: 1", "Score: 1", "Score: 1", " "Score: ]
+ Final Score: 1.0
(B) Preciseness.
User Input:
#User: The precision metric punishes controversial low-level descriptions that output description outputs contrast with the referencce. e.g.. blur for clear, high quality for low quality, colorful for monotonous, noisy for clean, bright for dark.
Evaluate whether oulput [The image quality is good, with clear and well-exposed scene. The lighting appears to be natural, as the sun is shining on the scene, which adds a warm and inviting atmosphere the image. The composition pathway, which creates a sense of depth and perspective. The aesthetics of the image are pleasing, with the use of natural light and the greenery of the trees contributing The overall feeling of the image is peaceful and serene, as the viewer is invited to walk down the pathway surrounded by the trees and the cityscape in the background.] precisely reflect reference [hhe overall clarity of this image is very low, with serious focusing issues and significant noise. The lighting is udequate, but the colors are monotone. The main subjects are trees and distant buildings, which are extremely blurry and lack any texture details. The composition is unbalanced, and the background very blurry. Therefore, the quality of this image is very poor.), Please rate score 2 for no controversial low-level description, for less controversial low-level description than matched descrpition, and 0 for more controversial low-level description than matched description. Please only provide the result in the following format: Score:
well well-balanced, with the focus on the trees and the
visually appealing scene.
5- Roondd GPT Answers:
Scoore: 0", "Score: 0", "Score: 0", "Score: 0", "Score: 0"]
+ Final Score: 0.0
(C) Relevance.
User Input:
#User: Evaluate whether the description The image is a low-level shot of white dog walking through a dark forest. The dog appears to be somewhat blurry, suggesting a level of motion in the picture. The photo is not very detailed, and the colors in the image might be somewhat muted due to the darkness of the forest. Overall, the picture has somewhat mysterious and moody atmosphere.] relevant the low-level visual information, which may include blur, noise, exposure, artifact, color, lighting. focus, composition, etc.
17

---

Published as a conference paper at ICLR 2024
Please rate score 2 for completely relevant, 1 for partly relevant, and 0 for totally irrelevant. Please only provide the result in the following format: Score:
5-Round GPT Answers:
["Score: 2", "Score: 1", "Score: 1", "Score: 2", "Score: 1"]
- Final Score: 1.4
Algorithm Pytorch-style Pseudo Code for Softmax-based Strategy for IQA with MLLMs
from PTL import Image
from my mllm model
m import Model, Ickenizer, embed image amage_andd__ text
model, tckenizer Model O), Tokenizer
prompt "AAseer: Rate the quality of the image. An" \ "HAssistant: The quality of the image is"
good_idx, pecr_ids tokenizer (["geod", "poor"]) .tolist ()
image = Image. open (" i"inage_for_iqa__ee
input_embeds embeds = embed_ andgge__nn__ettttte (image, prompt)
output_logits logits model Ltw.ppuutembbedd-iuuttteeeeeemmt (mtpepdsuttembes Logiiss0,,-1
prea [issput_oggit|//// I ggood_ [oor_idx]] / 100) softnax (0) [0]
A.2.3 EVALUATION DETAILS FOR ASSESSMENT ABILITY
Example Pseudo Code for MILMs on IQA:
In Algo. [ we provide an example on how to evaluate image quality with MLLMs. The algorithm is simple with only 9 lines, and could be easily integrated with any new MILMs (based causal LLMs), so as to allow these models to quantitatively predict the quality of images.
IQA Evaluation Strategy for CLIP-ViT-Large--e
In Tab, 4 we compare the IQA performance of MLLMs with LiT---eT- 14, the visual back- bone of the majority of MLLMs. Attempting to understand whether the new language part (LLM) can do better than the original language part of CLIP, we try to compare between CLIP and MLLMS
in relatively aligned setting. Firstly, noticing that most MLLMs will resize images into 224 224 as their input sizes, we align this setting on CLIP, and ignore the strategies as proposed by (Wang al. 2022). Secondly, same the strategy MILMs,, we also apply sof pooling between good and poor, as in the CLIP's zero-shot classification format: a photo of good quality and photo
of poor quality. Besides the two alignments, similar existing practices (Wang al. 2022 Wu
ct al. 2023b Zhang el al. 2023c), the quality scores of CLIP- -LIP-VvT-Large- 14 are obtained as follows:
Consinniiillaitty fa good quality CosineSimilarityy) paacE) I' quality
9pred,CLIP
CosineSimilarity (fpeacttttjyyy fa goo qualit)
(2)
Special IQA Settings for Flan-T5-based InstructBLIP:
For InStructBLIP (Dai et al. 2023) (Flan-75-XL), different from the majority of LLaMA-based (or MPT-based Otter-vl) MLLMs, the two top-frequency tokens are high (89%) and low (8%) instead of the common good-poorr Henceforth, based on our motivation to only modify the argmax into softmax and follow the default top-frequency output tokens of MLLMs, we replace the probabilities of good into those of slo in Eq. for T5, defined as follows:
high
Ipred, T5
High SCORE JURAN
(3)
1ORDN
18

---

Published as a conference paper at ICLR 2024
What Questions
How Questions
Yes-or-No Questions
Distortion
Overall
98
4
Other
In-context Other
In-context Distortion
LLavA-v1.5-7B
LLaVA-v1.5-138
IDEFICS-Instruct
Qwen-VL
Shikra
Otter-v1
instructBLIP-TS
InstrucIBLIP-VE
VisualGLM-6B
MPLUG-OWl
-LaMA--Adpter-V2
LLavA-v1
MiniGPT-4
Figure 9: Radar chart for the Perception ability. where the performance of all MLLMs is presented by subtracting the accuracy of random guess. See Tab. [orr the respective numerical results.
Table 6: A brief view of 15 open-source MILMMs evaluated in the Q-Bench in chronological order.
MokerMode
Vision Architoctures (V)
V-l.
Language Architectures (1.)
Backbone
ASize
Alignment
Backbonc
Type
00 LLaVA-115 (Vromm-u7.5578)
Lii
20230
CLIP-NNT-Lage
336
MLP
Vicum-vi1.5.0 1.5.BB
pure-decoder
Liu 21250
FLIP--iI-Larg--E
336
project MLP
Vicuma-v1.5-133
pure-devoder
Set InternLM-XX MeteanLLMXXCCoooee--- (Juterm)
2023a
EVA-CLIP-Gian-L
224
Q-Fommer
InterLM
pure pure-decoder
IDEFICS-SIIE IDEFICS-Instmidl (MA-M))78)
Hugginglace
2123
CLIP-ViT-Hage-
14
224
cross-atth
LLLMA-78
decodder pure decouer
Qwen-VVL (QwenLM) ctall 2025
CLIP-VIT-Giat-I
448
cross-atto
QWenLM
MM Shikra (Virwno-7W) CChen
CLIP-VVT-Lagge argg--1
224
project layers
Vicuna-78
deroder pure decoder
Ottes-vl (MPF-7B)
Lt 20236
CLIP-VIT-Larg---
224
cross-alto
MPT-78
^m Kosmos-2 Peng
CLIP--VT-LLarg--.. Lrrge-1
project layers
custom (1B)
pure pure-deroder
InstructBL PPTooo- X5-X Dhi
2023
EVA-CLIP-Gant-I
224
Q-Forner
Hlnn-T-XLL
encoder-deeoderr
Moo InstructBLIP (Vicana-7B)
sl.
2025
SV-CLIP-Giant-
224
Q-Foreer
Vicuna-7B
pure- decoder
V6 (GLM-68)
DA-CCLPP-GBAS
14
224
Q Former
GLM-6B
encoder decoder
mPLUG-Owi (LLaMA-78)
Ye al
2023
CLIP-VIT-Lag--1
224
Q-Fomee
LLaMA-7B
pure decoder
Ax LLLMMA-Adppter
Largge1--
pure
L*LLVVA-v (Vicwna-1336
138)
CLIP-ViL Lil--Vi--
336
project layers
Vicuna-155
pure-deecude
Ax MiniGPT-4 (Vicuma-13B)
Zha
CLIP-VIT-Larg-I1
224
Q-Former
Vicuna- 13B
pure reco-dr
the high+>low pair generally predicts better than good<>poor on majority databases. The better performance on MLLM-specific top-frequency tokens by side validates the effectiveness of our methodology for MLLMs on IQA.
As validated in our experiments (Tab. 10
Further Improving IQA Abilities of MLLMs with Synonym Ensemble:
The quality assessment scores for the synonym ensemble strategy can be derived as:
StEP A.COR JOREN
(pred DIEP TORDRF_TOKTN + EIEN #'CORF.TOKEN
where P indicates the positive token set (from good. fine, high, etc.), while A represents the negative token set (from poor, bad, low, etc.). The results of different P and N are listed in Tab. II
Special Validation Protocol for CGIQA-6K:
The CGIQA-6K (Zhang et al. 2023b) dataset contains two separate sub-sets which consist of 3,000 game images and 3,000 movie images respectively, with different instructions for human annota- tors during its subjective experiments. Therefore, validate the MLLMs assessment performance on the two sub-sets individually and average the results for the final exhibition. The results of NIQE and CLIP-ViS ViTV-a---r 14 are also obtained under the same protocol for fair comparison.
19

---

Published as a conference paper at ICLR 2024
Table 7: Results on the dev subset for the low-level Perception ability of MLLMs. MLLMS with 3op-3 performance in each sub-category and the overall LLVisionQA is emphasized with boldface.
Sub-categories
Question Types
Quadrants of Low-level Concerns
In-coniext Distortiont
In-contest Other
Overall
Model (variant)
Yés-or-Nof
Whart
Howt
Distortion?
Other?
randow guess
50.00%
27.86%
33.31%
37,89%
38.48%
38.28%
35,82%
37,80%
LLaVA-vT.5 5.5 [Mouna-97.577)
66.366
58.19%
50.51%%
39.42%
65.74%
54.61% %
70.61%
58.66%
LLaVA-v1.5 VVeuma--1_5-1333
65.27%
64.38%
56.59%
56.03%
67. 13%
61. 18%
67.35%
62. 14%
InternLM MLLLLM--XCooooWWVVV VL (ItterLMM)
69.45%
65.27%
60.85%
61.67%
70.14%
56.91%
75.10%
65.35%
IDEMC'S- MebHiic--nsttuutttccccc (LLaMA-7B)
18%
44.69%
44.02%
80%
54.17%
44.74%
56.33%
48.70%
Owen VL (QwenLM)
63.09%
58. 19%
56. 391
50.58%
62.73%
57.89%
73.88%
59,40%
Shikra (Vicuna77B)
65.64%
47.35%
49.09%
48. 83%
59.49%
50.00%
64.08%
54.65%
Otter-vl (MPT-78)
57.09%
40.71%
99.55%
42. 22.
49.31
44.08%
52.65%
46.35%
InstuctBLIP (Flon-T5-XL)
67.64%
59.96%
55.98%
23%
65.51%
58.22%
69.39%
61.47%
InstuciBLIP (Vicuna77B)
71.64%
52.65%
43.81%
6..
62.50%
55.59%
64.90%
56.72%
VistalGLM-68 (GLM-68)
60.18%
54.20%
46.25%
51.75%
54,40%
53.62%
57.14%
53,78%
mPLL Owl (LIaMA-7B)
66.0%
54.87%
44.02%
51.36%
55.09%
54.28%
65.71%
55.38%
MLaMA-Naapeer- V2
66. 18%
59.29%
52.13%%
57. 39%
56.25%
63.16%
64,90%
59.46%
LLaVA-v1 (Vicuna-138))
54.00%
53.10%
54.63%
55.59%
63.27%
54.18%
MiniGPT-44 (Vicuna CiViiiP--VVciuaeea 138)
55.82%
50.22%
40.37%
42.02%
48.38%
$1.97%
61.22%
49.03%
A.3 EXTENDED EXPERIMENTAL RESULTS
A.3.1 ARCHITACTURES OF DIFFERENT MILMS
As compared in Tab. 6 the 15 variants of MILMS as evaluated in the Q-Bench are with varying vision and language architectures, as well as the alignment strategies between the two modalities. It can be noticed that all MLLMs are combined with version of CLIP Radford et al. (221)) and a large language model, which are generally connected under one among three strategies: direct project layers (MLP or linear layer), Q-Former (a transformer to abstract visual features into LLM tokens), or cross-attention (use visual features as conditions for text generation).
A.3.2 EXTENDED RESULTS FOR PERCEPTION
Results on the dev subset:
In Tab. 7 we list the results on the dev subset of the LLVisionQA benchmark set for the low- level perception task. This subset is planned to be opened to public in the future. Therefore, the performance in it will only be taken yet seen this subset, so it can be taken as a cross-validation with the test subset. From Tab. and Tab. 2 validate that MILMS perform pretty similar between the two subsets, suggesting that LLVisionQA is reliable and stable benchmark set for question answering on low-level vision.
reference. At present, all MLLMs evaluated have not
Radar Chart for Different MLLMs:
In Fig. 9 we show the radar chart to compare the low-level perception abiliies among different MLLMs. Despite the observations revealed in Sec. 3.1 also notice two extra fun facts: 1) Adding the content context does not degrade the performance of MLLMs. On the contrary, MLLMs can answer better on in-context questions. This result validates the aforementioned conjectures that appropriate higher-level contexts as prompts may help improve the preciseness of low-level visual perception; 2) MILMs have strong capabilities of answering what questions, suggesting potential reasoning abilities. In the future, we will excavate more interesting characteristics of MLLMs and try to improve their perception accuracy through better guidance based on these characteristics.
"Yes or No?": How Biased are MLLMs?
In this section, we take a deeper analysis on the Yes-or-No judgment ability of MILMSs, that whether these models can get similar accuracy on questions that should be answered with Yes, as those should be replied as No. Sadly, we notice that all MLLMs have higher prediction accuracy on Yes-questions than No-questions, while some MLLMs are more very severe biased (e.g. IDEFICS- Instruct). Considering that our LLVisionQA dataset contains more (62%) Yes res-questions than No- questions (38%) and may introduce biases while comparing different MLLMs, we further compute a de-biased accuracy for all these methods, as the mean value of the accuracies on two types of questions, and present the respective de-biased rank for all participating MILMs,, as listed in Tab g
20

---

Published as a conference paper at ICLR 2024
Table 8: Judgment accuracies of MLLMS on questions with correct answers Yes No.
Model (variant)
all
correct answer: Yes
correct answer:
mean
de-bbased
random guees
50.00%
50.00%
50.00%
50.00%
Shikra (Vicuna-7B)
66.91%
71.79%
60.00%
65.90%
LLaVA-v1 (Vicuna-138)
57.10%
60.29%
51.66%
55,97%
MiniGPT-4 (0TT-4(Viicuu (Vicuna-13B)
57.50%
70.00%
37.38%
53.69%
LLLMA-Addpiers V2
67. 12%
68.80%
64.76%
66.78%
2
InstructBLIP (Flon-T3-XL)
68.67 %
80.14%
50.23%
65.19%
InstructBLIP (Vicana-7B)
71.40% %
84.32%
50.47%
67.39%
Otter-v1 (MP1-78)
57.74%
70.14%
37.38%
53.76%
IDEFICS-Instruu (LLaMA-7B)
59.74%
88.65%
13.09%
50.87%
mPLUG-Owl (LLaMA-7B)
69.31%
95.82%
26.67%
61.25%
Question: there any blur this image?
Question: Which part of the bumnn cropped of the image? A. head; His leg; His hand.
A. Yes, B.
Correct Answer: Yes.
Correct Answer: A. His head.
(b)
Shikra: The answer is arm.
LLaVA-vl: The purt the of the image leg.
is cropped
Shikra: The unswer is yes.
LLaVA-vI:
MiniCPT-4: B. His leg.
MiniGPT-::A A. Yes.
Kosmos-2: His leg.
Kosmos-2: Yes.
LLaMA--Adapte--Vz The image crops out the player's
AdaMA-Adapter-V2: Yes, there is blur in the image.
hand, focusing on the baschall player's log as he slides into the base.
InstruetBL.IP-E A. Yes.
InstructBLIP-Vieue The answer yes, there is blur this image.
InstructBLIIP-TS.C
X CeatructBLP-Vieumee D. His amm
Otter-v1: B. No.
X Otter-vl: His leg.
IDEFICS-Instrut: A. Yes.
X IDEFICS-Instruet: B. leg.
mPLUG-Owl: Yes, there is blur in this image.
X mPLUG-OWl: The image crops out the man's leg.
Figure 10: Qualitative comparison for MILMM perception responses.
We hope this study on the biases and the de- de-biased results can provide a fairer comparison among them, as well bring insights on the future improvements of MLLMs for low-level perception.
Qualitative examples of MLLM responses:
In Fig. 10 we show qualitative examples of MLLM responses on questions in the LLVisionQA dataset, that MLLMs are still unstable on basic low-level attributes such as blurs (Fig. TO(a)). and may fail on in-context questions that are easy human (Fig. [O)).. These unsatisfactory results suggest that we still need to improve the basic low-level perception ability of these models.
A.3.3 EXTENDED RESULTS FOR DESCRIPTION
Bar Chart for Different MLLMs:
In Fig. I] we show the bar chart to visualize MLLM capabilities on the three dimensions of low- level visual description. From the figure, notice that current MLLMs still struggle describing complete and accurate low-level information. As the relevance scores are generally higher (showing that most MLLMs follow this abstract instruction well), the results suggest that the main bottle- neck of MILMS on enhancing their description ability is still the perception on low-level attributes.
A Qualitative Comparison on the Descriptions:
21

---

Published as a conference paper at ICLR 2024
0.8
0.6
0.4
0.2
Completeness
Precision
Relevance
Sum.
LLaVA-v1.5-7B
LLaVA-v1.5-13B
IntemtM-XCCompose-VV.
IDEFICS-Instruct
Qwen-VT.
Shikra
Otter-v1
Kosmos-2
InstructBLIP-T5
InstructBLIP-vicuaa
VisualGLM-6B
MPLUG-OWl
LLaMA-Adapter-V2
LLaVA-v1
MiniGPT-4
Figure 11: Bar chart for the Description ability, with scores in all dimensions normalized into [0, 1].
Table 9: Effectiveness of the proposed softmax probability-basee strategy against the bascline argmax strategy, on multiple MLLMs and different IQA datasets. Metrics are SRCC/PLCC.
Dataset Type
In-the-wild
Generated
Artificial
Model Dataset
Straiegy
KONiQ-10K
SPAQ
LIVE-FB
LIVE-iw
AGIQA-3K
KADID- TOK
Shikra (Vicuna7BB)
azgmax
0.178/0.201
0.277/02281
0.152/0.169
0.248/0.267
0.513/0.562
0,245/0.246
Shikra (Vicma-78)
softmax
0.314/0.307
0.327/0.337
0.237/0.241
0.322/0.336
0.640/06611
0.324/0.332
LLaVA-vI (Vicuma-131)
argmax
0.03870.045
0.101/0.108
0.036/0.033
0.009/00077
0.24000297
0.005/0.00 0005000005
LLaVA-vl (Vicuma-33B)
softmax
0.462/0.457
0.442/0.462
0.264/0.280
0,404/0.417
0.626/0.684
0.349/0.372
LLaMA-Adupler--V?
argmax
0:218/0.237
0,417/0.423
0.222/0.257
0.205/0.239
0,545/0.579
0,228/0.229
ILaMA-Adapter- V2
so fomax
0.354/0.363
0,464/0.506
0.275/0.329
0.298/03600
0.604/0.666
0.412/0.425
InstructBLIP (Vicuma77B)
argmax 0.284/0.352
0.662/0.664
0.156/0.249
0.195/0.264
0.505/0.567
0.305/0.307
InstrueBBI. JP (Vicuma-78)
softmax
0,359/0.433
0.683/0.689
0.200/0.283
0.25300.367
0.629/0.663
0.337/0.382
mPi. (f..MAA
argmax
00.11770.554
0.463004699
0.088 123
0.166/702237
6.410/04466
0.203/..20
mPLUG-Owl (LLaMA-7B)
soltmax
0.409/0.427
0.634/0.644
0.241/0.271
0,437/0.487
0.687/0.711
0.466/0.486
In Fig. 12 and Fig 12 qualitatively compare among different MILM low-level descriptions an Al-generated image and a natural photograph. While most MILMs can precisely describe their contents (which actually not instructed in divergences on their quality and related low-level attributes, especially on the Al-generated image. Some MILMs describe it clear, colorful, or aesthetically pleasing, which typically incorrect; on the contrary, some correct descriptions are also seen, such colors are subdued, not as clear as could be, or not very detailed. This qualitative study validates our quantitative conclusion that current MILMMs general cannot provide noise-free low-level visual descriptions of images in a stable manner. Moreover, we notice that even given the same prompt, different MILMS tend to describe the image with diverse output styles and lengths, while the longer descriptions usually come with larger percentage of descriptions on irrelevant information.
prompt), different MLLMs may have several
A.3.4 EXTENDED RESULTS FOR ASSESSMENT
Radar Chart for Different MLLMs:
In Fig. 14 visualize the IQA performance of different MLLMs seven IQA datasets. The visualization proves that MLLMs can notably positively correlate with human ratings, and could very highly align with human perception relatively coarse situations (AGIQA-3K, SPAQ). As
22

---

Published as a conference paper at ICLR 2024
User: Describe the quality, aesthetics and other low-level appearance of the image in details.
Golden Description: This photo has relatively low clarity, making it difficult to discern the details of brishstrokes the center artwork. There substantial loss of detail, and the ovenll color saturation is very low, giving it a dim and dull appearance. Therefore, the quality of this photo low.
The image features a portrait of a woman, with hair parted to side. Her facial features are realistic and clear, and she appears
Shikra:
looking the side with serious expression. The overall quality of the image is good, with no obvious flaws or distortions in the depiction..
The image is pencil driwing of woman with serious expression. The quality of the drawing is quite good, with clear and detailed representation of the woman's 's facial features and hair. The exposure appears well-balanced, and there is no significant noise level characterized by the use muted colors. which give the image somewhat somber and serious tone. The composition is well-balanced, with the woman's face taking up most of the space in the drawing. Overall, the image has a professional and polished appearance, with a on woman's facial expression her overall demesnue.
LLaVA-vi:
image. aesthetics of the drawing are
MiniGPT-4: The image portrait weman with long. straight black and scrious expression. The woman has pale complexion and dark eyes. which are locused on the viewer. The woman's face oval in shape and her features are well defined. The woman's nosc is straight and ber lips are thin and closed. The woman's eyes large and her eyebrows are thick and stright. serious. The imuge well lit and the lightnng highlights the woman's features. The image in color and the colors are muted and subdued. The image is of high quality and the details are well defined. The image is aesthetically pleasing and overall appearance of image professional.
woman's face symmetrical and facial expression
Kosmos-2: image has low-quality quality, as is painting woman with a long, dark hair, The quality of the painting is not clear it could be, the womm's face not fully visible. The lighting is not well- balanced, and the overall atmosphere is not pleasant. The painting lacks a sease of style and fools unfinished.
LLMAA-Aaaters V2: The image is a black and white drawing of woman with a blue background. The quality of the image is clear, with good exposure no noticeable noise. The aesthetics of the image simple and minimalistic, with focus on the woman's face and the blue background. The composition is balanced and well- framed, drawing attention to the subject. The style of the drawing is realistic, captaring the essence of the woman's facial features and expression. The overall feeling of the image is serene and contemplative, conveying sense of calm and introspoction.
InstructBLIB
The image depicts a portrait woman with long. hair and brown background. The image has high level of quality and well-exceuted in terms of color, contrast, and texture, It also has strong contrast between the light and dark parts of the image, which creates a sense of depth and dimension. Overall, the image well-executed terms quality, aesthetics, and other low-level appearance.
InstruetBI JPP-VVe-naaa The image painting of woman's face, depicted realistic style. The painting features a close-up view of the woman's face, focusing on her eyes, nose, and mouth. Her hair is styled in a simple yet elegant manner, emphasizing her facial features. The overall aesthetics of the painting convey sense of realism and attention detail, capturing the subtle nuances of the woman's facial expression and appearance. The painting also showeases the artist's skillful use of color and shuding to create a lifelike representation of the subject.
The image features partrait of woman with long hair, painted in an oil painting style, The woman's s face is the main focus of the painting. and her features, such as her eyes, nose, and mouth, are well- defined, Her hair deseribed as dark, which adds depth and contrast to the overall composition. The painting appears to be close-up of the woman, emphasizing her facial features and creating sense of depth and detail.
Otter-yl:
The image is drawing a woman with long hair. The woman has small nose and a small mouth. The hair is brown and the eyes are dark, drawing in shades of gray and background white. The woman's hair long and flowing. The drawing very detailed and the woman's face not very clear. The drawing not very aesthetically pleasing and the woman's face is not very attmactive.
IDEFICS-Instruct
mPLUG-OWl:
The image is portrait of a woman with a brownish-ycllowish complexion, wearing white shirt. The woman's
the focal point of the painting, and her eyes are the most prominent feature. The painting has soft, dreamy quality, with warm, inviting atmosphere. The lighting natural, creating soft glow on the woman's face and bair. The composition well-balanced. with the weman's face tkking up most of canvas,
background providing a sense depth and context. The style the painting realistic, capturing the woman's likeness and emotions realistic manner. Overall, the painting is aesthetically pleasing and evokes a sense of calm and serenity.
Figure 12: A qualitative comparison for MLLM descriptions on an Al-generated image.
23

---

Published as a conference paper at ICLR 2024
User: Desoribe the quality, aesthetics other low-level appearance of the image details.
The overall clarity of this image is high, with rich and vibrant colons The compusition is exquisite and aesthetically pleasing. The texture details of the photographed fruit subject are abundani and clear. accurately cupturing the reflection effect of light. Therefore, the quality of this image is very high.
Golden Deseription:
Shikra; The image presents close-up view of delicious dessert, which appears be piece of fruit scoop of ice cream with berries on top. The dessert is lopped with whipped cream, giving it an appetizing and visually appealing appearance. In addition to the dessert, there are three red berries on top, adding a touch of color and sweetness the dish. The overall quality
imuge good, with a clear representation of the dessert and the berries, making it inviting and attractive scene for viewers.
VA-V: The image quality is high, with clear and well-exposed shot of delicious-lookke lookingg dessert. The dessert is fruit-baed based dish, featuring slice of peach with whipped cream on top. The lighting in the imuge is bright and evenly distributed. which enhances the colors and overall appearance of the dessert. The composition visually appealing, with the fruit image are further calanced by the use of natural light, which adds warm and inviting atmosphere the scene. Overall, the image is visually appealing, with a high level of quality and aesthetics that make it an enjoyable sight for viewers.
whipped cream arrunged in a pleasing manner. The aesthetics the
The image shows a close-up view of dessert dish consisting of sliced peaches topped with whipped cream and a sprig of mint. The peaches are fresh and ripe, with vibrant color and a smooth,, glossy texture. The whipped cream smooth and fluffy, adding creamy texture and a sweet flavor the dish. The mint leaf adds fresh and aromatic touch to the dish. The ovenall aesthetics of the image are pleasing and appetizing. The bright colors of the peaches and the whipped cream are visually appealing and make the dish look appetizing. The use fresh ingredients and the attention detail in the presentation look appetizing and well-made. The lighting the image is soft and natura, which enhances the colors and texture of the dish. The quality of the image is high, with good resolution and clarity.
MiniGPT-4:
make
The fruit is displayed a white plate, and the dessert is gamished with sprig of mint and few berries The quality of the photo is clear, with no distracting elements or noisc. The acathetics of the dossert are pleasing, the white plate and fruit creating a visually appealing scene. The low-light conditions, such as the white background and the bright colors of the fruit, contribute to the overall pleasant appearance the photograph.
Kosmos-2:
LaMA-Adaterr-V2 The image has high-quality, clear, and well-exposed photo of blue and white dessert. The aesthetics of the image are pleasing, with the blue and white color schemne creating visually appealing and appetizing presentation. The composition of the image is well-balanced, with the dessert placed prominenty in the venter of the frume, surrounded by white background The style showcasing the dessert way that highlights its beauty and appeal. The feeling of the image inviting and eaticing. making it an ideal choice for sharing on social media or as a visual representation of delicious dessert.
image is clean and professional.
The image captures a close-up of a dessert consisting of apple, strawherry, and piece of ice cream. The texture of the ice cream is smooth and creamy, with hint of sweetness and a hint of tartness. ice cream, which adds touch of color and flavor to the dessert, Overall,
InstructBL.IP-T
There also berry on lop the image has high level of quality and aesthetic appeal.
InstrucIBI.IP- Fee-Vin Vicuma: In the image, there is lociciiss-ookkngg dessert displayed on white plate. The dessert consists of various fruits, such as peaches and berries, arranged in a visually appealing manner. The fruits are placed on top of each other, creating a colorful and eye-catching display. Additionlly, the dessert is adorned with sprig of fresh mint leaves, adding a touch of freshiness and flavor to the dish. Overall, image showcases well-presente and visually appealing dessert, highlighting the quality and aesthetics of the dish.
Otter-v1: The image has high-quality appearance with clear and sharp details, acathetically pleasing colors, and balanced composition.
The image is clear and well-exposed, with no visible noise. The composition is well- balanced, and the colors are vibrant and appealing. The style is artistic, and the feeling is refreshing and invigorating
IDEFICS- Insttruct:
mPL.IG-Ow.: The image is close-up of a dessent with a slice of fruit on plate. The dessert is presented in dish with white background. which creates a clean and visually appealing appearance. The ftuit slice is placed on top dessert, adding touch of volor and contrast to the overall scene. The image well-exposed. with no notiocable noise or distertion. The aesthetics of the image calaneed by the use of shallow dopth of field. which focuses on the dessert and fruit slice. creating a sense of depth and drawing the viewer's attention to the subject. The image has a simple and clogant style, making an appealing and visually pleasing representation of the dessert.
Figure 13: A qualitative comparison for MLLM descriptions on an mild photograph.
24

---

Published as a conference paper at ICLR 2024
LIVE-FB
SPAQ
LIVE-IW
KONIQ-10k
5
20
CGIQA-6K
KADID-10K
AGIQA-3K
LLavA-v1.5-B
LLaVA-v1.5-138
mntrnLM-XCooppser-V
IDEFICS-Instruct
Qwen-VL
Shikra
Otter-v1
Kosmos-2
InstructBLIP-T5
InstructBLIP-Vicuna
VisualGLM-6B
MPLUG-OW
LLaMA-Adapter-V2
LLaVA-v1
MiniGPT-4
Figure 14: Radar chart for the Assessment ability, where the performance of all MLLMs is presented by an average of SRCC/PLCC metrics. See Tab. for the respective numerical results.
Table 10: Comparison between hight>low and good+>poor InstructBLIP (1an-T5-XLL.
Dataset Type
In-the-wild
Generated
Artificial
Model / Dataset
tokens
SPAQ
LIVE-FB
LIVE-iw
AGIQA-3K
KADID-IO ToK
InstructBLIP (Flan-T5-XL.)
goods-poor
0.581/0.618 0.334/0.362
0.288/0.289
0.221/0.231
0.017/0.020
0.264/0.281
0.264/0.220
InstructBLIP (Flan-T5-XL)
blow
0.582/0.599 0.248/0.267
0.113/0.113
0.378/0.400
0.211/0.179
such results are obtained via no direct alignment with human perception and from the simplest prompts ("Rate the quality of the image."), they suggest the exciting underlying abilities of general intelligence to naturally understand "quality" via their vast training data. However, the performance are still yet more than 95% high-quality images (i.e. quality score > 50/100), CGIQA-6K, made up entirely by relatively high-quality images collected from video games or movies. This suggests that MLLMS still need to improve the measurability their predictions through well-designed fine-tuning.
accurate on finer-grained situations, such LIVE-FB, which is constructed by
Deep Dive Into the Probabilities:
(A) Softmax Argmax:
In the first part of the deep dive, we quantitatively evaluate the correlation with human perception on simple a araax strategy between good+>bad and our proposed softmax strategy. In Tab. 9 we confirm that for MILMs on IQA datasets, the more measurable so ffax fmmaaax strategy predicts better than the argmax strategy, which degenerates into only two scores, 0 and 1. Though the result generally expected, the experiments validate that MLLMs have quantitative assessment ability hidden behind their word outputs, and prove the effécctiveness of our softmax-based IQA strategy.
(B) [For T5-based InstructBLIP] hight>low vs good+
We further conduct special study for InstructBLIP (Flan-75-XL). With different LLM as lan- guage backbone, even pre-trained with the same settings, the T5-version of InstructBLIP tends to predict more hight+loow than good+>poor, different from its Vicuna-78 ---aadd counterpart. The perimental results Tab I0 validate that the more probable hight>low tokens are more competitive in IQA than gaa tokens, suggesting that toe-freyyyncy tokens more quality-distinetive.
25

---

Published as a conference paper at ICLR 2024
Table 11: Evaluation results on the synonym ensemble strategy for the (A3) Assessment ability on MLLMs with top-5 results in the default A3 leaderboard of the Q-Bench. After ensemble, the rankings among them are not changed. Metrics are SRCC/PLCC.
Dataset Type
Inethe wild
Generated
Artifcial RADID-FOR
Prompt/ / Dataset
RONIO-TOR
SPA@
LIVE.FB
LJVE-iw
CGIQLAK
AGI@1.3K
Average
LLaVA-v1.5 (V5c-ne-11.--1BB)
goodsspoor
0.4480.400
0.56300584
0.31000.399
0.44504481
0.28500297
7.664/0.744
0,39000
0:444/0.45 475
finetbbu
0.44900.487
0.583/0 597
0.31690.360
0-466/0.513
0.349/0.36
0.6500.749
0,425/0.437
0.463/0.501
hight+low
0,456/0/182
0,5290.553
0.286/03006
0,1890.513
0.27640.284
0.683/0.752
0.316/0.381
0.434/0.160
goodthiighe-poorkoo
0,462/0.884
0,548/0,573
0.303/0.327
0.48000509
0,28300294
0.687/0.763
0.350/0.363
0.445/0.473
good-fine- sapeeeepooooadd bad
0,463/0.483
0,579/0.596
0.321/0.356
0.467/0.505
0.326/0.139
0.670/0.76 762
0.420/0.426
0,464/0.495
good righefiff spoor slow thad
0.47401.988
0.565/0.888
0.31480.345
0.488/05211
0.311/0.322
0.692/0.711
0.382/0.392
0,461/0:491
1.5 (Vicune-r1.5-7B)
gond-apoor
0,463014499
0.443/0467
0.30550.321
0.345/0 358
0.32120.333
0.672/0.758
0.41770.440
0:42440:445
fine+bod
0,45301469
D.45700.1822
0.258/0.288
0.303/0.338
0.29410.302
0.558/0.617
0.389/0.400
0.388/0.416
highe-iow
0,47480:176
0,370/0.866
0,261/0.262
0.13200429
0 0,26600269
0.669/0.716
0.301/0331
0.397/0.110
0.49180.491
0.416/0.436
0,293/0.300
0.6 0.696/0.751
0.413/0.416
0.298/0.304
0,359/0.389
0.424/0.441
good finee sho
0,482/0.482
4.461/0.485
0.3000.320
0. 0.644/0.708
0.3390.3357
0.327/0.336
0.425/0.451
0.425/0.449
good high siighefnee-poorssoooo
0,512005513
4.44300465
0,30380315
0.408/0415
0L318/0.324
0.697/07722
0.392/0.421
0.439/0458
mPLUG-OW (LLaMA-7D)
goud--ppor
0,409K1,427
6.6340.644
0.24180.271
0,457/0.487
0,14880.180
0.687/0.711
0,46690.486
0.432/0.455
Bonetboo
0.357X0.398
0.622/0.636
0,260/02900
0:422/0:155
0,178/0.224
0.606/0.616
0.536/0.534
0.426/0.155
highe-low
0.353/0.369
0.610/0.224
0,176/0.187
0.436/0.464
1.1
0.662/0.663
0,361/00378
0.387/0.401
good-high--poorlon
0,382/0.402
0.626/0.642
0.208/0.228
0.446/0.483
0.125/0.144
0.684/0.697
0.409/0.432
0.41 1/0.422
good fine- spoorsbod bad
0,403/0440
6.635/6665
0.260802292
0.446/0.493
0.17280.213
0.664/10094
0.525/0.527
0.443/0.41
0,395/0,421
666300667
0,23880228
0:455/0496
0.14770.175
0.68500704
0.463/0.43
0,43000.555
Owen VL (QwenI.d)
goud--poor
0,77000.44
0667606669
0.29800.339
0.50405552 532
0.27300.844
0.617/0.686
0.186600-456
0.475/0.506
fine+bbdd
0.467/0.507
0.352/0.365
0,205/0.238
0.451/0:472
0.188/0.185
0,599/0.627
0.354/0.378
0.374/0.396
highe-low
0,531/0.578
0.626/06666
0,281/0.290
0.574/0.560
0,286/03114
0.637/0.692
0.332/0.344
0.467/0.485
good hight pporrlow
0.539/06600
0.68406673
0.299/0.324
0.565/0.568
0.30600130
0.6600.721
0.414/0.422
0.495/0.520
good flnne poan--ppoorrtee
0,49580.558
0.5960.581
0.26480.007
0.521/0:548
0.27000.70
0.64006691
0.435/0.449
0,4000 486
gomffthighhfinee www.eiighiiiii--poooooeeeeee booned
0,541X0.600
6.63206177
0,286803116
0.5700.577
0,301X03118
0.664/0.719
0,416/0.429
0,487/0.511
InternL Comppsssrr VL (Iutern[.M)
good-spoor
0.564K0.615
0.7300.750
0,36000.116
0.612/0.76
0.221330.265
0.732/0.7775
0,516/0.572
0.341/0.551
fonetsbad
0,546/0.597
0.7200.736
0.341/0.389
0.626/0.671
0,213/0.227
0.681/0.708
0,494/0.479
0.517/0.544
highe low
0.543/0590
0.70400.220
0.331/0.372
0.612/0.656
0,2230.251
0.716/0.755
0.490/0.500
0.517/0.549
good hight poor vow
0.56483.613
0.723/0743
0.38490.405
0.621/0.666
0.238/0.244
0.73480.775
0.522/0.546
0.537/0.575
0,573/0.636
0,73500755
0,36680.420
0.62906687
0.23600260
0.732007771
0.531/0.551
0.5430.581
0.571K1.621
0.728/0.448
0,36000.10
0.629/0683
0.236/0.61
0,734/0.773
0,521/0.538
0,54000.576
A.3.5 Synonym Ensemble: FURTHER IMPROVING IQA ABILITY (A3) FOR MLLMS
As shown in Table I1 the synonym ensemble strategy (as proposed in Eq. on top-5 methods (i.e.
X-Coolose--p--------- VL, QWen-VLL LLaVA-v1.55 (13B). mPLUG-OWl, and LLaVA-v1.5 (7B)) can in average lead to up to 2% accuracy improvement (in average 1.3%). We believe it is a useful boost improve the performance of MI.LMs IQA task.
Nevertheless, we also notice that different MLLMs perform best with different specific prompt combos. For example, the good+ffine++poo-bbrr performs best on MiernLM-XCooposer but comes with reduced accuracy on QWen-VL compared with only good+xpoor While good+>poor is proved overall best single word pair for the evaluation and shows stable results across MLLMs, we decide to keep the current strategy in Q-Bench to evaluate MLLMs.
B STATEMENT ON DATA CONTAMINATION
The Q-bench contains three tasks, where the first two tasks, (A1) perception and (A2) description, are evaluated with our own datasets proposed with the paper. For these two tasks, the questions, answers, or low-level descriptions in the two datasets are not seen by any existing MLLMs. Half of LLVisionQA (i.e. the test subset) and full of LLDescribe labels are kept private, to avoid being added to the training sets of any MLLMs. We hope that this measure will allow Q-Bench to have long-term significance as an indicator of low-level visual abilities.
For the third task. (A3) assessment, the situation is a bit more complicated. For open-source models as tested, almost all of them have provided their technical reports, and as far as we know, no image quality assessment (IQA) dataset has participated in the multi-modality training stages of them.
26

---

Published as a conference paper at ICLR 2024
While text knowledge about image quality assessment should have been injected them (e.g. blurry image low quality image) during their pure-language training stages, we think this should not be regarded data contamination for IQA, because the images cannot be by language model. Instead, they are important knowledge for MILMs to better link particular visual attributes (blur) human opinions (quality), which motivates explore MLLMs for these tasks.
C LIMITATIONS AND DISCUSSIONS
observed that MILMS frequently respond with "yes" to Yes-or-No questions. It's worth noting that the current LLVisionQA dataset is skewed, with 62% of its questions being Yes-questions and only 38% being No-questions. This imbalance could introduce biases when com- paring various MLLMs. To fully address this, we aim to balance the dataset by preparing a reversed version for each question in our subsequent work, ensuring a less biased evaluation.
In Section A.3.2
For the description task, we acknowledge that judging whether a description matches the gold description subjective process, which may not have an absolute standard. Even when evaluated by humans, the scores rated for the MILM descriptions are subject to individual differences. Though have employed the 5-round GPT-assisted evaluation protocol, which could be the most reliable and reproducible way at present, it may still unavoidably contain hallucinations (from GPT), We will continue to explore how to design more reliable evaluation protocol for the low-level visual description task in our follow-up works.
While the proposed Q-Bench has offered comprehensive evaluation on the low-level visual capa- bilities of MLLMs, it does not provide direct guidance enhancing these capabilities. As our next steps, we intend to progressively scale up the LLDescribe and LLVisionnA datasets to eventually allow reliable low-level visual instruction tuning process that further improve the low-level abilities for MLLMs.
27


Reviews:
[{"review_id": "15nKpcXvUB", "reviewer": "Reviewer_gz7s", "summary": "This work introduces a benchmark for low-level perception and understanding. They introduce three tasks for assessing the MLLM. The main focus is low-level, thus they build an evaluation bench from various aspects (low-level attributes, visual distortions, etc.). The experiments are extensive and comprehensive.", "questions": "As questions in weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is sufficient, and this benchmark is specifically designed for low-level tasks rather than a holistic evaluation of general abilities.\n2. The tasks consist of classification and description, as well as probability-based quantitative evaluation, making it well-organized for evaluating the low-level abilities of the current MLLM.\n3. The experiments are extensive and comprehensive, which are helpful in diagnosing the strengths and shortcomings of current MLLMs.", "weaknesses": "The main concern is about the trustworthiness of the evaluation.\n\n1. As the ChatGPT also suffers from hallucination, how can we ensure the reliability and confidence of the GPT-assisted evaluation?\n2. What is the difference between the proposed metric and perplexity (only considering good/bad)? Is the PPL equivalent to this new metric?\n3. As current models typically leverage large amounts of data, how can we avoid contamination of the evaluation dataset in the training set?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2RYXJp5YRH", "reviewer": "Reviewer_1bay", "summary": "The rapidly developed Multi-modality Large Language Models (MLLMs) have progressed greatly. However, there is still an inadequacy in assessing the abilities of MLLMs on low-level visual perception and understanding. Accordingly, this paper proposed Q-Bench, a holistic benchmark of MLLMs for low-level visual perception, low-level visual description, and overall visual quality assessment. Q-Bench constructs solid benchmark datasets and reasonable evaluation pipelines, including LLVisionQA for low-level perception ability, LLDescribe for low-level description ability, and a compounded IQA dataset for image quality assessment with an innovative softmax-based strategy. Q-Bench provides a viable solution to reveal the potential of MLLMs for low-level visual understanding.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Q-Bench is composed of benchmark datasets and reasonable evaluation pipelines for low-level perception, low-level description, and image quality assessment. The contributions are solid. \n2. The experiments and study are extensive and convincing. \n3. The Q-Bench is well presented, and the details are clear.", "weaknesses": "1. In the evaluation of LLDescribe, the softmax is calculated between good and poor. How about also considering their synonyms, e.g., great, excellent, fine, bad, low, etc? Here is a possible solution: (1) merge the logits of great, excellent, fine into good, and bad, low into poor. (2) calculate the final score with merged logits.\n2. As a benchmark paper, the author may update the recent SOTA MLLMs into the leadboards of this paper, e.g., QWen-VL [1], InternLM-XComposer [2], and LLaVA-1.5 [3]. \n\n[1] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond\n[2] InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition\n[3] Improved Baselines with Visual Instruction Tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dXFHxuKQsB", "reviewer": "Reviewer_rhgp", "summary": "This is a benchmark paper on the performance of low-level visual perception and understanding of MLLMs. To this end, the paper collected and annotated a new benchmark dataset Q-Bench including 1) LLVisionQA about the low-level perception of 2990 images; 2) LLDescribe about the description of image quality of 499 images; 3) how to align visual quality scores with people perception. The paper evaluated 10 recent public available MLLM on Q-Bench. The evaluation indicates current MLLMs have decent low-level abilities yet still a long way to go for general low-level visual assessment.", "questions": "It is definitely a must-to-ask question that how the performance of GPT-4-Vision on Q-Bench, which shall establish the baseline for commercial SotA MLLM.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The benchmark of current MLLMs’ abilities to access low-level image quality appears a quite interesting topic to me. The paper presented a thorough and in-depth evaluation and the finding may inspire some further research. The evaluation design of Q-Bench makes sense. The paper is well-written.", "weaknesses": "This is a descent benchmark paper on an interesting topic. I would recommend acceptance given the performance of GPT-4V on Q-Bench is provided in the revised version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)