T2I-Scorer: Quantitative Evaluation on Text-to-Image Generation via Fine-Tuned Large Multi-Modal Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 15:30:55

Abstract: Text-to-image (T2I) generation is a pivotal and core interest within the realm of AI content generation. Amid the swift advancements of both open-source (such as Stable Diffusion) and proprietary (for example, DALLE, MidJourney) T2I models, there is a notable absence of a comprehensive and robust quantitative framework for evaluating their output quality. Traditional methods of quality assessment overlook the textual prompts when judging images; meanwhile, the advent of large multi-modal models (LMMs) introduces the capability to incorporate text prompts in evaluations, yet the challenge of fine-tuning these models for precise T2I quality assessment remains unresolved. In our study, we introduce the T2I-Scorer, a novel two-stage training methodology aimed at fine-tuning LMMs for T2I evaluation. For the first stage, we collect 397K GPT-4V-labeled question-answer pairs related to T2I evaluation. Termed as T2I-ITD, the pseudo-labeled dataset is analyzed and examined by human, and used for instruction tuning to improve the LMM's low-level quality perception. The first stage model, T2I-Scorer-IT, has reached superior accuracy on T2I evaluation than all kinds of existing T2I metrics under zero-shot settings. For the second stage, we define an explicit multi-task training scheme to further align the LMM with human opinion scores, and the fine-tuned T2I-Scorer can reach state-of-the-art accuracy on both image quality and image-text alignment perspectives with significant improvements. We anticipate the proposed metrics can serve as a reliable metric to gauge the ability of T2I generation models in the future. We will make code, data, and weights publicly available.

Main paper: 
 related works 2.1 text-to-image (t2i) generation: 
 Recent advancement of Text-to-Image (T2I) generation has been pre-dominated by diffusion models, which have shown better image quality and image-text alignment than GAN-based generators [49] .As pioneer by latent diffusion [46] , a typical diffusion-based T2I generation model [45, 48] consists of a VAE [20] encoder/decoder to convert images to/from latent spaces, a U-Net (or a transformer for most recent studies [11, 41] ) to denoise latent inputs, and a CLIP or T5-XXL [44] text encoder to allow text control on the synthesized images. In short, state-of-the-art T2I generation models [8, 10, 17, 34, 42] have shown excellent abilities to synthesize images given a wide range of text prompts. Nevertheless, the abilities of current T2I generators still vary a lot under different text prompts and face failure cases (either low image quality or poor prompt following), calling for a robust and accurate objective metric for T2I generation.Large Multi-Modal models (LMMs) are a novel type of Multi-Modality foundation models that generate text responses from visual and text inputs. As represented by LLaVA [31] , InstructBLIP [7] , mPLUG-Owl [69] , LMMs typically include a visual encoder (such as CLIP [43] , DINO [40] , or SAM [21] ), a text-only large language model (LLM) (e.g LLaMA [52] ), and a connector between the two parts, which could be either a shallow multi-layer perception (MLP) or a deeper cross-attention abstractor [29, 69] . LMMs not only show superior ability on visual question answering benchmarks [2, 15, 35] , but also show exciting abilities on understanding and following diverse user instructions [13, 31, 32, 72] , and even seamlessly dialog with human about the visual inputs. In this work, we utilize the strong ability of LMMs to serve as backbones for the proposed T2I-Scorer.IQA Methods. Image quality assessment (IQA) is a traditional domain for multi-media studies. Traditional IQA methods, such as SSIM [55] , NIQE [37] , and ILNIQE [73] typically rely on statistical features to evaluate the quality of images and prove their effectiveness on artificial distortions (compression artifacts , white noise, gaussian blur, etc) [24, 25] . Nevertheless, these methods usually do not work well on in-the-wild quality evaluation scenarios, as they are not aware of any visual semantics. On the contrary, deeplearning-based IQA methods [19, 27, 51, 74, 75] usually achieve better performance on these in-the-wild datasets. Among them, some methods [53, 59, 76 ] have adopted CLIP, by fine-tuning the softmax-pooled cosine similarity between images and text-defined levels (e.g. good, poor) as the quality score for images.IQA on T2I Images. Recently, with the advancement of T2I generation, several subjective studies have collected IQA datasets [5, 26, 54, 77] for T2I-generated images. There are also several weaklylabeled human opinion databases [67, 68] LMMs for IQA. Given the strong ability of LMMs, several pioneer studies have investigated LMMs for IQA. Several benchmarks [60, 66, 79, 81] have proved decent zero-shot quality assessment as well as related perception abilities, which is still notably behind GPT-4V and has plenty of room for improvement. Given this insight, several studies [18, 62, 71] have collected questionanswering datasets to improve quality perception and evaluation abilities of LMMs, including not only single-image settings but also multi-image comparative settings [64] . Despite training LMMs for direct text outputs, a most recent study, Q-Align [63] , also proposes a human-alike strategy for LMMs to output precise quantitative scores, reaching state-of-the-arts on traditional IQA datasets. Inspired from these explorations, we design a two-stage training scheme to fine-tune an LMM into the proposed T2I-Scorer. 
the t2i-itd dataset: 
 In this section, we elaborate on the Text-to-Image Instructional Tuning Dataset (T2I-ITD). We first introduce the image preparation (Sec. 3.1), and then discuss prompts for GPT-4V to generate qualityrelated question-answering data (Sec. 3.2) on both single images and image pairs, resulting in 397K question-answer pairs. Finally, we analyze the reliability of the GPT-4V-annotations (Sec. 3.3).We include the images from the recently-released database with 20K images by Li et al.. The images are generated from 15 different T2I models [4, 8, 10, 16, 17, 34, 42, 45, 47, 48, 50] , generated via diverse hyper-parameter configurations such as iterations, Classifier Free Guidance (CFG), resolution, aspect ratios. It excludes outdated Generative Adversarial Network (GAN) [49] and Auto-Regressive (AR) models. The prompts used to synthesize these images are sourced from real user inputs of the AIGC community, selected from Diffu-sionDB, undergoing filtering (removing not-suitable-for-workplace prompts & prompts with special characters) to ensure content diversity and appropriateness, resulting in 20K prompts for model inputs (each image is with different prompt), final yielding a large-scale and diverse database for the following GPT-4V pseudo labeling process. 3.2.1 General Schemes. Being used for the first training stage for the quantitative T2I-Scorer, the core principle of the T2I-ITD dataset is to make LMMs provide easy to quantify outputs. Henceforth, the answers for all question-answering data are designed to be distinguishable from the first word: each answer is either limited to one single word (e.g. Yes/No), or has more than one words but the first word is different for all possible answers (e.g. First image/Second image/Tie). This has allowed us to directly convert the logits for candidates in the first token, and further convert them into quantitative scores [60, 78] . Despite that, given existing observations [28, 57] that image contents notably affect both subjective and objective quality evaluations, we include the text prompts into the instructions for LMM fine-tuning. The instruction format for T2I-ITD is as follows, explicitly asking for the simplest answer: Single Images: This is an image from the text-to-image generation, by the text prompt: <prompt>. Please reply with the simplest answer: <question> Image Pairs: This is a pair of images from text-to-image generation. The prompt for the first image is <prompt1>, and the prompt for the second image is <prompt2>. Please reply with the simplest answer: <question>We introduce the details of different question-answering subsets for single images (Sec. 3.2.2) and pairs (Sec. 3.2.3) as follows.Image Question-Answering. The single image questionanswering setting is the most common setting for LMMs. For single images, to facilitate the task for quantitative scoring, we collect two types of questions where answers could be converted into numerical values: Yes-or-No questions and How questions, as follows:Yes-or-No questions contain binary judgements related to quality, e.g. Is the lighting of the image good?, with answer limited to [Yes, No] . The answers could map to numerical values as follows:𝑓 YN : {Yes, No} → {0, 1}, where 𝑓 (Yes) = 1, 𝑓 (No) = 0 (1)Being more 'fine-grained' than Yes-or-No questions, How questions contain multi-level evaluation related to quality, e.g. How is the The examples of GPT-4V generated pairs for Yes-No and How questions on single images are illustrated in Fig. 3 (a) and (b).Answering. Despite single image settings, many recent studies [66, 79] have reported GPT-4V's stronger quality-related perception ability on image pairs than single images. Henceforth, to better teach open-source LMMs, we similarly collect two types of question-answering data on pairwise visual inputs, including the Yes-or-No questions, which are generally the same with their counterparts for single images, only difference as comparative targets, e.g. Does the second image appear to have better clarity?, and an unique type of questions, the Which questions.The Which questions raise a query related to quality, e.g. Which image has better texture details?, and the answer could only be chosen from First image/Second image/Tie. Similar to How questions, the answers of Which questions can also be converted into relative numerical scores (from perspective of first image):𝑓 Which : {First, Tie, Second} → {0, 1, 2}, where 𝑓 Which (First) = 2, 𝑓 Which (Tie) = 1, 𝑓 Which (Second) = 0 (3)The examples of GPT-4V generated pairs for Yes-No and Which questions on image pairs are illustrated in Fig. 3 (c ) and (d).As the T2I-ITD is labeled by GPT-4V instead of human annotators, it is important to analyze its composition and reliability before using it for training. For image composition, after GPT-4V labelling, we deliberately remove all data including images synthesized by DALLE-2 or DALLE-3 (5% of all images), which is reserved to examine the cross-model generalization of T2I-Scorer-IT. We further discuss the composition of questions, the proportions of answers, and the reliability of the collected data as follows.3.3.1 Composition of Questions. The 397K question-answer pairs are generally evenly composed of four question types. They include 97K Yes-or-No and 79K How questions on single images, as well as 116K Yes-or-No and 111K Which questions on image pairs. About the quality-related concerns, we use Wordcloud [38] to compute the frequencies of quality-related words in questions, and group the synonyms (e.g. clarity, clear, clearly) by GPT-4. Among all questions, the clarity dimension has the highest frequency (16.8%), followed by lighting (14.2%), color (10.3%), composition (6.7%), and artifacts (4.7%), with similar proportions to human quality descriptions in Q-Pathway [62] . Despite questions about whole images, there are also 7.4% questions asking about the main object, and 5.1% about the background, contributing to local in-context low-level perception [28, 60] . In general, these GPT-4V-raised questions have widely covered a rich variety of quality-related concerns.While the questions are evenly distributed with covering a rich variety of quality-related concerns, we observe that the answers are pretty "biased" towards positive responses (i.e., Yes and Good). For Yes-or-No questions, the answer Yes makes up 69.3% of all answers; for How questions, Good even shares 81.1% among all answers. Nevertheless, the answers on Which questions are pretty balanced, with 44.1% for Second Image, 41.5% for First Image, and 14.4% for Tie. Therefore, such positive "bias" might come from the overall high quality of input T2I-generated images. To better understand the origin of the bias, we examine the data reliability of T2I-ITD, discussed as follows.To analyze the reliability of the T2I-ITD, we randomly sample 1500 questions from the whole database and ask 7 human experts to examine its correctness. Without seeing the GPT-4V answers, human experts independently choose one among all available answers for the question, and then we use a majority voting to determine the human label (𝐿 human ) of each sample. For question types with three choices (i.e. How&Which), if no choices get > 3 votes, 𝐿 human for this case will be regarded as 'Divergent'; for Yes-or-No questions, a 4:3 vote is considered 'Divergent'. Denoting GPT-4V answer as 𝐿 GPT-4V , the results of human evaluations on different question types are listed in Tab. 1, which primarily validates the reliability of the T2I-ITD dataset. Despite general reliability, the human examination also comes with several conclusions: (1) Positive answers (Yes and Good) are only slightly less accurate than others, suggesting that the distribution of 𝐿 GPT-4V mainly comes from overall acceptable quality of T2I-generated images; (2) GPT-4V answers on image pairs are slightly more accurate than single images, aligning with existing observations [79] . (3) Ignoring the human-'Divergent' samples, the GPT-4V answers can reach >85% agreement rate with human, proving them sufficient to serve as training data for the first stage. 
the t2i-scorer: 
 In this section, we discuss the structure and training of T2I-Scorer-IT and T2I-Scorer (Fig. 4 ), a family of LMMs with improved ability on T2I evaluation. For the choice of LMM, we adopt the mPLUG-Owl2 [70] structure. The LMM (denoted as M) includes a CLIP-ViT-L14 [43] visual encoder E 𝑣 with 304M parameters, a visual abstractor Ê𝑣 with 82M parameters, and a LLaMA2-7B [52] LLM (denoted as L). Denote input images as I, previous text tokens as {𝑡 0 , . . . , 𝑡 𝑁 -1 }, the LMM autoregressively predict the logits for the 𝑁 -th text token (O 𝑁 -1 ) as follows:EQUATIONwhere E 𝑡 is the text embedding layer, and H 𝑣 is the visual embedding, concatenated (⊕) with E 𝑡 outputs and fed to LLMs.The training scheme is conducted in two stages. In the first stage, we utilize the collected T2I-ITD to fine-tune the mPLUG-Owl2 into the T2I-Scorer-IT (Sec. 4.1), which is not only able to provide Eq. ( 6)&( 7)(Inference) (Training)with output text Eq. ( 5) During inference, we introduce the logit-based Soft Scoring Strategy (defined in Eq. 6&7) for quantitative evaluation.Predicted Scoreanswers on quality-related questions but also provide quantitative evaluations. Afterwards, we further fine-tune the T2I-Scorer-IT with converted human opinion scores, into the T2I-Scorer (Sec. 4.2), which can provide more accurate multi-perspective T2I evaluation.4.1.1 Training Scheme. We train the T2I-Scorer-IT under the general supervised fine-tuning (SFT) [31, 80] scheme for LMMs. Specifically, denote the instruction text as T in , the SFT loss L only supervises on the answer text {𝑡 0 out , 𝑡 1 out , . . . } as follows:EQUATIONi.e. next token prediction loss on all tokens that are within the answer part of the conversation. Furthermore, as we have in total 397K question-answer pairs with very short total length, to best utilize training, we group up to three rounds of question answering for the same image(s) into one data item, resulting in 162K data for the first training stage. With grouped data, the training time cost is reduced from 7 hours to 3 hours on 8*A100 GPUs.Strategy for Quantitative Evaluation. Primarily, the T2I-Scorer-IT is able to provide answers on open-vocabulary questions about T2I evaluation. Additionally, given that the answers can be converted to numerical levels (Eq. 1, 2, 3), the T2I-Scorer-IT can also predict quantitative scores for generated images.For quantitative evaluation, a trivial strategy is to directly collect 𝑓 YN (T out ) for an image with general Yes-or-No questions, e.g. 'Is the image with good quality?', or 𝑓 How (T out ) on general How questions, e.g. 'How is the image generated?'. However, such scores can only provide finite levels without enough precision. On the other hand, as mentioned in Sec. 3.2.1, as the answers in T2I-ITD are distinguishable from the first word, we can get the probabilities of each answer via softmax from the logits on first output token (O 0 out ):EQUATION)where C is the candidate answer set (i.e. {Yes,No} for Yes-or-No questions, {Good,Fair,Poor} for How questions). With the probabilities, we obtain 𝑠 pred via the soft scoring strategy, as follows:EQUATION)where 𝑓 is the general representation of the mappings in Eq. 1, 2, 3.With the soft scoring strategy, the T2I-Scorer-IT is able to predict real quantitative scores for T2I generation, which is proven more accurate than the directly mapped scores as well as existing T2I metrics. It also shows high consistency between Yes-or-Noderived or How-derived scores, suggesting the effectiveness of the first stage training. We evaluate its quantitative ability in Sec. 5.4.The T2I-Scorer 4.2.1 Data Conversion from Human Opinion Scores. For the second stage (fine-tuning), we would like to further train LMMs to provide subjective-aligned evaluations. Henceforth, we fine-tune the T2I-Scorer-IT with AIGIQA-20K [23] dataset. Following ITU [1] standards, we similarly convert the original scores (in range [0,5]) into 5-point likert scales: Bad,Poor,Fair,Good,Excellent.The conversion is formulated as follows: EQUATIONwhere 𝑠 gt is the original human opinion score. This conversion has allowed unified training objectives (as in Eq. 5) with the first training stage, and the same quantitative evaluation strategy (Eq. 7) as T2I-Scorer-IT. The training data template is defined as follows.  
experiments 5.1 experimental setups: 
 We initialize the T2I-Scorer with the pre-trained checkpoint of mPLUG-Owl2 [70] . Before feeding to the LMM, images are first padded to square and then resized to 448 × 448. We use 8×NVIDIA A100 80G GPUs for training, with DeepSpeed ZeRO-3 optimization. We only use the final checkpoints for evaluation instead of picking checkpoints by validation set performance. All parameters of the LMM are updated during training.We adopt the two most popular T2I evaluation datasets, AIGIQA-20K and AGIQA-3K, with fine-grained labels as evaluation datasets for T2I-Scorer-IT and T2I-Scorer. For AIGIQA-20K, 19K (DALLE-2&DALLE-3 images excluded) images are used for first stage training (with T2I-ITD), and 17K labeled images (further excluded Dream-Gaussian [10]) are used for fine-tuning with human opinion scores (stage 2). Henceforth, we evaluate T2I-Scorer-IT on all images (i.e. blind to labels) as well as the DALLE-2/DALLE-3 subset (i.e. completely blind); for the fine-tuned T2I-Scorer, we evaluate it on the 3K test set non-overlapped with the 17K training labeled images. Compare to normal random train-test splits, the split-by-generationmodel strategy helps us to reach more reliable conclusions on how the metric can be applied to evaluate new T2I generation models in the future. For AGIQA-3K, neither its images nor its labels are used during training. We use it to evaluate the cross-set generalization ability of the proposed metrics. Following [54] , we average the quality and alignment scores as an additional overall perspective, evaluated via human user study.We include a wide variety of baseline models for comparison: General IQA Methods. We include representative IQA methods in different categories as baseline models:• Statistical IQA Methods: NIQE [37] and BRISQUE [36] .These methods are not trained on any IQA datasets. • Deep-learning-based IQA Methods: including pure visual IQA methods NIMA [51] , DBCNN [74] and MUSIQ [19] , as well as CLIP-based IQA methods CLIP-IQA [53] and LIQE [76] .We compare T2I-Scorer-IT with their pre-trained models on general IQA datasets, and compare T2I-Scorer with their fine-tuned version on AIGIQA-20K [23] under the same traintest splits (17K training set, 3K test set, cross-model).Specialized T2I Metrics. Despite comparison with general IQA methods, we further compare the T2I-Scorer-IT with two popular specialized T2I metrics: ImageReward [68] and HPS [67] . These two T2I metrics are trained with human preference data and do not support further fine-tuning with opinion scores, so we compare their official weights with T2I-Scorer on alignment perspective.Zero-shot LMMs. To validate the improvements of the first tage, we further compare the T2I-Scorer-IT with representative zeroshot LMMs: LLaVA-v1.5-13B [30] and mPLUG-Owl2 [70] (our base model). Zero-shot LMMs are tested with their optimal settings [61] . In Tab. 2, we compare the proposed T2I-Scorer-IT with different kinds of existing metrics for T2I evaluation. For the proposed T2I-Scorer-IT, it has reached state-of-the-art zero-shot performance on all three evaluation settings: it has more than 10% improvement than any existing T2I metrics. Furthermore, we notice that existing models usually have some flaws: statistical IQA methods can bearly evaluate T2I generation; deep-learning-based IQA methods experience notable performance drop on many generative models (on AIGIQA-20K (all)), and the specific T2I metrics instead fall short on evaluations within a few models (on AIGIQA-20K (DALLE-3)). Consequently, these existing metrics might face challenges to simultaneously accurately compare across models and evaluate on individual generated images, while the proposed T2I-Scorer-IT can handle both scenarios better than any existing approaches. Despite peer comparison, we further reach several important observations about the T2I-Scorer-IT: 1) It shows similar accuracy on unseen images (DALLE-3 subset) in comparison to images with pseudo labels during training, proving the first training stage can learn general knowledge about T2I evaluation; 2) Within Yes-or-No questions, positive prompting (i.e. good images receive Yes) in general shows higher accuracy than negative prompting, which may suggest the inductive bias from its training data that tends to ask questions in a positive manner. 3) The performance of Yes-or-No questions and How questions are generally on par, showing that its evaluation ability is consistently elevated across question types.Despite main results, we further qualitatively analyze pairwise evaluation ability of T2I-Scorer-IT in supplementary materials.In this section, we evaluate the fine-tuned T2I-Scorer on multiperspective T2I evaluation, as shown in Tab. 3 (image quality) and Tab. 4 (image-text alignment), as follows.Image Quality. As shown in Tab. 3, the proposed T2I-Scorer is notably superior than existing IQA approaches on this setting. The improvements are especially significant (leading all existing IQA methods by more than 10%) on the test set of AIGIQA-20K, which contains only images generated by T2I models not included in the training set. This setting is meaningful as it measures whether the evaluator can robustly evaluate T2I-generated images in the future instead of over-fitting on the appearances of current T2I models, while the T2I-Scorer has proven its competitiveness on this meaningful setting. Additionally, T2I-Scorer also shows notable improvements upon base LMMs and the pre-traned stage 1 model (T2I-Scorer-IT) on both cross-model (AIGIQA-20K (test)) and cross-dataset (AGIQA-3K) evaluations, proving that our stage-2 fine-tuning is effective on image quality perspective. Image-Text Alignment. Compared with the image quality perspective, the image-text alignment perspective marks a notably more difficult scenario: while T2I-Scorer is able to outperform all existing similarity-based T2I metrics as well as the baseline LMMs, none of the method achieves over 0.8 SRCC on this perspective, suggesting that there is still plenty of room for improvements to more accurately evaluate the image-text alignment perspective of T2I generation. Among existing metrics, we notice that T2I-Scorer has more significant improvements than ImageReward on AIGIQA-20K test set, which has a notable longer average prompt length (3 times as long as AGIQA-3K), suggesting the proposed LMM-based metric can better understand more complex text prompts. As similar improvements are also observed on baseline LMMs in comparison with CLIP, this effect proves our aforementioned claim that LMMs may better understand complex prompts in T2I generation. Effects of scaling up T2I-ITD. In Fig. 5 , we illustrate the accuracy change of T2I-Scorer-IT with different amount of T2I-ITD data used in first training stage. We notice that scaling up the T2I-ITD dataset consistently improve the accuracy of the T2I-Scorer-IT, which is not even saturated with the whole T2I-ITD used. The results have demonstrated that the pseudo-data training is not only useful, but also potentially further scalable to larger amount of data; on the other hand, the results also by-side validate that existing LMMs are still not sufficiently pre-trained for T2I evaluation.Effects of T2I-ITD on fine-tuned results. While Tab. 2 has shown the significant direct improvement of the first stage training with T2I-ITD dataset, in Tab. 5, we further discuss its contributions to the fine-tuned results of T2I-Scorer. As shown in the table, the first stage training not only notably boosts the results on the image quality perspective as expected, but also slightly improves the image-text alignment perspective which is not the direct objective of the first stage training, which might be because prompts are included in the instruction template for the first stage training.Effects of explicit multi-task tuning. In Tab. 5, we discuss the effects of the explicit multi-task fine-tuning scheme (as defined in Sec. 4.2), by comparing with the variant with implicit questions How do you rate the quality of the image for dimension i (i=1,2,3)? for the perspectives. As shown in the table, explicitly asking questions will notably improve performance for both perspectives. This suggests that the proposed explicit multi-task training can better  
conclusion: 
 In this work, we have proposed T2I-Scorer, the LMM-based evaluator for T2I (text-to-image) generation. It is trained by the T2I-ITD dataset, the GPT-4V-pseudo-labeled dataset with 397K questionanswering pairs, and then further trained under an explicit multitasking training scheme to align with human-annotated opinion scores. The proposed T2I-Scorer-IT (pre-trained evaluator) and T2I-Scorer (fine-tuned evaluator) both achieve state-of-the-art accruraies under their respective settings. Furthermore, our evaluation is especially conducted on images across generation models or across different databases, demonstrating the generalized effectiveness of the proposed metric, and its eligibility to evaluate novel T2I generation models in the future. In the future works, we aim to explore how to further improve the image-text alignment ability on current evaluators, so as to fully unlock the strong text modeling capacity of LMMs for more holistic evaluation on T2I generation.  
: 
 only generate high-quality and aesthetic images, but also generate images that follow specific text depictions (i.e. prompts).Nevertheless, there still lacks a reliable objective system to evaluate the quality of T2I generation. On the one hand, image quality assessment (IQA) methods [3, 19, 74] only take the generated images as inputs, without consideration of their alignment with the text prompts. On the other hand, vision-language similaritybased metrics [29, 43, 67, 68] are usually not enough sensitive to traditional low-level quality issues (color, clarity, brightness, etc). Moreover, models based on CLIP [43] have proven to be inferior in understanding sentence-level complex semantics, making them also sub-optimal to sufficiently evaluate image-text alignment of T2I generation. Given these limitations of existing metrics on evaluating T2I generation, the majority of recent T2I generation models still use human studies to evaluate their abilities. The community is in need of a more comprehensive metric to evaluate T2I generation.In this study, we propose a better T2I evaluator, T2I-Scorer, via discovering through the pivotal question:What abilities should a more reliable T2I evaluator possess? First, can understand text prompts comprehensively, which is inalienable on evaluating image-text alignment (Fig. 1(a) ). While CLIP-based models typically fall short on it, recently emerging large Multi-Modal models (LMMs), as represented by GPT-4V [39] and Gemini [14] , as well as excellent open-source counterparts [9, 30, 31, 70] , have shown stronger ability on understanding more comprehensive text inputs or instructions [32, 72] . Though with fundamental visual and text understanding abilities, existing studies find zero-shot LMM evaluations [6, 22, 33, 65] only weakly correlate with human opinions. Henceforth, for the proposed T2I-Scorer, we propose to fine-tune an LMM for a better quantitative T2I evaluator.Second, can effectively perceive both high-level (content) and low-level (quality) visual attributes, which is important for better fidelity assessment [26] on T2I generation. While opensource LMMs can generally perceive well on high-level objects and themes of images, several benchmark studies have pointed out that they still have a significant gap with GPT-4V in terms of low-level visual perception, especially while comparing multiple images [61, 64, 66, 79] . To improve open-source LMMs, we employ GPT-4V to collect 397K low-level-related question-answering pairs on 20K T2Igenerated images, denoted as T2I-ITD (Text-to-Image Instructional Tuning Dataset). The T2I-ITD includes not only questions on single images, e.g. How is the clarity of the image?, but also on image pairs, e.g. Which image is brighter?, to better improve LMMs on quality-related perception. The T2I-ITD dataset is used for the first stage instruction training, yielding the T2I-Scorer-IT, an LMM capable of answering open-vocabulary questions on T2I-generated images. Additionally, as the answers in T2I-ITD are designed to be convertible to numerical levels (e.g. 1,2,3), the T2I-Scorer-IT can also provide quantitative evaluations on T2I generation, which is proven more accurate than existing metrics. Third, can be explicitly aware of different evaluation criteria for different perspectives. While most existing image or video quality assessment approaches [12, 19, 56, 58, 75] are based on implicitly regressed scores, T2I evaluation has two different perspectives: alignment/fidelity, which might not be distinguished effectively under implicit regression. To avoid this problem, during the stage 2 fine-tuning on human opinion datasets, we convert the human opinion scores from two different perspectives into two different question templates that explicitly query the corresponding perspectives, e.g. How does the image align with the text prompt? for alignment. Then, we convert the continuous scores to ITU-standard [1] 5-point likert scales as answers.In summary, we propose the T2I-Scorer, the first LMM-based scorer for T2I generation, with contributions as follows:•  
 
 Appendix:  


Decision: This paper received mixed scores, including two "weak accept" and one "weak reject." Reviewer KVsA and Reviewer H17m were generally satisfied with the overall quality of the paper. After the rebuttal, Reviewer Uu9x lowered the score to "weak reject" but stated that they maintained the original score, without explaining the reason for continuing to insist on the negative score. Based on this, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: This work addresses the problem of quantitatively benchmarking the output of T2I models. The authors propose T2I-Scorer, an LMM-based evaluator for T2I (text-to-image) generation. To conduct their analysis, the authors collect the T2I-ITD dataset, a GPT-4V-pseudo-labeled dataset with 397K question/answering pairs, and then perform a first fine-tuning, using a general supervised fine-tuning scheme, followed by a second alignment with human-annotated scores.
strength_weakness: strength_weakness: - I sympathize with the author's idea regarding the need of having more comprehensive quantitative benchmarks for T2I generative models
- I find this study well structured and easy to read
- I think this work could inspire further exploration in this direction
limitations: limitations: - I find that the author should have better clarified the differences between their approach and other scorers, based on LMMs
- I think that the presented Data Reliability study could have been done at a more significant scale, than having 1.5k samples given to 7 human experts
- The author could expand the supplementary material with many qualitative examples, to further prove their point
paper_summary: paper_summary: This paper constructs a large-scale VQA dataset, T2I-ITD, using GPT-4V. Utilizing this dataset, the authors trained T2I-Scorer-IT and T2I-Scorer in both zero-shot and fine-tune settings, achieving impressive performance on IQA tasks.
strength_weakness: strength_weakness: - The paper constructs a large-scale synthetic VQA dataset.
- In the zero-shot setting, the Soft Scoring Strategy effectively converts the model's output logits into scores, which makes sense to me.
- The metrics look promising, including both zero-shot and fine-tuning results.
limitations: limitations: I have some concerns about the **evaluation** of T2I-Scorer-IT. The T2I-ITD dataset consists of 20k GPT-4V synthetic images, with GPT-4V using the T2I model from DALL-E 3. T2I-Scorer-IT is trained on T2I-ITD, which means it uses images generated by DALL-E 3 during training. This might cause the metrics for T2I-Scorer-IT in Table 2 to **not be truly zero-shot**, since the AIGIQA-20K dataset includes images generated by DALL-E 3.
paper_summary: paper_summary: This paper introduces T2I-Scorer, a novel framework for evaluating the quality of text-to-image (T2I) generation models. It proposes a two-stage training methodology utilizing a large pseudo-labeled dataset (T2I-ITD) and fine-tuning to align with human judgment scores. The T2I-Scorer is designed to assess both image quality and image-text alignment, aiming to serve as a comprehensive metric for future T2I models.
strength_weakness: strength_weakness: - The two-stage training process is a creative solution to enhance the evaluation capabilities of large multi-modal models
- The use of a substantial pseudo-labeled dataset (397K question-answer pairs) provides a rich source for training the model.
limitations: limitations: - 397K question-answer pairs are generated by GPT-4v. How to ensure their quality of them, especially the reliability?
- The authors mention manual testing, and the 397K tests are believed to be time-consuming and costly, and the authors need to explain this
- The model's performance may heavily depend on the quality and representativeness of the pseudo-labeled data, which might not always be reliable or generalizable.
- The paper should provide more insight into how closely the model's judgments align with human perception and whether there are any noticeable discrepancies.