AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 04:22:11

Abstract: The highly abstract nature of image aesthetics perception (IAP) poses significant challenge for current multimodal large language models (MLLMs). The lack of human-annotated multi-modality aesthetic data further exacerbates this dilemma, resulting in MLLMs falling short of aesthetics perception capabilities. To address the above challenge, we first introduce a comprehensively annotated Aesthetic Multi-Modality Instruction Tuning (AesMMIT) dataset, which serves as the footstone for building multi-modality aesthetics foundation models. Specifically, to align MLLMs with human aesthetics perception, we construct a corpus-rich aesthetic critique database with 21,904 diverse-sourced images and 88K human natural language feedbacks, which are collected via progressive questions, ranging from coarse-grained aesthetic grades to fine-grained aesthetic descriptions. To ensure that MLLMs can handle diverse queries, we further prompt GPT to refine the aesthetic critiques and assemble the large-scale aesthetic instruction tuning dataset, i.e. AesMMIT, which consists of 409K multi-typed instructions to activate stronger aesthetic capabilities. Based on the AesMMIT database, we fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert. Extensive experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision. The dataset, code and models will be made publicly available.

Main paper: 
 introduction: 
 Multimodal large language models (MLLMs) have attracted significant attention in the research community [4] . These foundation models, like GPT-4V [49] and LLaVA [34] , have demonstrated remarkable progress in serving as general-purpose visual assistants, capable of interacting and collaborating with users [46, 47] . Concurrently, their outstanding cross-modality capacities further promote a paradigm shift in the computer vision domain, with researchers seeking to transcend the limitations of traditional task-specific approaches and develop multi-modality foundation models that generalize effectively across various visual tasks [28, 34, 45] . Despite the advancements achieved, experiments on current MLLMs reveal obvious limitations in the highly-abstract image aesthetics perception task [20] , which covers not only the extensively-studied image aesthetics assessment (IAA) [50] , but also fine-grained aesthetic attribute evaluation (e.g., color, light, and composition), aesthetic emotion analysis, and image aesthetics caption [15, 23, 40] . The aesthetic perception abilities of MLLMs are crucial for a wide range of practical applications, such as smart photography, album management, photo recommendation, and image enhancement [19, 30] . Consequently, it is urgent to build a unified foundation model that possesses general abilities across these aesthetic tasks and can precisely respond to open-ended human queries on image aesthetics perception.Due to the inherently data-hungry nature, current MLLMs rely on vast amounts of multi-modality instruction-following data to meet general-purpose visual and language understanding [3] . Therefore, researchers have constructed numerous instruction fine-tuning datasets, such as COCO-VQA [2] , Science QA [35] and LLaVA-Instruct-150K [33] . However, existing instruction fine-tuning datasets are mainly engineered to enhance the general capacities of MLLMs, e.g. visual question answering [2] , image captioning [16] , object segmentation [26] and content understanding [4] . A notable gap in these datasets is the inadequate focus on visual aesthetics. To address the dilemma, we construct a comprehensively annotated Aesthetic Multi-Modality Instruction Tuning (AesMMIT) dataset, based on which we further fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert, which delivers significantly better aesthetic perception performances than the state-of-the-art MLLMs. Intuitive comparisons and examples are shown in Figure 1 . Specifically, this work encompasses three stages:Stage 1: Collecting human aesthetic feedback from subjective experiments. To bridge the gap between MLLMs and human aesthetics perception, we invite human subjects to provide direct feedback on aesthetic perception and understanding via progressive questions, including three parts: 1) The coarse-grained aesthetic evaluation (e.g. This image looks quite beautiful/unattractive. ). 2) The fine-grained reasoning and explanation based on elemental aesthetic attributes (e.g. clarity, color, light, image object and composition, etc.). 3) The finer-grained description on aesthetic feeling (e.g. novel shooting view, interesting content and expressed emotions). With the three parts, the collected human feedbacks, denoted as AesFeedback, can capture the basic aesthetic perceptions and the evaluation reasoning process. The AesFeedback dataset contains 88K human feedbacks on 21,904 multi-sourced images.Stage 2: Refining feedback with GPT for instruction-following data. The constructed AesFeedback dataset plays a crucial role in finetuning MLLMs for aesthetic instructions. However, to fully harness aesthetic capabilities, the dataset should also encompass an aesthetic question-answering component. To obtain rich questionanswer pairs, inspired by the existing works (e.g. COCO-VQA [32] and ShareGPT [6] ), we leverage GPT to transform human feedback into instruction-following formats, which include both open-end and multiple-choice question-answer pairs. To ensure that MLLMs can handle diverse queries, the instruction-following pairs cover diversified aesthetic perception dimensions (e.g. quality, attribute, emotion, interpretation, enhancement, and context reasoning), and commonly-used question types (e.g. Yes-or-No, What, How, Why, and other open-ended questions). Through the above operations, we obtain the final AesMMIT dataset, which consists of 409K multityped instructions to activate stronger aesthetic capabilities.Stage 3: Building multi-modality aesthetics foundation model based on AesMMIT. We introduce the instruction fine-tuning to improve the open-source MLLMs [33, 52] based on the AesMMIT dataset, which not only enables the models to retain their original general knowledge but also facilitates the aesthetics perception capabilities, obtaining the multi-modality Aesthetic Expert models, dubbed AesExpert.The contributions of this study are summarized as follows:• Aesthetic instruction-following dataset. We construct a corpus-rich aesthetic critique database with 21,904 diverse-sourced images and 88K human natural language feedbacks to align MLLMs with human aesthetics perception. Further, we prompt GPT to refine the human aesthetic critiques and assemble the large-scale aesthetic instruction tuning dataset (AesMMIT) to ensure that MLLMs can handle diverse queries, which consists of 409K instructions covering multiple aesthetic perception dimensions, to activate stronger aesthetic capabilities of MLLMs.• AesExpert model. We propose multi-modality aesthetic expert models with the aid of the proposed AesMMIT dataset via instruction fine-tuning. Extensive experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro.• Open-source. We release the following sources to the community: (1) the constructed aesthetic multi-modality instruction tuning dataset; (2) the proposed AesExpert models including codes and checkpoints; (3) the visual AesExpert-Chatbot demo. We believe this work would shed light on building more advanced MLLMs with comprehensive aesthetic capabilities. 
related work 2.1 multi-modality foundation models: 
 Recently, large language models (LLMs), such as GPT-3 [38] , Flan-T5 [7] and LLaMA [41] , have made remarkable progress in puretextual tasks. The success of LLMs has also driven the research on vision-language interaction, resulting in the development of various multi-modality foundation models, e.g. LLaVA [34] , MiniGPT-4 [58] , mPLUG-Owl [51] , Otter [28] and Instruct-BLIP [8] . These models typically contain a pre-trained visual encoder for image processing, an LLM for interpreting instructions and generating responses, and a cross-modality module to align the vision encoder with the LLM [3] . Despite their impressive performance in general-purpose visual tasks, their performance in handling image aesthetics perception remains underexplored. Therefore, this work is dedicated to building multi-modality aesthetics large language model. Instruction tuning is an optimization method for multi-modality foundation models, aiming to improve their ability to perform specific tasks [34] . With the development of MLLMs, researchers often employ ChatGPT [38] or GPT-4 [49] to generate diverse and expansive instruction-following data. For example, Zhu et al. [58] employed GPT-3.5 to generate and improve detailed captions for highquality instruction-following data (MiniGPT-4), which are mainly designed for general visual tasks. Liu et al. [33, 34] proposed the multimodal instruction-following dataset (LLaVA-Instruct-150K) based on the existing COCO [32] bounding box and caption dataset using GPT-4. In InstructBLIP [8] , the authors transformed 13 visionlanguage tasks (e.g. OCR-VQA [36] ) into the instruction-following format for instruction tuning. Chen et al. [6] proposed a large-scale image-text dataset featuring 100K highly descriptive captions generated by GPT-4V and 1.2M high-quality captions generated by the proposed caption model, named ShareGPT4V. A more comprehensive survey can be found in [55] .The existing instruction tuning datasets are mainly constructed for general visual tasks, and the lack of image aesthetic data limits the aesthetic perception ability of MLLMs to a large extent. Further, the existing studies suggested that hallucination has become one of the most urgent problems for current MLLMs [13, 18] , therefore, using MLLM to directly generate instruction fine-tuning data for aesthetic tasks may further exacerbate this situation. To bridge this gap, this paper presents a corpus-rich aesthetic critique database by collecting human natural language feedback, based on which we establish a comprehensively annotated aesthetic multi-modality instruction tuning dataset, i.e. AesMMIT. Further, we propose multimodality aesthetic expert models based on aesthetic instruction fine-tuning, achieving significantly better aesthetic perception performances. 
dataset construction: 
 In this section, we provide a detailed expatiation of the process involved in building the AesMMIT dataset, which is illustrated in Figure 2 . Specifically, subsection 3.1 describes the preparation of the subjective experiment including the collection of images and the recruitment of participants. Subsection 3.2 elaborates how we conduct the subjective experiment to obtain the 88K human feedbacks on 21,904 multi-sourced images. Subsection 3.3 explains how we prompt GPT-4 to refine the aesthetic critiques and obtain the 409K instruction-following data covering multiple aesthetic perception dimensions.Image Collection: To guarantee the diversity of image types, we first collect a large number of images from various sources, including natural images (NIs), artistic images (AIs) and artificial intelligence-generated images (AGIs), as shown in Figure 2(a) . Then, we use a well-trained scene classification model [39] to automatically predict the scene label for each image, based on which we sampled these images to maintain scene diversity. To reduce the long-tail distribution of randomly sampled images [50] , we add 1239 high-aesthetic images from LITE [42] and Impressions [25] datasets, and 1944 low-quality images from SPAQ [11] and KonIQ-10K [17] Table 1 : Overview of the image source datasets.NIs AADB [24] 1096 PARA [48] 3694 TAD66K [14] 4003 LITE [42] 1087 Impressions [25] 152 SPAQ [11] 1153 KonIQ-10K [17] 791AIs BAID [53] 2970 CAD [27] 59 ArtEmis [1] 1957AGIs DiffusionDB [43] 4228 AGIQA-3K [56] 570 AGIQA-1K [29] 144 datasets. Finally, a total of 21,904 images are collected, which are further fed to human subjects to collect aesthetic feedbacks based on subjective experiments. Detailed image source of the AesMMIT dataset is summarized in Table 1 .Subject Selection: To ensure the completeness of annotation and the diversity of corpus, we recruit subjects by considering five different perspectives, including age, gender, education, health status and personality, as shown in Figure 2(b) . Specifically, we ensure that each subject is in good health and passes the Ishihara color blindness test [48] . To maintain the quality of annotation, all subjects need to have a high school degree or above, and photography experience. Following the existing works [44, 48] , to ensure the diversity of annotation and language, we further consider age distribution, gender balance, and personality diversity [31] . Finally, considering the difficulty of labeling artistic and artificial intelligence-generated images, we specially invited 6 art practitioners and 9 researchers in the field of image aesthetics evaluation to join the subjective experiment and focus on labeling these two kinds of images. In addition, we also organize two aesthetic knowledge training sessions before the experiment covering evaluation, attribute, emotion and suggestion, aiming to provide them with a more comprehensive understanding of image aesthetics annotation. Finally, based on the principle of voluntary participation [48] , we invite 48 eligible human subjects to ensure the validity and reliability of annotations.To enhance the understanding and interpretation capabilities of MLLMs in terms of aesthetics perception, We invite all 48 subjects to contribute their insights and descriptions on image aesthetics through progressive questions, as illustrated in Figure 2(c ). This process includes three parts for each test image from coarse to fine. A screenshot of the annotation interface is shown in Figure 3 .1) Coarse-grained aesthetic evaluation. The focus of this question is to collect simple aesthetic grade judgments on images. Participants could express their overall impression of image aesthetics using simple sentences such as good-looking, beautiful, average or bad-looking, etc. Referring to common settings in the field of image quality assessment [9, 24] , we recommend that experimental participants establish a uniform measure of five aesthetic grades and guarantee language diversity.2) Fine-grained reasoning and explanation. On the basis of simple aesthetic grade judgments, this question is designed to mine the detailed reasons that influence human aesthetic judgments. Participants need to explain what specific aspects of the image enhance or reduce the aesthetic appeal, such as clarity, color, light, image object and composition. In addition, we encourage participants to provide suggestions for the improvement of low-aesthetic images.3) Finer-grained description on aesthetic feeling. This question breaks through the description of the inherent aesthetic attributes and aims to explore the impact of human emotions on aesthetic perception when viewing images. Participants need to describe some of the more abstract emotional factors in the image, e.g. novel shooting view, interesting content, expressed emotions, etc.. Furthermore, we encourage participants to analyze the impact of these factors on image aesthetics.The setting of the subjective experiment refers to the ITU-R Recommendation BT.500-13 standard [22] . By systematically collecting feedback from the above three parts, we construct the corpus-rich aesthetic critique database, called AesFeedback, which contains 88K human feedbacks on 21,904 multi-sourced images. Since these annotations are all provided by human subjects, AesFeedback can capture the characteristics of basic aesthetic perceptions and the evaluation reasoning process, providing a valuable resource for improving the aesthetic capabilities of MLLM.While the corpus-rich aesthetic critiques in the AesFeedback dataset can provide rich knowledge for aesthetic instruction tuning of MLLMs, we further design more instruction-following data to allow MLLMs to respond to a variety of human queries, achieving stronger aesthetic capabilities. Similar to existing works [6, 45] , we leverage GPT-4 to transform human feedback into instructionfollowing formats, as illustrated in Figure 2(d) . Through this process, the proposed AesMMIT dataset includes 409K instruction-response pairs, with its details as follows.Aesthetic Description: Similar to aesthetic caption [40] , the ability of image aesthetic description is crucial for MLLMs. As shown in Figure 2 (c), the collected AesFeedback dataset contains direct and comprehensive human natural language critiques on image aesthetics. Furthermore, these critiques provide aesthetic interpretation from overall aesthetic grades to fine-grained aesthetic attributes (e.g. clarity, color, light, and image object), which can activate the preliminary aesthetic interpretation abilities of MLLMs (see Figure 5 ). Therefore, we directly use the questions in the AesFeedback dataset as instructions and human critiques as responses, based on which we obtain the 88K instruction-following pairs of the proposed AesMMIT dataset.Aesthetic Question Answering: In addition to directly adopting the AesFeedback as aesthetic instruction-following data, we further propose a GPT-assist approach to refine the aesthetic critiques and assemble a larger-scale aesthetic question-answering subset (AesVQA). Inspired by COCO-VQA [32] and ShareGPT [6] , we prompt GPT-4 to generate diverse-style open-ended questions and provide corresponding brief answers, where both the questions and answers are based on human aesthetic descriptions. To ensure that MLLMs can handle diverse queries during the interaction, the questions are generated to cover commonly used question types. Specifically, Yes-or-No-style questions are straightforward queries that demand a simple yes or no as an answer, which are mainly used to provide a clear, binary response. What-style questions are leveraged to measure more comprehensive and complex aesthetic perception (e.g. various aesthetic attributes). How-style questions are used to ask more details about aesthetic emotion, aesthetic attributes, and improvement suggestions; Why-style questions are employed to explore the foundational aspects of what makes images aesthetically appealing or unappealing, revealing the rationale of beauty and visual appeal. To ensure the diversity of instructions, following the existing works [33, 35] , besides the direct answers, we also generate several distracting answers for the questions and convert them into an additional multi-choice question format [47] . Through the above process, we obtain the final AesMMIT dataset, which consists of 409K multi-typed instructions, aiming to activate stronger aesthetic capabilities. Figure 4 shows the frequently occurring words in the proposed AesMMIT dataset. It can be observed that most nouns, adjectives and adverbs are related to image aesthetic description, which is quite different from common semantic-based tasks. More details are provided in the Supplementary. 
model fine-tuning: 
 To verify the efficacy of the proposed AesMMIT dataset, we introduce instruction fine-tuning to improve the open-source MLLMs to enhance the aesthetic perception capabilities, obtaining multimodality Aesthetic Expert models.The proposed AesExpert model follows the design of LLaVA-1.5 [33] , which includes three components: (1) A visual model based on the CLIP-ViT-L14 [39] with an input size of 336×336, which converts the input image into 576 tokens. (2) A visual-language projector based on two-layer multi-layer perception (MLP), which is employed to connect the visual modality and language modality. (3) A language model based on the open-source Vicuna-v1.5 [57] , which is used to interpret instructions and generate responses. In this work, we Table 2 : Structures of the multi-modality foundation models for aesthetic instruction tuning.Visual Model V→L Language ModelLLaVA-1.5-7B CLIP-ViT-L14 MLP Vicuna-v1.5-7B LLaVA-1.5-13B CLIP-ViT-L14 MLP Vicuna-v1.5-13B mPLUG-Owl2 CLIP-ViT-L14 Abstractor LLaMA2-7Bbuild the AesExpert models based on two variants of LLaVA-1.5, including LLaVA-1.5-7B and LLaVA-1.5-13B. In addition, we also introduce the mPLUG-Owl2 [52] as the backbone to validate the aesthetic perception performance based on the AesMMIT dataset. The structures for these multi-modality foundation models are summarized in Table 2 .In general, the training of open-source MLLMs [33, 34, 58] includes two stages: aligning the representation space of the visual backbone and the LLM with million-scale web data [35] , and visual instruction tuning with a combination of multi-modality datasets [6, 8] .Considering that our purpose is to improve the aesthetic perception ability of the current MLLMs, we directly use the proposed AesM-MIT dataset to perform supervised instruction fine-tuning on the models pre-trained on general-purpose visual tasks [33] . Following existing works [5, 45] , supervised instruction fine-tuning not only enables the models to retain their original general knowledge but also facilitates the aesthetics perception capabilities. In this work, to enhance the training efficiency and compare fairly, we freeze the vision model and focus on fine-tuning the projector and the language model. Based on supervised instruction fine-tuning, we implemented three different versions of multi-modality aesthetic expert models.  
experiments 5.1 implementation details: 
 Training setting: In this work, we finetune three pre-trained MLLMs based on the constructed AesMMIT dataset in full schedule mode, including LLaVA-1.5-7B (Vicuna-v1.5-7B) [33] , LLaVA-1.5-13B (Vicuna-v1.5-13B) [33] and mPLUG-Owl2 (LLaMA-2-7B) [52] .In implementation, to ensure fairness, we follow the default hyperparameters provided by the original models. All models are trained based on 8 NVIDIA Tesla A100 80G GPUs, and the evaluation experiments are conducted on 2 NVIDIA RTX 4090 24G GPUs. Benchmark: To verify the performance of our proposed AesExpert models on image aesthetics perception, extensive experiments and comparisons are conducted on the AesBench [20] , which is a well-designed benchmark for MLLMs on aesthetics perception evaluation. Specifically, AesBench contains 2,800 images and four evaluation criteria designed from four dimensions, including (1) Aesthetic Perception (AesP) focuses on the ability of MLLMs to recognize and understand aesthetic attributes. (2) Aesthetic Empathy (AesE) evaluates the ability of MLLMs to resonate with the emotional aspects conveyed through aesthetic expressions like humans. (3) Aesthetic Assessment (AesA) evaluates the ability of MLLMs to judge aesthetic grades and predict quality scores based on the language description. (4) Aesthetic Interpretation (AesI ) involves the ability of MLLMs to interpret and analyze the reasons for aesthetic quality. Each dimension contains 2,800 questions and correct answers. For the first three dimensions, the accuracy of the answers is used to measure the performance of the model, while the AesI is evaluated based on GPT scoring.In this section, we compare the performance of our proposed AesExpert models (three versions) with 16 state-of-the-art MLLMs, including the popular GPT-4V [49] and Gemini Pro Vision [12] , as well as 13 state-of-the-art variants with open sources, i.e. LLaVA (LLaMA-2-Chat-7B) [34] , LLaVA-1.5-7B (Vicuna-v1.5-7B) [33] , LLaVA-1.5-13B (Vicuna-v1.5-13B) [33] , ShareGPT4V (Vicuna-v1.5-7B) [6] , Q-Instruct (LLaVA-v1.5-7B) [45] , mPLUG-Owl2 (LLaMA-2-7B) [52] , In-structBLIP (Vicuna-7B) [8] , MiniGPT-4 (Vicuna-7B) [58] , MiniGPT-v2 (LLaMA-2-Chat-7B) [5] , IDEFICS-Instruct (LLaMA-7B) [21] , GLM (ChatGLM-6B) [10] , Otter (MPT-7B) [28] , TinyGPT-V (Phi-2) [54] and Qwen-VL (QWen-7B) [3] . Detailed information about these models can be found in [55] . The results are listed in Table 3 .From Table 3 , we can find that the three versions of AesExpert achieve the top three results. Among them, AesExpert based on LLaVA-1.5-13B achieves the best performance, which is significantly better than the most advanced GPT-4V. For the existing open-source models, ShareGPT4V [6] performs best, but lags behind our AesExpert (L-13B) by more than 10%. These experimental results reveal that the proposed AesExpert models have the best aesthetic perception abilities and highlight the advantage of the constructed AesMMIT dataset for improving multi-modality foundation models.In this section, we quantitatively evaluate the aesthetic perception abilities of MLLMs after aesthetic instruction tuning in the four tasks defined by AesBench [20] . The experimental results are summarized in Tables 4, 5, 6 and 7, respectively. 4 , we can observe that fine-tuning baseline MLLMs using AesMMIT can significantly improve their image aesthetic perception abilities. Specifically, among the three baseline MLLMs, LLaVA-1.5-7b achieves the most performance improvements (over 12% on the overall score). For the four different perception dimensions, we noticed the most significant improvement for composition. The possible reason is that composition is a very important element in human aesthetic perception, which also can be found in Figure 4 , indicating that composition is the most frequently occurring word in the AesMMIT dataset. Therefore, the proposed AesExpert model fine-tuned on AesMMIT has excellent composition perception capabilities. In addition, among the three types of images, artificial intelligence-generated images obtain the biggest performance improvement. This is mainly because that existing instruction fine-tuning datasets usually contain very few artificial intelligence-generated images, and our AesMMIT dataset makes up for this shortcoming, achieving significant performance improvement. Finally, the biggest performance improvement of the four question types is 'How'. These findings inspire us to further expand our dataset to cover more perception dimensions and more question types in future studies.Aesthetic Empathy Ability. From Table 5 , it is observed that our AesExpert is superior to the baseline models by a large margin (more than 12%), especially on LLaVA-1.5-7B, with performance improvement up to 15.72%. Moreover, the average improvement on Uniqueness is more significant than on other dimensions, indicating that the major concerns for empathy raised by humans in the AesMMIT dataset are related to the uniqueness of perspective. Among the three types of images, artificial intelligencegenerated images still obtain the biggest performance improvement (more than 15%). In addition, for the four question types, HOW questions achieve the biggest performance improvement. In summary, these results underscore that AesMMIT can significantly improve the aesthetic empathy abilities of MLLMs, and even our AesExpert based on LLaVA-1.5-7B markedly surpasses the current top-performing GPT-4V (refer to Table 3 ). Aesthetic Assessment Ability. The observations from Table 6 underscore that the aesthetic instruction tuning also notably improves the aesthetic assessment ability of MLLMs, especially on the artificial intelligence-generated images with an average improvement of 10.46%. In contrast, the average improvement on artistic image (+1.87%) is less significant, implying that, due to the highly abstract nature, the aesthetic assessment of artistic images is still a relatively difficult task. We look forward to better solutions for artistic image in the future.Aesthetic Interpretation Ability. Hallucination has been regarded as one of the critical challenges for MLLMs [13, 18] , which imagines incorrect details about an image in visual question answering. To alleviate this problem, the proposed AesMMIT dataset is collected from human natural language feedback rather than machine-generated annotations. As can be seen from Table 7 , the Precision of interpretation has been significantly improved. For three different baseline MLLMs, significant improvements of 41.50%, 48.10% and 25.90% have been achieved improved, respectively. This result proves that the proposed AesMMIT allows the MLLMs to learn the style of human language and enhance the precision of aesthetic descriptions. In addition, from the overall score, the aesthetic instruction tuning based on AesMMIT significantly improves the aesthetic interpretation ability of MLLMs (average performance improvement of 15.47%), especially for LLaVA-1.5-7B, with a performance improvement up to 19.20%. These results demonstrate that the AesMMIT dataset could significantly benefit the existing MLLMs for obtaining enhanced aesthetic interpretation ability.To verify the effectiveness and necessity of collecting human feedback for improving the aesthetic perception abilities of MLLMs, we further conduct experiments to compare our dataset with the AVA-Comments dataset [37] , the largest multi-modality dataset in the image aesthetics domain, which contains over 250K images with It can be observed from Figure 5 that although AVA-Comments contains more images and instructions, it cannot provide effective aesthetic information to MLLM, resulting in poor aesthetic interpretation ability, especially on Precision. In contrast, the human feedback we collected (AesFeedback) can achieve pretty good aesthetic interpretation abilities for MLLMs. More importantly, the proposed AesMMIT dataset expanded by GPT can further improve the model performance. These results clearly demonstrate the effectiveness and necessity of the proposed AesMMIT dataset. 
conclusion: 
 In this work, we have made an attempt to exploit the aesthetic perception ability of the multi-modality foundation model. Specifically, we first build a corpus-rich aesthetic critique database via human natural language feedback (AesFeedback), based on which we further establish a comprehensively annotated aesthetic multimodality instruction tuning dataset (AesMMIT). In addition, we propose multi-modality aesthetic expert models based on aesthetic instruction fine-tuning, achieving significantly better aesthetic perception performances. We believe this work is a solid step in improving the aesthetic perception ability of MLLM, and we hope that our contribution will encourage the research community to build multi-modality foundation models that can understand highly abstract image aesthetics like humans. 
 
 Appendix:  


Decision: The paper proposes a method for image aesthetics perception using MLLMs. 

The reviewers mention (+) improving aesthetic perception performance over SOTA MLLM methods, (+) the large-scale dataset and corresponding evaluation model improving the state of this subjective task, (+) many experiments showing the capabilities of the model.

They are however concerned about (-) comparisons to the AVA Captions dataset, (-) some missing data from the paper, (-) missing comparison to VisionPrefer and Q-Bench.

The reviews were all tending positively, with initially two borderline accepts and one weak accept. After the rebuttal, it resulted in one borderline accept and two weak accepts. One reviewer judged for "poster", another mentioned that while it has some remaining issues, it has "valuable insights to the multimedia field". 

I believe overall this work should be accepted and I'll recommend doing so as poster.

Reviews: review: paper_summary: paper_summary: This paper presents a human-annotated multi-modality aesthetic dataset named AesMMIT, which consists of 409K multi-typed instructions. Based on the AesMMIT database, we fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert. Extensive experiments demonstrate the efficacy of AesMMIT and AesExpert.
strength_weakness: strength_weakness: **Strengths**

Good writing, well motivated with an interesting introduction/motivation, well presented methods section with very good illustrations.
limitations: limitations: **Limitations**

I have two main concerns regarding this paper:

* I am aware of several existing datasets that use MLLMs, i.e., GPT-4v related to image aesthetics or image quality, like VisionPrefer and Q-Bench. I believe the author should conduct comparative ablation studies on these datasets (if author can obtain them).

* I am somewhat concerned that if incomplete information appears in human feedback, it might prevent GPT-4 from correctly generating the instructed data.
paper_summary: paper_summary: The highly abstract nature of image aesthetics perception (IAP) poses significant challenge for current multimodal large language models (MLLMs). The lack of human-annotated multi-modality aesthetic data further exacerbates this dilemma, resulting in MLLMs falling short of aesthetics perception capabilities. 

This work introduces a comprehensively annotated Aesthetic Multi-Modality Instruction Tuning (AesMMIT) dataset. The database contains 21,904 diverse-sourced images and 88K human natural language feedbacks, which are collected via progressive questions, ranging from coarse-grained aesthetic grades to fine-grained aesthetic descriptions. The authors prompt GPT to refine the aesthetic critiques and assemble the large-scale aesthetic instruction tuning dataset, i.e. AesMMIT, which consists of 409K multi-typed instructions to activate stronger aesthetic capabilities. Based on the AesMMIT database, they fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, AesExpert.
strength_weakness: strength_weakness: Extensive experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision.
limitations: limitations: Image aesthetic assessment has always been a very abstract yet highly significant task. This work constructs a large-scale dataset and corresponding evaluation model, which greatly promotes the progress of aesthetic assessment.
paper_summary: paper_summary: This paper explores the problem of using multi-modality foundation models to handle image aesthetic perception task. To this end, the author constructed an aesthetic comment database (AesFeedback) through human natural language feedback, and further established an aesthetic multi-modal instruction tuning dataset based on this. Based on the AesMMIT database, the author fine-tuned the open source general foundation model and implemented a AesExpert model. Experiments demonstrate that the proposed AesExpert model has better aesthetic perception performance than state-of-the-art multi-modal large language models.
strength_weakness: strength_weakness: 1.	This study fills the gap of existing instruction tuning datasets for image aesthetic. The authors provide a new research path for multimodal aesthetic perception by collecting large amounts of human natural language feedback and constructing a new AesMMIT dataset.
2.	The paper verifies the performance of the AesExpert model through many experiments. The experimental results show that the AesExpert model is significantly better than the SOTA MLLMs in aesthetic perception performance.
limitations: limitations: 1.	It would be beneficial to provide more examples of image-label pairs from the AesMMIT dataset.
2.	In Section 5.4, you mentioned conducting experiments to compare your dataset with the AVA-Comments dataset to verify the effectiveness and necessity of collecting human feedback for improving the aesthetic perception abilities of MLLMs. However, it is worth noting that the AVA-Comments dataset also consists of comments from human feedback. How to prove the above point?
3.	AVA-Captions (Aesthetic Image Captioning From Weakly-Labelled Photographs) dataset contains aesthetic comments that have been screened. What are the results when using this data set?
4.	More details about the AesExpert model, including its architecture and training process, would be helpful. This information would provide a deeper understanding of the model and could aid others in replicating or building upon your work.