Semantics-Aware Image Aesthetics Assessment using Tag Matching and Contrastive Ranking

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 05:25:51

Abstract: The perception of image aesthetics is built upon the understanding of semantic content. However, how to evaluate the aesthetic quality of images with diversified semantic backgrounds remains challenging in image aesthetics assessment (IAA). To address the dilemma, this paper presents a semantics-aware image aesthetics assessment approach, which first analyzes the semantic content of images and then models the aesthetic distinctions among images from two perspectives, i.e., aesthetic attribute and aesthetic level. Concretely, we propose two strategies, dubbed tag matching and contrastive ranking, to extract knowledge pertaining to image aesthetics. The tag matching identifies the semantic category and the dominant aesthetic attributes based on predefined tag libraries. The contrastive ranking is designed to uncover the comparative relationships among images with different aesthetic levels but similar semantic backgrounds. In the process of contrastive ranking, the impact of long-tailed distribution of aesthetic data is also considered by balanced sampling and traversal contrastive learning. Extensive experiments and comparisons on three benchmark IAA databases demonstrate the superior performance of the proposed model in terms of both prediction accuracy and alleviating long-tailed effect. The code of the proposed method will be public.

Main paper: 
 introduction: 
 The rise of mobile Internet and the widespread popularity of social media platforms, such as Instagram and WeChat, have transformed the way we share and consume images. In this era, users have become increasingly focused on the aesthetic appeal of images. In view of this, image aesthetics assessment (IAA) has witnessed a notable surge in research interest in recent years, attracting a growing number of scholars dedicated to investigating its multifaceted dimensions. IAA approaches have extensive applications [5] , such as image recommendation [39] , image enhancement [8] , and image retrieval [44] , etc. In the past few years, a proliferation of novel IAA models have been reported in the literature. Early IAA models primarily used hand-crafted features and simple machine learning methods to classify images into high and low aesthetic categories, which were inspired by people's intuitive judgments of image aesthetics and widely accepted photography rules [4, 19] . With the boom of deep learning, the mainstream methods for aesthetics assessment started leveraging Convolutional Neural Networks (CNN) to unearth common aesthetic criteria embedded within large datasets. Accordingly, the IAA has been refined into three specific tasks: aesthetic binary classification [22, 41] , aesthetic score regression [21, 24] , and aesthetic distribution prediction [3, 40] . More recently, the emergence of large language models like GPT [37] has fueled the rapid development of the multimodal vision field. The IAA community has also made attempts on investigating the subsidiary effect of the language modality [7, 10, 18, 38] .The perception of image aesthetics is built upon the understanding of semantic content. Numerous existing IAA methods have proven the advantages of integrating semantic content into aesthetic perception modeling. There are two main types of integration: one involves incorporating semantic categories as additional labels for training in a multi-task manner [26, 41] , while the other directly utilizes semantic features (e.g., features from pre-trained models) as the foundation for learning aesthetic features [6, 13] . These approaches are designed based on an assumption that aesthetic perception can be more easily achieved among images with similar semantic contents. However, there is no explicit comparison made among these images, so the model may not effectively discern and perceive aesthetic distinctions within the same semantic context. The aesthetic distinctions among images are typically determined by two key factors. Firstly, the inherent aesthetic attributes of images, encompassing elements like composition (e.g., balanced or unfocused) and lighting (e.g., exposure or evenness), play a pivotal role in shaping its overall aesthetic quality. Secondly, the distinctions are directly reflected in the distribution of images with similar semantic backgrounds but different aesthetic levels. Analyzing the comparative relationships among these images helps to evaluate how well each image aligns with the aesthetic expectations and preferences associated with its semantic background.Inspired by the above facts, this paper presents a semanticsaware image aesthetics assessment approach, which first analyzes the semantic content of images and then models aesthetic distinctions from two aspects: aesthetic attribute and aesthetic level, as illustrated in Figure 1 . Concretely, we propose two strategies, dubbed tag matching and contrastive ranking, to extract knowledge pertaining to image aesthetics. In view of the differences in the lexical representation of semantic and aesthetic attribute in aesthetic description, two lexical tag libraries for semantic recognition and aesthetic attribute analysis are built. The tag matching then extracts similarity matching relationship between individual images and the predefined semantic and attribute tag libraries. The contrastive ranking is designed to uncover the comparative relationships between image sets with different aesthetic levels within specific semantic backgrounds.The contributions of this work are threefold:• We propose a semantics-aware image aesthetics assessment approach using tag matching and contrastive ranking (TMCR), which first analyzes the semantic content of images and then models aesthetic distinctions from two perspectives: aesthetic attributes and aesthetic levels. • We design a novel contrastive ranking method to investigate the information about aesthetic levels from images under similar semantics, which extracts both intra-level aesthetic criteria and inter-level aesthetic ordering. Moreover, the problem of the long-tailed distribution is alleviated through balanced sampling and traversal contrastive learning. • We conducted extensive experiments on three benchmark databases for image aesthetics assessment, including AVA, AADB, and PARA. The experimental results demonstrate that the proposed TMCR model outperforms the state-ofthe-art methods and exhibits better long-tailed prediction capability. 
related work 2.1 image aesthetics assessment: 
 Early IAA models utilized hand-crafted features and simple machine learning to classify images into high and low aesthetic categories, based on photography rules. Datta et al. [4] extracted nine features from images, such as exposure of light and colorfulness, composition of regions and depth-of-field, based on which the Support Vector Machine (SVM) is utilized to build the aesthetic classification model. Ke et al. [19] employed perceptual features to classify high and low aesthetics images, including spatial distribution of edges, color distribution, and hue count, etc. Although these features have clear physical interpretation, it is challenging to comprehensively capture the aesthetic characteristics of images, due to the inherent abstraction of image aesthetics and the limited understanding of the underlying mechanisms of aesthetics.With the rise of deep learning, IAA methods began to adopt deep neural networks to uncover the common aesthetic criteria present in large datasets. Meantime, the IAA has been refined into three specific tasks: aesthetic binary classification, aesthetic score regression, and aesthetic distribution prediction. Murray et al. [33] first established a large-scale database for aesthetic visual analysis, called AVA. Recognizing the importance of both the global and local characteristics of images in aesthetic assessment, several approaches based on patch-level analysis have been developed [30, 31] . Lu et al. [29] proposed the RAPID algorithm, which leverages a double-column convolutional network to separately handle the global and local views of images for aesthetic classification. There are also some theme-oriented IAA networks, motivated by the fact that perception of image aesthetics is built upon the semantic understanding of content [21, 41] . He et al. [6] presented a network that combines aesthetic attributes and color space to jointly predict aesthetic distributions. Li et al. [26] proposed a theme-aware visual attribute reasoning framework, which utilizes a bilevel Graph Convolutional Network (GCN) to simulate the human aesthetic perception process. More recently, Jia et al. [13] designed an effective full-resolution technique to perform the IAA task based on theme information.Currently, with the rise of multimodal learning, aesthetic-related information from the language modality has been introduced into IAA. Large pre-trained vision-language models, like CLIP [36] , have been proved to be more suitable for image aesthetics modeling, because the language modality contains rich semantics and aesthetics clues [7] . Zhou et al. [52] constructed the AVA-Comments database, the first multimodal database for IAA that combines visual and textual information. They also proposed a multimodal IAA model based on Deep Boltzmann Machine (DBM) for binary aesthetic classification. Ke et al. [18] pretrained an image-text encoder-decoder model using contrastive and generative objectives to learn multimodal aesthetic representations from user comments. Sheng et al. [38] proposed an attribute-aware contrastive learning strategy to mitigate the domain shift from the general visual domain to the aesthetics domain, improving CLIP's performance on the IAA task. These studies have provided compelling evidence that the integration of textual information can greatly enhance the representation capability of IAA models.Contrastive learning [23] is a representation learning technique that compares similar and dissimilar data pairs to learn meaningful patterns and discriminative information. It encourages the model to group similar instances together and separate dissimilar ones, resulting in effective feature representations. Wu et al. [43] proposed 'Instance Discrimination', which treats two augmented views of an image as a positive pair and other images as negative pairs, aiming to train the model to learn instance-specific features invariant to different image enhancements. Li et al. [25] introduced a method that combines prototypical networks with contrastive learning for unsupervised representation learning. By integrating prototypical Pipeline "Semantics" "sky" "portrait" "face" "scene" "building" "eyes" "ribbon" tree" "water" "food" "bridge" "wall" "door" "landscape" "house" "feather" "child" "fish" "people" "…" "Attributes" "shot" "composition" "focus" color" "creative" "center" "reflection" "texture" "detail" "crop" "white" "contrast" "sharp" "tone" "exposure" "abstract" "saturation" "…" information, the model learns more discriminative and semantically meaningful representations, leading to improved performance in downstream tasks. Contrastive learning has been successfully applied in various domains, allowing models to capture and utilize valuable insights from the data [47, 49] .In the area of quality perception, Chen et al. [2] applied contrastive learning to video quality assessment, using distortion types as positive and negative selection criteria to learn effective representations for video quality score regression. Zhao et al. [50] developed diverse degradation strategies to generate distorted images and utilized content-aware and quality-aware comparisons to derive representations related to image quality. Contrastive learning also has great potential for addressing long-tailed problems. Kang et al. [16] discovered that the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced. They further introduced a K-positive contrastive learning that effectively combines the advantages of supervised learning and contrastive learning to achieve balanced and discriminative representations. Zhu et al. [53] introduced the Balanced Contrastive Learning (BCL) method, which utilizes class-averaging and class-complement strategies to address the optimization bias in Supervised Contrastive Learning (SCL) [20] and enhance the performance of long-tailed visual recognition tasks. These studies provide insights into the investigation of contrastive learning to explore the aesthetic comparative relationship and mitigate the impacts of the long-tailed distribution of aesthetic data. 
proposed method 3.1 problem formulation: 
 Image aesthetics assessment (IAA) aims at mining the general public's criteria of aesthetics from large-scale image aesthetics data, and then generating aesthetic evaluation ŷ in line with the groundtruth aesthetic quality of a given image 𝐼 , which can be formulatedEQUATIONwhere M (•) represents the established IAA model with 𝜃 as the parameter, 𝑨𝒆𝒔 𝑮 refers to the generic aesthetics consensus learned from large-scale aesthetic annotation data.The perception of aesthetics is grounded in semantic analysis, and previous studies have shown the benefits of utilizing semantic analysis for modeling aesthetic perception [6, 11, 13, 21, 26, 41] . In this study, we specifically examine the connection and integration of semantic analysis to aesthetic attributes and levels. This is accomplished by the proposed tag matching (TM) and contrastive ranking (CR) strategies. By incorporating these two relationships, we train the IAA model to generate aesthetic predictions. The whole process can be expressed as:EQUATIONwhere, 𝑨𝒆𝒔 𝒂𝒕 𝒕 𝒓 𝒔𝒆𝒎 and 𝑨𝒆𝒔 𝒍𝒆𝒗𝒆𝒍 𝒔𝒆𝒎 represent the prior knowledge related to aesthetic perception modeling, obtained from parameters 𝜃 𝑇 𝑀 and 𝜃 𝐶𝑅 , respectively. Figure 2 shows the overall framework of the proposed TMCR model.When describing an image in aesthetic perception, aesthetic description and semantic description are often intertwined, and aesthetic description cannot exist independently from semantics. Additionally, words describing semantics are often associated with objects or themes, while words describing aesthetics are often related to attributes such as clarity, composition, and feeling. Recently, Zhong et al. [51] conducted manual selection of object-related words and aesthetics-related words to calculate the Aesthetic Relevance Score (ARS) of a sentence, which has contributed to the development of aesthetics assessment and aesthetics captioning. Building upon the observations and previous studies, we construct Aesthetics Description: This image looks very beautiful. The image is very clear, with both the hills in the distance and the wooden cabin nearby giving the objects a comfortable spatial composition. The overall tone looks very eye-catching. The country house after the rain seems to make people smell the fragrance of earth and green grass, making people feel relaxed and leisurely.Prompt: Please list the words that describe semantics (object, theme, subject, etc.) and aesthetics attributes (clarity, color, light, composition, feeling etc.) in {Aesthetics Description}.Sure, here's a list of words from the sentence that pertain to semantics and aesthetics attributes: Semantics: aesthetic attribute analysis. These libraries are built using existing multi-modal aesthetics databases, including DPC2022 [51] , which is currently one of the largest datasets of aesthetic comments, and EPAD [12] , which contains high-quality aesthetic comments from experts. Specifically, we utilize GPT4 and manual filtering to create these tag libraries, and the process is shown in Figure 3 . Finally, we obtain 2145 semantic recognition tags and 1659 attribute analysis tags.• hills • wooden cabin • country house • rain • earth • green grass Aesthetics Attributes: • clear • comfortable spatial composition • eye-catching • tone • fragrance • relaxed • leisurely ………… "Attributes" "Semantics" Manual FilteringNext, we employ the multimodal pretrained model CLIP to identify the semantic content and aesthetic attributes of an image. Initially, the textual tags are converted into deep feature representations using the text encoder 𝐸 𝑇 of CLIP, which is expressed as follows:EQUATIONEQUATIONwhere F 𝒎 𝒔𝒆𝒎 and F 𝒏 𝒂𝒕 𝒕 𝒓 represent the transformed semantic tag feature and attribute tag feature, respectively. Given an image 𝐼 , cosine similarity is utilized to establish the correlation between the image and the two tag libraries:EQUATIONEQUATIONwhere 𝑺 𝒊𝒎 𝑴 𝒔𝒆𝒎 and 𝑺 𝒊𝒎 𝑵 𝒂𝒕 𝒕 𝒓 respectively represent the similarity features with two tag libraries. 𝐸 𝐼 is the image encoder of CLIP. Finally, the direct concatenation of the two similarity features is utilized to represent the interconnection between semantic content and aesthetic attributes in an image:EQUATIONwhere, F 𝒂𝒕 𝒕 𝒓 𝒔𝒆𝒎 represents the fused feature, and 𝑐𝑜𝑛𝑐𝑎𝑡 (•) denotes the feature concatenation.Contrastive ranking [8, 9] Negative bins [8, 9] Anchor bin [5, 6] Positive bin[2, 3] [3, 4] [4, 5]Negative bins [8, 9] Anchor bin [3, 4] [2, 3]Positive binA balanced-level contrastive batch under similar semantics Figure 4 : Illustration of the contrastive ranking. A balancedlevel batch is first sampled according to semantic contents and GAS. Then, pull-pushing force of contrastive learning is employed to explore intra-level aesthetic criteria and interlevel aesthetic ordering (Taking ' [8, 9] ' as an example).The goal of contrastive ranking is to uncover the comparative relationships between image sets with different aesthetic levels but similar semantic backgrounds. The comparative relationships are reflected in the distribution of images within and across aesthetic levels, representing aesthetic criteria and aesthetic ordering, respectively. To this end, the images within an aesthetics database are first semantically clustered, and then divided into subsets according to their semantic categories and generic aesthetic scores (GAS). Furthermore, considering the long-tailed distribution problem of existing aesthetic data, the majority of images in the training set are primarily distributed in the middle score segment, and there is a relatively small number of images in the low and high score segments, resulting in poor prediction performance for high-quality and lowquality images. To address this issue, we employ a balancing process for the divided sets, generating relatively balanced batches for each aesthetic level across different semantic backgrounds. Then, the traversal contrastive learning is designed to explore intra-level aesthetic criteria and inter-level aesthetic ordering. The process of contrastive ranking is illustrated in Figure 4 . Semantic Clustering. Given an image aesthetics database D 𝑎𝑒𝑠 = 𝐼 𝑖 , 𝑦 𝑖 𝑁 𝑎𝑒𝑠 𝑖=1 , where 𝐼 𝑖 denotes the i-th image, 𝑦 𝑖 denotes the aesthetic distribution (which can be transformed into GAS) and 𝑁 𝑎𝑒𝑠 represents the number of images. Firstly, Eq. 5 is utilized to identify the semantic content of images based on their similarity to the predefined semantic tag library. Subsequently, the K-means algorithm is employed to cluster the images with similar semantic backgrounds. Within each cluster, the images are reorganized based on different aesthetic levels. Currently, most databases use integer score ranges from 1 to 10 to collect aesthetic ratings. Therefore, at this stage, we also partition the data based on this rating level setting to obtain a new database D 𝑐 for subsequent training:EQUATIONwhere, [•] 𝑘 denotes k-th semantic cluster, and (•) 𝑙 in the cluster represents the image set at l-th aesthetic level. Balanced Sampling. Considering the uneven distribution of images across different aesthetic levels, it is crucial to mitigate the impact of this data distribution on modeling the contrastive relationship. Therefore, a balanced sampling process is conducted on the database D 𝑐 obtained from the previous step. Specifically, for a particular semantic cluster [•] 𝑘 , a balanced quantity value 𝑁 𝑠 is generated based on the distribution of images across different aesthetic levels within the cluster. Then, a random sampling operation is performed from the image sets for each aesthetic level, forming a balanced-level batch. Through repeated sampling, multiple balanced batches are obtained with different semantic backgrounds.Traversal Contrastive Learning. For a balanced-level batch, the traversal contrastive learning is designed to explore the aesthetic comparative relationship among images distributed at different aesthetic levels. Concretely, for samples within the aesthetic level interval ' [8, 9] ', which is regarded as an 'Anchor bin', the traversal contrastive process can be described as follows. In the first-round comparison, the samples within the interval are considered as positive samples for each other, while the samples in other intervals in the batch are considered as negative samples. In the second-round comparison, the 'Positive bin' is adjusted to ' [7, 8] ', and all level intervals less than 7 are considered as 'Negative bins'. This comparison process repeats until all level intervals in the batch are traversed, which represents a traversal comparison of the anchor bin ' [8, 9] '. The first-round comparison aims to learn intra-level aesthetic criteria and the subsequent comparisons aim to learn inter-level aesthetic ordering.The contrastive encoder 𝐸 𝐶𝑅 is introduced to convert images into deep features, which is represented as: F 𝒍𝒆𝒗𝒆𝒍 𝒔𝒆𝒎 = 𝐸 𝐶𝑅 (𝐼 𝑖 ), where, F 𝒍𝒆𝒗𝒆𝒍 𝒔𝒆𝒎 denotes the transformed deep feature. The contrastive loss based on the aesthetic level relationship of this batch is defined as:EQUATIONEQUATIONwhere, L 𝐶𝑅 is the contrastive ranking loss, 𝑎𝑛𝑐ℎ𝑜𝑟 𝑏𝑖𝑛 , 𝑝𝑜𝑠 𝑏𝑖𝑛 , 𝑛𝑒𝑔 𝑏𝑖𝑛 represent the anchor sample interval and its corresponding positive and negative sample interval in the process of traversal contrastive learning, respectively. Ψ 𝑐 is the contrastive loss, 𝑎, 𝑝, 𝑛 are samples from 𝑎𝑛𝑐ℎ𝑜𝑟 𝑏𝑖𝑛 , 𝑝𝑜𝑠 𝑏𝑖𝑛 , 𝑛𝑒𝑔 𝑏𝑖𝑛 , * represents the inner product operation, and 𝜏 is the temperature coefficient. By updating the parameters through balanced-level batches sampled from different semantic clusters, the contrastive encoder 𝐸 𝐶𝑅 extracts knowledge related to aesthetic levels in image sets across various semantic contexts:EQUATION)where 𝜃 𝐶𝑅 is the parameter of the contrastive encoder, 𝛼 is the learning rate, and 𝑁 𝑏 is the number of balanced batches.Based on the modeling of aesthetic attribute and aesthetic level, aesthetics prediction is performed in this stage. Specifically, we perform dimension transformation on the F 𝒂𝒕 𝒕 𝒓 𝒔𝒆𝒎 , F 𝒍𝒆𝒗𝒆𝒍 𝒔𝒆𝒎 , and then directly concatenate them:EQUATIONwhere, F 𝒂𝒆𝒔 represents the fused feature. A score 𝑀𝐿𝑃 is used to generate aesthetic distribution ŷ based on the fused feature:EQUATION)The Earth Mover's Distance (EMD) loss is applied to optimize the model parameters:EQUATIONEQUATIONwhere, L 𝑎𝑒𝑠 is the loss function, 𝐶𝐷𝐹 (•) is the cumulative distribution function, 𝑑 represents the dimensions of the aesthetic distribution, 𝑟 is set to 2 to penalize the Euclidean distance. 𝜃 (𝐸 𝐶𝑅 , 𝑀𝐿𝑃) denotes the parameter to be updated, 𝛽 is the learning rate. 
experiments 4.1 iaa databases: 
 To gauge the efficacy of the proposed IAA model, we conduct experiments on three benchmark IAA databases, including AVA [33] , AADB [21] , and PARA [45] . AVA (Aesthetic Visual Analysis) [33] is currently the largest and most widely used database in the field of IAA. The database contains a total of 255,530 images. Each image receives annotations from an average of 210 annotators, with scores ranging from 1 to 10. Researchers commonly employ a standardized data partitioning approach, using approximately 230,000 images for training and the remaining 20,000 images for testing.AADB (Aesthetics and Attributes DataBase) [21] is a database specifically designed for analyzing aesthetic attributes of images. The database comprises 10,000 images, each annotated by at least five users. The annotations include 11 aesthetic attributes and an overall aesthetic rating for each image. The attribute scores range from -1 to 1, while the overall aesthetic rating ranges from 1 to 5.PAPA (Personalized image Aesthetics database with Rich Attributes) [45] consists of 31,220 images annotated by 438 participants. It includes image-centric annotations such as overall aesthetic ratings (1-5), distortion quality ratings (1-5), scene categories (10 types), and aesthetic attributes (5 types). Desensitized subjective information is also recorded, including gender, age, education Training Settings. We implement the model using PyTorch [35] and utilize the AdamW [28] optimizer for parameter updating. In the tag matching stage, we employ the ViT-L/14 version of CLIP to extract similarity features. Notably, to preserve its matching ability, both the image and text encoders remain frozen throughout the process. In the contrastive ranking stage, IAA databases are reorganized based on semantic clustering and aesthetic levels.For aesthetics prediction, a score MLP consisting of two linear layers followed by Softmax is used. Regarding hyper-parameters, the learning rate 𝛼 for contrastive ranking is set to 1e-5 with weight decay of 5e-2. The temperature coefficient 𝜏 is set to 0.07. The batch size depends on the distribution of images across different aesthetic levels. The training process lasts for 50 epochs. For aesthetics prediction, the learning rate 𝛽 is set to 1e-4 with a batch size of 64, decaying by 0.9 per epoch until convergence. Evaluation Metrics. For aesthetic binary classification, Accuracy (ACC) is used to measure performance. For aesthetic score regression, Spearman Rank order Correlation Coefficient (SRCC), Pearson Linear Correlation Coefficient (PLCC) and Mean Squared Error (MSE) are used to quantify the consistency between the predicted scores and the ground-truth GAS. For aesthetic distribution prediction, EMD with 𝑟 = 1 and 𝑟 = 2 are calculated, which are represented as EMD1 and EMD2, respectively.Performance on AVA. We first conducted experiments to evaluate the performance of our model on the widely used AVA database. We compared our model with 13 IAA models, including both classic and the latest approaches, and the results are summarized in Table 1 . It is observed from the table that the proposed TMCR shows excellent performance, particularly in aesthetic score regression, surpassing Method SRCC↑ PLCC↑ NIMA [40] 0.708 0.711 Hosu et al. [9] 0.725 0.726 PA_IAA [27] 0.720 0.728 TANet [6] 0.738 0.737 MUSIQ [17] 0.706 0.712 Celona et al. [1] 0.757 0.762 TAVAR [26] 0.761 0.763 AADB TMCR 0.775 0.773 NIMA [40] 0.886 0.923 Hosu et al. [9] 0.842 0.892 PA_IAA [27] 0.877 0.919 TANet [6] 0.883 0.917 MUSIQ [17] 0.882 0.918 Yang et al. [45] 0.902 0.936 TAVAR [26] 0.911 0.940 PARA TMCR 0.915 0.945 all previous algorithms. Furthermore, compared with the methods with full-resolution input, our approach utilizes cropped patches as input and still achieves notably superior performance. Moreover, with an relatively lightweight backbone of ResNet-50, the proposed method still delivers very competitive performance.Performance on AADB and PARA. We further evaluated the performance of the proposed model on the AADB and PARA databases. These two databases have a relatively smaller number of raters per image compared to AVA. Consequently, the existing approaches primarily reported SRCC and PLCC metrics. Table 2 summarizes the performance of our model compared to the stateof-the-art IAA models on the AADB and PARA databases. The results demonstrate that our model achieves the best performance. Compared to methods trained with attribute labels contained in databases, such as TAVAR [26] , the proposed model achieved even better performance by solely utilizing aesthetic score annotations. [14, 15] . However, aesthetic data naturally exhibits a long-tailed distribution, as show in Figure 5 (a), and accurately discerning the tail-end data holds significant practical value.In the proposed method, we designed two strategies, namely balanced sampling and traversal contrastive learning, to alleviate the issue of long-tailed distribution when extracting knowledge related to aesthetic levels. For long-tailed testing, we divide score ranges into three regions ('Many', 'Med', and 'Few') based on the distribution of image quantities in the AVA training set. We sample an equal number of samples from each corresponding region in the original test set, creating a balanced test set (457 images for each region) for long-tailed evaluation.We compare the performance of the proposed TMCR model with NIMA [40] (considered as the baseline), TANet [6] , AesCLIP [38] , and VILA [18] (recent models with top performances) on the balanced test set. We reported the comparative results of relevant metrics in Table 3 , where MSE is employed to quantify the prediction performance across different regions, SRCC and PLCC are used to measure the correlations across the whole score range. From Table 3 , we have the following observations. First, all models experienced varying degrees of long-tailed impacts, manifested by a decrease in performance in the corresponding region of testing as the training data decreased. Secondly, the proposed TMCR achieves the best results in terms of overall SRCC and PLCC metrics, as well as MSE metrics across 'ALL', 'Med. ' and 'Few' regions, which demonstrates the advantage of TMCR in long-tailed testing. Furthermore, in Figure 5 (b), we visualize the actual prediction performance of these models. It can be observed that 'TANet' predominantly generates predicted scores within the middle range and 'VILA' exhibits the widest range of score prediction intervals. The higher MSE of TANet and VILA compared to NIMA suggests that TANet's predictions for the 'Few' region significantly affect the overall performance, while VILA exhibits severe prediction errors despite its ability to handle a wide range of scores. These findings highlight the importance of investigating the long-tailed problem in IAA. Impact of Semantics and Attribute Tags. We also conduct a comparison between using semantic tags alone and using both semantics and aesthetic attribute tags during the tag matching stage, and the results are listed in Table 5 . The results demonstrate that the Impact of Sampling Frequency. We further conduct experiments to analyze the impact of sampling frequency on the learned representations during contrastive ranking. The contrastive ranking is performed on balanced-level batches sampled from different semantic clusters. Figure 6 shows the model performance in the linear probe protocol with varying sampling frequencies. The results clearly demonstarte that increasing the sampling frequency results in enhanced feature representations.The proposed IAA model aims to capture the aesthetic distinctions among images with similar semantic backgrounds from two aspects: aesthetic attribute and aesthetic level. To visually demonstrate the effectiveness of the model in representation difference across the Semantic space, Attribute space and Level space, we conduct a visual analysis. Specifically, we utilize the T-SNE [42] to visualize the feature embeddings F 𝒎 𝒔𝒆𝒎 of individual images in relation to the semantic tag library. Subsequently, we visualize the attribute feature embeddings F 𝒏 𝒂𝒕 𝒕 𝒓 and the level feature embeddings F 𝒍𝒆𝒗𝒆𝒍 𝒔𝒆𝒎 of images from a balanced-level batch, which is sampled from a specific semantic cluster. In addition, we visualize three instances, their top-5 semantic and aesthetic attribute tags, as well as their mappings in three spaces. The visualizations of Sem space, Attr space and Level space are shown in Figure 7 . It can be observed that, features displaying clustering patterns in the semantic space show distinct separability in both the attribute space and the level space. Specifically, in the level space, features belonging to different aesthetic levels are clearly separated and arranged according to their hierarchical relationship. These findings provide evidence for the effectiveness of the tag matching and contrastive ranking strategies, thus demonstrating the efficacy of the proposed TMCR in modeling aesthetic perception. 
conclusion: 
 In this paper, we have presented a semantics-aware image aesthetics assessment approach using tag matching and contrastive ranking, dubbed TMCR. We learn effective representations in the aesthetic attribute space and aesthetic level space to model the aesthetic distinctions among images under similar semantic backgrounds. In view of the differences in the expression of semantics and aesthetic attribute in aesthetic description, we propose leveraging CLIP's text-image matching ability to model the relationship between individual images and two predefined tag libraries. Moreover, we proposed a novel contrastive ranking strategy to model aesthetic level knowledge. During the contrastive ranking process, the impact of the long-tailed distribution of aesthetic annotation data is alleviated through balanced sampling and traversal contrastive learning. Extensive experimental results demonstrate that our proposed model outperforms state-of-the-art methods. While very encouraging performances have been achieved in this work, further investigations are needed to explore the extraction and modeling of aesthetic-related information in the language modality, as well as addressing the challenge of aesthetic deep imbalanced regression problem.  
 
 Appendix:  


Decision: The paper introduces a novel approach to IAA that addresses important challenges in the field. While there were some initial concerns, the authors have adequately addressed most of them in their rebuttal. The paper's strengths in tackling semantic awareness in IAA and its effective handling of long-tail distribution make it a valuable contribution to the field. The integration of tag matching and contrastive ranking strategies offers a fresh perspective on IAA that could inspire further research in this direction. Therefore, based on the reviewers' feedback and the authors' rebuttal, I recommend acceptance for this paper. I encourage the authors to carefully consider the reviewers' feedback when preparing the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper proposes a semantic-aware image aesthetics assessment method, aiming to solve the challenges caused by the diversification of image semantic backgrounds in image aesthetics assessment (IAA). The method models the aesthetic differences between images from two perspectives: aesthetic attributes and aesthetic levels. Specifically, two strategies called tag matching and contrastive ranking, are proposed to extract knowledge related to image aesthetics. Tag matching identifies semantic categories and main aesthetic attributes based on a predefined tag library, and contrast ranking reveals comparative relationships between images with different aesthetic levels but similar semantic backgrounds through balanced sampling and traversal contrast learning.
strength_weakness: strength_weakness: 1.	The proposed semantics-aware image aesthetics assessment method, using tag matching and contrastive ranking, fills the gap in existing IAA methods by distinguishing aesthetic differences within the same semantic context.
2.	By employing balanced sampling and traversal contrastive learning, the method effectively mitigates the long-tail distribution problem in aesthetic data, improving the model's prediction performance across different aesthetic levels.
3.	Experiments on benchmark databases demonstrate the competitive performance of the proposed model in terms of prediction accuracy and handling long-tail effects.
limitations: limitations: 1.	CLIP's matching capability is not been fully utilized. Specifically, the way the authors input words into the text encoder to obtain text features is inconsistent with CLIP's original pre-training method using prompts (sentences). In that case, using other single-modal models might achieve similar results, as the interaction between text and vision, a key feature of CLIP, is not being leveraged. 
2.	The construction of semantic and aesthetic attribute tag libraries relies on manual filtering and tools like GPT-4, increasing the complexity and cost of implementing the method. The accuracy and coverage of manually curated tags directly impact the model's performance.
paper_summary: paper_summary: In the paper, authors argue that Image Aesthetics Assessment is closely related to the image semantics, so they use tag matching strategy to model the image semantics, and proposed an image aesthetics assessment method using tag matching and contrastive ranking.
strength_weakness: strength_weakness: This work provides a new perspective for image aesthetic assessment, that is, the impact of image semantics on IAA. They introduced tag matching and contrastive ranking into IAA.
limitations: limitations: The analysis of experimental results is not sufficient. In Tab 1, some metrics is lower than AesCLIP (TMCR Swin-T(b) vs AesCLIP ViT), but the author did not analyze the reasons. 
I doubt the advancement of the method. Compared TMCR Swin-T(b) and AesCLIP ViT, 
most of the metrics of the proposed method are higher, but the metrics of TMCR ViT is almost lower than AesCLIP ViT. So, with the same backbone, the effect of this method is not as good as AesCLIP.
Some details of the method are not clearly described. The diagram of model architecture is too simple. It is best to give a complete structure and indicate which parts are trainable and which parts are frozen. Is this model trained in three stages, tag matching stage, contrastive ranking stage and aesthetics prediction stage? It is better to describe clearly. The details of clustering are not clearly described, such as the number of categories.
paper_summary: paper_summary: Semantic Content Analysis: The approach begins by analyzing the semantic content of images to establish an understanding of their meaning and context.
Modeling Aesthetic Distinctions: It models the aesthetic differences among images from two perspectives:
Aesthetic Attribute: Identifying the dominant aesthetic attributes based on predefined tag libraries through a strategy called "tag matching."
Aesthetic Level: Uncovering comparative relationships among images with similar semantic backgrounds but different aesthetic levels using "contrastive ranking."
Consideration of Data Imbalance: During contrastive ranking, the impact of long-tailed distribution in aesthetic data is addressed using balanced sampling and traversal contrastive learning to ensure accurate assessments.
strength_weakness: strength_weakness: This paper integrates tag matching and contrastive ranking.
The analysis of long-tailed performance presented in this paper is of particular significance.
limitations: limitations: 1.	The significance of Fig 1, the teaser image, is unclear; it neither reveals the structure of the entire paper nor distinguishes it from existing methods.
2.	The paper lacks consistency throughout, such as the discrepancy in some formula symbols in Fig 2 compared to those used later on.
3.	In constructing the tag libraries, DPC2022 was used, which has a broader coverage than AVA. Could you analyze potential data leakage issues?
4.	Prompt templates in equations 3 and 4 could be discussed in more detail, including a comparison of different templates.
5.	Equations 5 and 6 seem to have incorrect dimensions; the current notation suggests 'sim' is a scalar rather than a matrix.
6.	The need for prior semantic tag mining raises questions about fairness, and the paper appears not to compare with other studies that design anchor images.
7.	A clearer explanation of Balanced Sampling is recommended.
8.	The explanation for Contrastive Ranking structure could be more detailed.