"Special Relativity" of Image Aesthetics Assessment: a Preliminary Empirical Perspective

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:57:12

Abstract: Image aesthetics assessment (IAA) primarily examines image quality from a user-centric perspective and can be applied to guide various applications, including image capture, recommendation, and enhancement.
The fundamental issue in IAA revolves around the quantification of image aesthetics.
Existing methodologies rely on assigning a scalar (or a distribution) to represent aesthetic value based on conventional practices, which confines this scalar within a specific range and artificially labels it.
However, conventional methods rarely incorporate research on interpretability, particularly lacking systematic responses to the following three fundamental questions: 
1) Can aesthetic qualities be quantified? 
2) What is the nature of quantifying aesthetics? 
3) How can aesthetics be accurately quantified? 

In this paper, we present a law called "Special Relativity" of IAA (SR-IAA) that addresses the aforementioned core questions. We have developed a Multi-Attribute IAA Framework (MAINet), which serves as a preliminary validation for SR-IAA within the existing datasets and achieves state-of-the-art (SOTA) performance. Specifically, our metrics on multi-attribute assessment outperform the second-best performance by 8.06% (AADB), 1.67% (PARA), and 2.44% (SPAQ) in terms of SRCC. We anticipate that our research will offer innovative theoretical guidance to the IAA research community. Codes are available in the supplementary material.

Main paper: 
 introduction: 
 Image aesthetics assessment (IAA) is a fundamental task in the field of computer vision. IAA primarily examines the quality of images from a user-centric perspective, rather than primarily focusing on signal distortion as conducted by image quality assessment (IQA).In recent years, the surge in social media usage and advancements in digital photography have led to an increased demand for IAA, aiming to automatically evaluate whether an image aligns with users' aesthetic preferences [1] [2] [3] . In fact, IAA can be applied to guide various applications, including image capture (e.g., enhancing the photographic capabilities of smartphones and cameras [4] [5] [6] ), recommendation (e.g., recommending social medias or advertising), and enhancement (e.g., facilitating the creation of visually captivating images [7, 8] with the rapid advancement of AI-generated content).The prevailing IAA methodologies still resort to assigning a scalar (or regressing a distribution to a scalar with maximum probability) to represent aesthetic value based on conventional practices, which confines this scalar within a specific range and artificially labels it. However, these methodologies are predicated on an implicit assumption that all images can be linearly ranked according to their aesthetic qualities; however, this assumption may be ill-posed itself.To our knowledge, conventional methods rarely incorporate research on interpretability, particularly lacking systematic responses to the following three fundamental questions around the quantification of image aesthetics:I. Can aesthetic qualities be quantified? Aesthetics is an abstract concept that lacks concreteness and is characterized by its inherent uncertainty: aesthetic perceptions vary among viewers and can even fluctuate for the same viewer over time; moreover, randomness is often considered when evaluating image aesthetics. Some researchers argue that the scientific method, known for its objectivity and rationality, is inadequate for quantifying aesthetics due to its subjective and uncertain nature [9] . Therefore, quantifying aesthetics appears to be impractical without establishing certainty.II. What is the nature of quantifying aesthetics? The prevailing methods of quantification involve scoring images, which serves two essential purposes: positioning a sample within the entire sample space and assessing aesthetic differences between samples. Existing methods quantify aesthetics based on numerical scores or distribution and their differences, overlooking the non-transitive nature of IAA (confused by the transitivity of mathematical inequalities). Specifically, if image A is considered more aesthetically pleasing than image B, and image B more so than image C, it does not necessarily imply that image A is more pleasing than image C.III. How can aesthetics be accurately quantified? Current annotation methods are evidently inadequate for achieving this quantification because they conflate two essential purposes of quantification. Assigning a numerical score (e.g., ranging from 0 to 10) to an isolated sample is a considerable challenge, as it is difficult for viewers to determine an accurate score for both purposes of quantification. Furthermore, the act of confining quantification within a specific range is inherently illogical, as it can be easily disproven: we can always encounter an image that surpasses all the samples in the dataset in terms of either ugliness or beauty.This paper aims to offer comprehensive and systematic responses to the aforementioned three fundamental issues concerning aesthetic quantification, while also presenting conclusive findings and viable solutions. However, it is important to acknowledge that due to the limitations inherent in existing datasets and the length constraint of a conference paper, our verification can only be considered partial and preliminary in nature. Our contributions can be summarized as follows:-To the best of our knowledge, this study represents the first comprehensive and systematic analysis addressing three fundamental issues in IAA. -This paper proposes a law called "Special Relativity" of IAA (SR-IAA), which investigates three fundamental issues around quantifying aesthetics. The term "special" implies that the law is based on a framework constructed by aesthetic qualities, viewers, cameras, and photographers, but does not account for temporal (e.g. across eras or ages) or spacial (environmental) influences, nor dynamic alterations in human neuronal structures and their synaptic connections. -Guided by SR-IAA, we have developed a Multi-Attribute IAA Framework (MAINet), which serves as a preliminary validation for SR-IAA within the existing datasets and achieves SOTA performance. 
related works 2.1 quantification of aesthetics: 
 Possibility. The subjective nature of image aesthetics poses the primary challenge in IAA, as it can be influenced by cultural background, personal experiences, education attainment and psychological state [10] . Some researchers argue that the scientific methods may not be sufficient to quantify aesthetics due to its subjectivity and uncertainty [9] . Conversely, the presence of the "Law of Large Numbers" in statistics supports the feasibility of using objective computational methods. Additionally, some studies aim to integrate more objective criteria into IAA by considering attributes such as semantic content and artistic aspects, with the goal of providing a reliable basis for evaluating aesthetic properties of different photographs [11, 12] . According to the aforementioned viewpoints, the crux of the dispute over aesthetic quantification lies in determining if reliability can be derived amidst uncertainties. Existing methodologies mainly rely on statistical perspectives like "Law of Large Numbers" or "majority consensus". However, this kind of statistical certainty is not only significantly constrained by the number and representativeness of the sampled population, but also it lacks effectiveness in addressing personalized individuals. Methodologies. In current research, general IAA tasks are typically classified into three categories: binary classification (positive or negative aesthetics) [13, 14] , aesthetic score regression [15, 16] and score distribution prediction [17] [18] [19] . On the other hand, personalized IAA customizes an aesthetic model to suit individual user preferences [20, 21] . Correspondingly, existing datasets [22] [23] [24] [25] aim to collect more comprehensive scores by increasing the number of annotators or broaden the range of aesthetic attributes. Moreover, other studies [26, 27] have harnessed self-supervised learning paradigms to formulate image sequences exhibiting subtle aesthetic fluctuations for data augmentation.However, it is often overlooked that the prevailing methods of quantification involve scoring images, serving two essential purposes: positioning samples within the entire sample space and assessing aesthetic differences between them. Clearly, existing methods struggle to effectively balance these two purposes using a single scalar, often resulting in neglecting one or the other.Comparison-based methods [28] [29] [30] [31] [32] are useful to learn metrics for quantifying perceptual concepts, like urban appearance quantification [33] . In the past decade, comparison-based approaches have proven effective in the IAA field. Kong et al. [22] introduced a loss function called "pairwise ranking loss" to ensure accurate image ranking. Lee et al. [34] developed a unified approach for scoring regression, binary classification, and personalization based on comparison. Ko et al. [35] proposed using pairwise comparison to evaluate image aesthetics.However, the existing comparison-based methods, whether tailored for IAA [22, 34, 35] or more generally applicable in other fields [28] [29] [30] [31] [32] , although capable of providing relative quantitative metrics and circumventing the fixed range limitation of absolute quantitative metrics, presuppose an ordered data structure without considering the unique characteristic of "aesthetic inequality does not satisfy transitivity". This paper endeavors to address this issue through specialized processing of existing aesthetic datasets.Furthermore, drawing inspiration from Shin et al. [28] , who introduced the concept of "using two reference instances to form a window," and Burges et al. [29] , who posited that "the relative order of items is more significant than their absolute scores," we developed two strategies to effectively overcome the limitations imposed by reliance on absolute scores and have clearly delineated the purposes of aesthetic quantification. Firstly, we adaptively select reference images to position a sample within the entire sample space. Secondly, we assess aesthetic differences between samples by perceiving differences in aesthetic attributes.Multi-attribute IAA takes into account various attributes or features in an image, such as color, contrast, composition, etc. It incorporates the photographic rules, camera parameters, and viewer preferences more comprehensively and provides multi-attribute explanations. Early works [14, 36] utilized hand-crafted features based on standard photographic rules, such as the rule of thirds, colorfulness, or saturation, to distinguish between aesthetically pleasing and displeasing images. Recently, the availability of multi-attribute datasets [22] [23] [24] allow multi-column networks [37] and multi-task universal models [22, 38, 39] to evaluate various aesthetic attributes. Notably, Transformer or CLIP models [1, [40] [41] [42] [43] [44] have been widely applied to extract aesthetic information and map visual features to annotated labels. Multi-attribute IAA primarily focuses on measuring aesthetic differences rather than quantifying individual sample positions.However, aesthetic attributes are typically delineated based on individuals' perceptual knowledge, and comprehensively identifying all aesthetic attributes within the existing cognitive framework is essentially unattainable. Furthermore, the definition and selection of aesthetic attributes depend on distinct aesthetic values, thereby resulting in existing aesthetic datasets with inherent limitations in scalability and consistency. This significantly hinders any modifications or additions to aesthetic attributes, necessitating a comprehensive reworking of the datasets.To address the challenges of multi-attribute IAA while validating our law, we propose MAINet for more accurate generation of aesthetic scores. Additionally, guided by our law, the issue of modifying or adding aesthetic attributes is resolved in an unsupervised manner. 
law called sr-iaa: 
 Regarding a rational viewer: I. Image aesthetics can be quantified by consistently and definitively determining the relative preference between two comparable images within a given duration. However, aesthetic inequalities do not adhere to the transitivity observed in mathematical inequalities.Note: The duration varies among individuals depending on the extent to which their neurons are adequately stimulated to significantly influence their aesthetic preference; If the duration is brief without additional stimulus input, it indicates similar aesthetic in both images. Furthermore, the aesthetic comparability of any two images cannot be guaranteed, as evidenced by the contrasting themes of probability statistics and natural scenery.II. Quantifying image aesthetics primarily aims to simulate two abilities: perceiving aesthetics without any reference and perceiving aesthetics with a reference. Note: The first ability involves "locating" the input sample within an "experience sample" space, while the second ability involves calculating aesthetic differences between two input samples.III. Quantifying aesthetic perception without any reference can be represented by establishing pairwise relative relationships among N samples, while the quantification with a reference involves identifying variations in multiple aesthetic attributes between two samples.Note: The value of N should be large enough, with an aesthetically uniformly distributed sample set; the aesthetic attributes, which are defined and selected based on individual aesthetic values, can all be assigned to three roles [45] : camera, photographer, and viewer.Due to length constraints, we provide an outline of foundations underlying the law (hypothesis) here: 1) philosophically, SR-IAA is the only comprehensive sampling approach for assessing image aesthetics (addressing both goals of aesthetic quantification), unlike others (achieving at most one goal of aesthetic quantification), to our knowledge; 2) psychologically, the biological instinct of "stimulus response" and "seeking advantage and avoiding harm" manifests in psychology as the cognitive process of "pursuing beauty while avoiding ugliness"; 3) mathematically, aesthetic inequalities fail to satisfy the transitivity property; hence, traditional quantification by scoring is inherently inaccurate, while SR-IAA circumvents this issue. For further elucidation, please consult the Appendix A.1. It is conceivable that another future publication may be necessary to comprehensively expound and substantiate the underpinnings of SR-IAA; however, within the scope of this paper, proposing the law (hypotheses) suffices without compromising its integrity.Based on this premise, we posit that SR-IAA would be wellfounded, we propose a framework for multi-attribute IAA and aim to preliminarily validate the law by using it as guidance to enhance IAA performance on the existing datasets. 
mainet for validation 4.1 limitations and compromises: 
 The available datasets are primarily constrained by the following three limitations, necessitating tailored compromises to be made, which is why we refer to it as a "Preliminary Empirical Perspective." -Limitation I: Current data annotation systems linely rank images based on their scores which contradicts "SR-IAA I." Compromise I: Fortunately, we have observed that image sequences with a common theme exhibit a certain level of transitivity like mathematical inequalities, which is the fundamental reason behind the effectiveness of theme-based TANet [1] . Therefore, guided by "SR-IAA I & II," the aesthetic quality of an image can be approximated by comparing it to other ones with the same theme that are more or less pleasing. -Limitation II: Existing datasets exhibit limited scalability in terms of new or redefined multi-attributes.Compromise II: The aesthetic difference between images, as stated in "SR-IAA III," is determined by multiple attributes.To measure these attributes, the idea of "Compromise I" can also be employed. When extending to a new or redefined attribute, quantifying the editing degree of an image can approximate the difference in aesthetic attributes between the original and edited versions. This approach enables unsupervised scalability of aesthetic attributes. -Limitation III: The majority of available datasets are annotated using average ratings from multiple assessors.Compromise III: The average rating can be approximated to that of a "being smoothed" viewer, although this approximation may present new challenges (refer to Appendix A.1.III).For any given input image 𝑥, MAINet necessitates the identification of a lower reference image (𝑅 𝑙 ) and an upper reference image (𝑅 𝑢 ) within a common theme in order to establish aesthetic boundaries. Subsequently, we conduct a regression task to evaluate aesthetic differences within a defined triplet {𝑥, 𝑅 𝑙 , 𝑅 𝑢 }.EQUATIONwhere the set 𝑆 = {𝑥, 𝑅 𝑙 , 𝑅 𝑟 } represents the triplet, 𝑦 is the ground truth, and 𝜃 is the parameter to be optimized for model M.The architecture is illustrated in Fig. 1 . Given an RGB image 𝑥 ∈ R 𝐻 ×𝑊 ×𝐶 , the Reference Selection Module (RSM) selects two reference images within a common theme, forming a triplet denoted as 𝑆 ∈ R 3×𝐻 ×𝑊 ×𝐶 . Subsequently, the images in 𝑆 are split into patches of size 𝐻 32 × 𝑊 32 using a patchify strategy. These patches are then processed by three modules: the Position Identification Module (PIM), which assigns each patch a position through a learnable matrix 𝑃 ∈ R 3× 𝐻 32 × 𝑊 32 ; the Patch Embedding Module (PEM), which converts these patches into embeddings; and finally, the Attribute Perception Module (APM), which analyzes their attributes for comparison purposes. Ultimately, these module outputs are integrated and fed into a decoder to predict differences in aesthetic quality between images before being converted into an aesthetic score.The task assigned to RSM, in accordance with Compromis I, is to select two reference images that share a common theme with an input image. To simplify this process, RSM has been designed to progressively master four-level comparison tasks. The complexity of each task depends } , where the index range of 𝑖 is from 1 to N (the total number of input images) and the tiers span from 1 to 4. The detailed construction method for this dataset can be found in Algorithm 1.𝑃 𝑃 𝑃 𝑃 •• •• •• •• 𝑃 𝑃 𝑃 𝑃 •• •• •• •• Aesthetic Difference Aesthetic Score Feed ForwardIt is essential to note that we ensured thematic consistency during the dataset construction phase. In selecting images, we utilized a pre-trained Theme Understanding Network [1] to filter and compile image combinations with similar themes as the final training data for the RSM. The considered themes span across 47 categories, including landscape, plant, and human. During the inference stage, we employ a consistent strategy to ensure that the reference images compared with the input images share the common theme.In the training phase, RSM sequentially approaches tasks from the simplest (𝑒 = 4) to the most complex (𝑒 = 1) training datasets. For each difficulty level 𝑒, RSM processes a training sample (𝑥,𝑇 𝑒 𝑙 ,𝑇 𝑒 𝑢 ) and generates the output (𝐵 𝑒 𝑙 , 𝐵 𝑒 𝑢 ). In the final level, the gap between 𝑇 1𝑙 and 𝑇 1 𝑢 is 1. This process can be described by the following formulation:EQUATIONFinally, after sequential training across four tiers, the trained RSM is capable of predicting matched boundaries (𝐵 By employing a four-tier training strategy, we innovatively endowed the model with the capability for self-correction. That is, although the model may err in the predictions at each tier, the likelihood of errors occurring across all four tiers is minimal. This significantly enhances the accuracy of the network's final prediction, thereby establishing certainty in aesthetic assessments. Moreover, selecting images of the common theme narrows the scope of the solution space to compare within the common thematic context. 𝑖 ← 𝑖 + 1 19: end while 4.3.2 Attribute Perception Module (APM). Aesthetic differences between two images are determined by aesthetic attributes (refer to "SR-IAA III"), which should be included in aesthetic comparisons [46] . According to Compromis II, APM uses a self-supervised training approach to predict editing operations and intensities, enhancing its comprehension of different attributes.The latest research reveals that specific image editing operations have distinct effects on image attributes, influencing the aesthetics [26, 27] . Identifying the types and parameters of these editing operations applied to synthetic images can help models understand aesthetic features. With this goal in mind, we introduce certain tasks.Fig. 3 demonstrates the utilization of 21 operations 𝑂 to identify the 7 attributes. These operations can be adjusted at 8 different intensity levels 𝐿, with cropping operations having only 2 intensity levels. The effectiveness of operations with different intensity levels can be consulted in Appendix A.4. It should be noted that some operation intensenty levels appear excessively strong, making them improbable in real photographs (for example, Gaussian noise levels of 50). However, it is essential to clarify that APM does not predict aesthetic score; instead, it solely predicts operation intensity. The employment of high-intensity parameters to ensure coverage of all possible extreme scenarios in real-world situations and to broaden the applicability of APM's feature perception is thoroughly justified.Simultaneously, it is important to note that operation intensity does not directly correlate with aesthetic score, a higher level of operation intensity does not necessarily imply a superior aesthetic score. Therefore, operation intensity is not used as a direct basis for aesthetic comparison but rather serves as additional information for the decoder.For dataset construction, a sample is defined as a triplet ( 𝑥, 𝑜, 𝑙), where (𝑜, 𝑙) indicates that the image 𝑥 was edited using operation 𝑜 ∈ 𝑂 at parameter level 𝑙 ∈ 𝐿. We refer to the dataset as the "augmented dataset" and validate its effectiveness in the ablation study section.APM processes patches 𝑝 extracted from synthetic image 𝑥, and then predicts the probability distribution for the type and intensity level of each operation applied to these patches. We calculate the prediction losses using categorical cross-entropy:EQUATIONwhere 𝑜 and 𝑙 is respectively the ground-truth operation and intensities, while their predictions are represented by ô and l. The APM outputs an embedding of size 21 × 8 × 𝐷.After obtaining the reference images 𝑅 𝑙 , 𝑅 𝑢 of the input image 𝑥, we divide the triplet 𝑆 ∈ R 3×𝐻 ×𝑊 ×𝐶 into image patches of size 32 × 32. These patches are then encoded by PEM, with each patch linearly embedded by a 5-layer ResNet [47] (Fig. 1 ).In the classical transformer architecture, 2D-aware position embeddings are typically added to patch embeddings for preserving positional information. However, when dealing with triplets of images as input, it becomes easy to confuse patches from different images, potentially leading to comparison errors. To address this issue, we incorporate 3D patch positional information (PPI), enabling us to accurately determine the source image and corresponding position for each patch using a 3 × 𝐷-dimensional embedding. Suppose the input resolution is 𝐻 × 𝑊 , triplet 𝑆 will be split into 3 × 𝐻 32 × 𝑊 32 patches. We define PPI by a learnable matrix 𝑃 ∈ R 3× 𝐻 32 × 𝑊 32 , for the patch at position i-th image of 𝑆, j-th row and k-th column, its position embedding is defined by the element at position 𝑃 𝑖,𝑗,𝑘 .The decoder of MAINet processes embedding tokens to predict the aesthetic difference of the triplet.I. Structure of Decoder. It receives various embedding tokens from PEM, PIM and APM, and consists of alternating layers of multi-head self-attention (MSA) and MLP blocks. LayerNorm (LN) is applied before every block, and residual connections after every block.We append a [𝑐𝑙𝑎𝑠𝑠] token at the beginning of the token sequence to capture important features from the entire sequence. Finally, we use MLPs with two layers and GELU non-linearity to predict image aesthetics:EQUATIONwhere the output of [𝑐𝑙𝑎𝑠𝑠] token after the decoder, denoted as 𝑧 0 𝐿 , serves as the final output and is calculated as follows:EQUATIONThe current datasets need to be reconstructed for the decoder to predict aesthetic differences within the existing data framework.The given triplet sample (𝑥, 𝑅 𝑙 , 𝑅 𝑢 ), (𝑦, 𝑦 𝑙 , 𝑦 𝑢 ), consisting of input image 𝑥 and two reference images 𝑅 𝑙 and 𝑅 𝑢 , along with their corresponding ground truth values 𝑦, 𝑦 𝑙 , 𝑦 𝑢 , is converted into an aesthetic difference value 𝑦 𝑡 by Eq. (6) .EQUATIONHowever, if the dataset lacks manually labeled ground truth, we provide annotations based on the following assumption: any modifications made to high-quality images will degrade their original aesthetics. The extent of aesthetic degradation is relevant to the degree of image degradation. To create an "Original Triplet", we collect high-quality images and subject them to various synthetic operations and intensity levels.For example, we rate the original image with high aesthetics as 10 which means without any degradation. When we apply two intensity levels of saturation operation (0.8 and 0.6), we produce two synthetic images with aesthetic levels of 2 (= 10 × (1 -0.8)) and 4 (= 10 × (1 -0.6)) respectively, in comparison to the original image. The values 2 and 4 are called as "aesthetic difference levels" rather than "aesthetic scores". This triplet can also be represented as 𝑆 𝑟 = {(𝑥, 𝑅 𝑙 , 𝑅 𝑢 ), (𝑦, 𝑦 𝑙 , 𝑦 𝑢 )} and can be transformed by Eq. ( 6). Finally, the training sample can be uniformly represented as 𝑆 𝑡 = {(𝑥, 𝑅 𝑙 , 𝑅 𝑢 ), 𝑦 𝑡 }.During the training phase, both RSM and APM will be frozen; optimization will solely focus on PEM, PIM, and the decoder. The MAE loss is utilized to determine the difference in image aesthetics:EQUATIONFinally, in order to maintain consistency with other methods, we convert the aesthetic difference 𝑦 𝑡 into an aesthetic score by applying Eq. ( 6) in reverse. 
experiments 5.1 settings: 
 Training Phases of MAINet. MAINet consists of five learnable components: RSM, PIM, APM, PEM, and the decoder. To ensure stable training, we adopt a three-stage progressive approach. In the initial stage, our focus is on training the crucial RSM component for predicting matched boundaries and selecting reference images. Once this stage is completed, we freeze the parameters of RSM. Moving to the second stage involves self-supervised APM training to enhance attribute comprehension within the model. After completing this phase, we also freeze the parameters of APM. Finally, in the third stage of our approach, we train PIM, PEM and decoder components to predict aesthetic differences in images. Datasets. We evaluate models on three datasets: AADB [22] , SPAQ [23] , and PARA [24] . The dataset has no special requirements in the first and third stages. For the dataset containing 𝑁 samples, we apply Algorithm 1 to generate 4𝑁 samples and split them into 0.8 × 4𝑁 samples for training and 0.2 × 4𝑁 samples for testing. In the second stage, we extract 600 images with aesthetic scores of 8.0 or higher from the AADB dataset. Following the methodology delineated in Sec. 4.3.2, we construct a dataset consisting of 93,600 samples. This dataset is subsequently split into training and testing sets, with 20% allocated for testing. Evaluation Metrics. The performance is assessed using three popular evaluation metrics: Spearman rank correlation coefficient (SRCC, S), Linear correlation coefficient (LCC, L), and binary classification accuracy (ACC, A). These metrics are essential for measuring regression performance and evaluating the model's capability in classifying aesthetic qualities as negative or positive. Benchmark Models. We compare our method to 9 SOTA IAA models [1, 16, 17, 22, 41, 43, [48] [49] [50] . In selecting benchmarks, we prioritize generalization across multi-attributes and datasets rather than extreme optimization for a specific dataset; hence, we retrain 8 previously mentioned methods [1, 16, 17, 22, 41, 43, 48, 50] on multi-attribute datasets. Given that both our team and our peers in the field have faced the common bugs in reproducing the results of VILA [51] , its exclusion from our analysis is regrettably unavoidable.Table 1 : Comparison of the assessment effects of labeled atributes on the AADB, PARA and SPAQ datasets, where "Ours (SSL)" is assessed solely on self-supervision after training, with the red and blue bold numbers indicating the best and second best results.Furthermore, the absence of available code for TAVAR [49] necessitates our reliance on the metrics reported by TAVAR for comparative analysis with our method, as detailed in Table 2 . Implementation Details. The patch size is set to 32 during the training phase. The dimension 𝐷 for the decoder input tokens is fixed at 384, which is consistent across all embeddings. We use SGD with a momentum of 0.9 and initiate the learning rate at 10 -4 .Attribute Assessment on Labelled Data. Table 1 presents the results on the three multi-attribute datasets. Our method outperforms other models for all attributes, surpassing the second-best method by an average of 4.48% in terms of SRCC. Attribute Assessment on Unlabelled Data. To our knowledge, we introduce the first method capable of label-free evaluation of aesthetic attributes through comparison. To evaluate the performance, we train MAINet on three multi-attribute datasets with self-supervised learning (SSL), as detailed in Sec. 4.3.5, whereas labeled information is exclusively used for testing. As illustrated in Table 1 , despite being trained on unlabeled data, our method achieves equal or superior performance compared to other methods trained on labeled data, thereby confirming its effectiveness. Cross-dataset Evaluation. To evaluate the generalization ability of MAINet, the cross-dataset evaluations are conducted on four datasets. Table 2 demonstrates that MAINet outperforms on most datasets, especially when trained on a small dataset (AADB) with limited images. in an 11.7% increase in SRCC and a 14.9% increase in LCC compared to randomly selected ones. This logical finding implies that randomly chosen reference images often have aesthetic disparities or fail to adequately position the input image, thus lacking valuable information for aesthetic comparisons. Effectiveness of PIM. In addition, incorporating PIM enhances both SRCC and LCC by 1.8% and 3.4%, respectively. To further validate the reliance of our comparative framework on PIM, we also employed traditional 2D positional coding as an alternative to PIM, resulting in a decrease of 1.1% in SRCC and 2.3% in LCC. The unsatisfactory performance can be attributed to the model's inability to effectively capture the source information of image blocks, thereby causing confusion during the comparison process. Effectiveness of APM. Finally, the incorporation of APM and augmented dataset results in a significant improvement of 3.1% and 4.5% in SRCC and LCC, respectively. This confirms that including attribute information enhances the model's accuracy in aesthetics assessment. Moreover, a decrease in the intensity of image editing operations yields a corresponding decline of 3.1% in SRCC and 2.7% in LCC, as illustrated in row 6 of Table 3 .Predictions for Images. The examples in Fig. 4 illustrate the multiattribute IAA predictions. Similar to human cognition, MAINet assigns higher scores to images that exhibit superior performance across the majority of attributes. Saliency Maps. The Grad-CAM is applied to visualize saliency maps in Fig. 5 . MAINet can perceive multi-attribute features differentially, such as the DoF attribute which focuses on areas with salient objects and Vivid Color attribute which focuses on areas with vibrant colors.  
conclusion: 
 This paper focuses on quantifying aesthetics and proposes a law called SR-IAA. To validate this law, we have developed a multiattribute IAA framework that incorporates modules for automated reference image selection and integration of aesthetic attributes into the comparison process. To ensure reliable training, we have devised a novel scheme for constructing datasets and a progressive training strategy. Experimental results show that MAINet, guided by our proposed law, outperforms conventional state-of-the-art methods.However, MAINet only serves as a preliminary verification method but entails inherent compromises due to the limitations in existing datasets. Additionally, the limited availability of data that fulfills the requirements of SR-IAA, inadequate experimental validation, and length constraints impede this paper from establishing a comprehensive groundwork for SR-IAA; therefore, we eagerly anticipate undertaking an in-depth exploration of this subject matter in another future publication. 
 
 Appendix:  


Decision: The final ratings are 1 borderline reject (R1), 1 weak accept (R2), and 1 accept. The reviewers acknowledged that the proposed method is interesting and shows good performance. However, R1 raised a concern in the final justification that the SR-IAA law lacks sufficient evidence/justification and the proposed model is not directly related to the SR-IAA law. After carefully checking the comments, paper, and rebuttal, AC finds this concern is reasonable to some extent.

Reviews: review: paper_summary: paper_summary: This paper proposes a law composed of three elements to address three fundamental questions in Image Aesthetics Assessment (IAA). Based on this law, it introduces an IAA model called MAINet. MAINet consists of five modules and takes triplets as input to determine aesthetic scores. Experimental results demonstrate that MAINet outperforms existing methods in attribute assessment.
strength_weakness: strength_weakness: The proposed method achieves superior performance in attribute assessment compared to existing methods.
limitations: limitations: This paper raises fundamental questions regarding IAA and proposes laws and an IAA model that could address these questions. However, it is difficult to say that the questions have been adequately answered, and it is also challenging to see that the proposed methods were developed based on these answers.

1. The author argues in the first question that quantifying aesthetics is highly subjective and that it is necessary to establish certainty. As a method for this, they propose the RSM to create Triplets. However, can this be considered a sufficient method for quantifying aesthetics? Since it only evaluates good and bad within images of the same class, it cannot provide a fundamental answer for quantifying aesthetics. Moreover, the author is simply using existing datasets, which already use ground truth scores determined by subjective criteria. Therefore, demonstrating good performance in the context of existing datasets cannot be considered a fundamental answer to the first question. A more convincing answer is needed for the first question, and high performance in cross-dataset evaluation is not a enough answer either.

2. In the second question, the author mentions the non-transitive nature of IAA, but the method used in the paper is still based on numerical scores. Since the paper proceeds by stating in Compromise I that 'a common theme exhibits a certain level of transitivity,' it cannot be considered an adequate answer to the second question.

3. SR-IAA II and III address cases without references, but MAINet appears to be a model proposed primarily for cases with references, based on Triplets. Where can we find experimental or practical details regarding cases without references?

4. What is the reason for training RSM? The data already has ground truth scores, so it seems like we could simply group the data within the same class according to these scores. Why is training involved?

5. In SR-IAA III, it is mentioned that two images are compared based on multiple aesthetic attributes, and it seems that APM aims to reflect these aspects. However, I believe the attributes mentioned in SR-IAA III should be more comprehensive than just noise or color effects in Figure 3. I think the elements in Figure 3 are closer to IQA than IAA.

6. What is the reason for training PIM? Even with Triplets, the order of the data is fixed, so it doesn't seem like there would be any mixing of patches, even if you used a common sinusoidal position embedding.

7. The reason for the superior performance of the proposed method in Table 1 seems to be that it separately learns multiple attributes through APM, while the other methods do not, leading to lower performance. Therefore, it does not seem to be a fair comparison.

8. Regarding IAA performance, the proposed method scores 0.78 on AADB in Table 2, while EAT scores 0.77, which does not appear significantly different from existing methods.
paper_summary: paper_summary: The article presents a novel approach to image aesthetics assessment (IAA) termed the "Special Relativity" of IAA (SR-IAA), which addresses key questions about the quantification of aesthetic qualities. The authors introduce a Multi-Attribute IAA Framework (MAINet) that serves as an empirical validation of SR-IAA, achieving state-of-the-art performance on existing datasets. MAINet is designed to simulate two abilities of human perception: assessing aesthetics without reference and with reference, and it incorporates automated reference image selection and aesthetic attribute integration. The framework is trained using a progressive strategy, and experiments demonstrate its superiority over other IAA methods. The paper concludes by acknowledging the preliminary nature of the work and the limitations imposed by existing datasets, suggesting further exploration is needed.
strength_weakness: strength_weakness: This is an interesting work that conducts in-depth analysis of quantitative issues for IAA, and discusses and analyzes from three special perspectives: philosophy, psychology and mathematics.
1. **Innovative Framework**: The paper introduces a novel framework, MAINet, which is guided by the proposed "Special Relativity" of Image Aesthetics Assessment (SR-IAA). This framework aims to address the fundamental questions of quantifying aesthetics and offers a new perspective on image aesthetics assessment.
2. **State-of-the-Art Performance**: The authors claim that their framework achieves state-of-the-art performance on existing datasets, outperforming the second-best performance by significant margins in terms of Spearman Rank Correlation Coefficient (SRCC). This suggests that their approach is highly effective and pushes the boundaries of current research in this field.
3. **Comprehensive Analysis**: The paper provides a comprehensive analysis of the existing methodologies and their limitations in the context of image aesthetics assessment. It systematically addresses three fundamental questions related to aesthetic quantification and provides a structured response to each, demonstrating a deep understanding of the subject matter.

These strengths highlight the paper's potential contribution to the field of image aesthetics assessment through its innovative approach, empirical validation, and in-depth analysis.
limitations: limitations: The article presents a study on image aesthetics assessment (IAA) and introduces a framework called MAINet. However, it acknowledges several limitations:
1. **Dataset Limitations**: The article mentions that existing datasets primarily rank images based on scores, which contradicts the principles of the "Special Relativity" of IAA (SR-IAA). The authors had to make compromises to align the dataset with their framework, which may affect the generalizability of the results.
2. **Scalability of Multi-Attributes**: The existing datasets have limited scalability when it comes to incorporating new or redefined multi-attributes. The authors propose a method to approximate the difference in aesthetic attributes between original and edited images, but this approach may not fully capture the complexity of aesthetic perception.
3. **Reliance on Average Ratings**: Most available datasets are annotated using average ratings from multiple assessors. The paper suggests that this approach can be approximated to represent a "smoothed" viewer, but it may introduce new challenges and might not accurately reflect individual differences in aesthetic perception.
paper_summary: paper_summary: This study delves into the fundamental issue of image aesthetic assessment, specifically the challenge of quantifying image aesthetics.

The paper first address 3 key questions regarding whether aesthetic quality can be accurately quantified, the essence of quantifying aesthetics, and how aesthetics can be precisely measured, and then propose the “Special Relativity” of Image Aesthetics Assessment (SR-IAA). This principle encompasses the following three main points:
(1) Image aesthetic quality can be derived through the comparison of two images, i.e., through a referential assessment.
(2) Quantification of aesthetics can be categorized into no-reference aesthetics and reference-based aesthetics.
(3) No-reference aesthetics evaluation focuses on overall scores, while reference-based aesthetics assesses changes in multiple aesthetic attributes between two images.

Based on above mentioned, the paper establishes an algorithmic framework for multi-attribute aesthetic assessment, and achieving promising performance on conventional benchmarks.

For selecting reference images for aesthetic images to be evaluated, a Reference Selection Module (RSM) is constructed. This module selects reference upper and lower bound images with quality scores gradually approaching from data of the same theme. An Attribute Perception Module (APM) is built to predict various degradation operations and levels, as well as aesthetic-related attribute changes. The learning process is conducted through a self-supervised manner.
strength_weakness: strength_weakness: (1)	The paper attempts to address in more depth the important but understudied fundamental issues of IAA for the first time, conducting a thorough analysis and providing solutions.

(2)	The authors proposed an interesting framework that is novel and achieves a promising performance. 

(3)	The experimental results are impressive, demonstrating that MAINet, guided by the proposed law, significantly outperforms conventional state-of-the-art methods of Multi-attribute IAA.

(4)	The paper is well-organized and easy to understand.
limitations: limitations: (1)	It would be beneficial to include additional descriptions of the entire assessment process of RSM and APM.

(2)	Given the constraints inherent in the current datasets, this paper employed specialized processing techniques on existing aesthetic datasets to align them with the theoretical framework outlined. However, It is advisable for the authors to elaborate on potential solutions and enhancements while addressing the current limitations.

(3)	Typos:

-In the section “Importance of RSM,” the “Compromise I” should be bolded to maintain a consistent style with the preceding text.

-There should be no space before the first paragraph of each section; please revise the section 3.

-Similar formatting glitches are present in the subsections, such as in section 4.3.2.