Align-IQA: Aligning Image Quality Assessment Models with Diverse Human Preferences via Customizable Guidance

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 12:46:50

Abstract: The alignment of the image quality assessment (IQA) model with diverse human preferences remains a challenge, owing to the variability in preferences for different types of visual content, including user-generated and AI-generated content (AIGC), etc. Despite the significant success of existing IQA methods in assessing specific visual content by leveraging knowledge from pre-trained models, the intricate factors impacting final ratings and the specially designed network architecture of these methods result in gaps in their ability to accurately capture human preferences for novel visual content. To address this issue, we propose Align-IQA, a novel framework that aims to generate visual quality scores aligned with diverse human preferences for different types of visual content. Align-IQA contains two key designs: (1) A customizable quality-aware guidance injection module. By injecting specializable quality-aware prior knowledge into general-purpose pre-trained models, the proposed module guides the acquisition of quality-aware features and allows for different adjustments of features to be consistent with diverse human preferences for various types of visual content. (2) A multi-scale feature aggregation module. By simulating the multi-scale mechanism in the human visual system, the proposed module enables the extraction of a more comprehensive representation of quality-aware features from the human perception perspective. Extensive experimental results demonstrate that Align-IQA achieves comparable or better performance than SOTA methods. Notably, Align-IQA outperforms the previous best results on AIGC datasets, achieving PLCC of 0.890 (+3.73%) and 0.924 (+1.99%) on AGIQA-1K and AGIQA-3K. Additionally, Align-IQA reduces training parameters by 72.26% and inference overhead by 78.12% while maintaining SOTA performance.

Main paper: 
 introduction: 
 Image quality assessment (IQA) is an indispensable requirement in the fields of image processing and computer vision [35] . It focuses on dealing with different types of visual content, including natural images [30] that have undergone specific treatments such as compression, blurring, noise, etc., user-generated content (UGC) (e.g., images captured through smartphones) [20] , and the more recent popularity of AI-generated content (AIGC) (e.g., images produced using text-to-image models) [16] . Over the past few decades, considerable efforts have been invested in proposing various IQA methods. Depending on the necessity of reference images, they can be broadly classified into full-reference IQA (FR-IQA) [25] , reducedreference IQA (RR-IQA) [48] , and no-reference IQA (NR-IQA) [19] . In real-world scenarios where reference images are unavailable, NR-IQA is preferred.In recent years, deep learning-based NR-IQA methods [2, 38, 45 ] have demonstrated superior performance in evaluating authentically distorted images compared to traditional methods, such as NR-IQA based on natural scene statistics (NSS) [24, 34] . However, their generalization abilities are constrained by the limited size of existing IQA datasets. For instance, the largest authentic IQA dataset, FLIVE [41] , encompasses approximately 40,000 distorted images sourced from real-world scenarios. In contrast, the largest image recognition dataset, ImageNet [4] , contains more than 14 million labeled images. Accordingly, existing IQA datasets are too small to reflect the diversity and complexity of real-world distortions adequately.Researchers propose a range of "pre-training and fine-tuning" strategies to address the problem of limited IQA dataset size. As described in Fig. 1 (a), a straightforward way [1, 32] is to directly finetune models that are pre-trained on large-scale non-IQA datasets (e.g., ImageNet) for leveraging knowledge from other computer vision tasks (e.g., image recognition). Nonetheless, it should be noted that these pre-trained models are not specifically designed for IQA, and they focus more on learning semantic-aware representations rather than quality-aware representations. As depicted in Fig. 1 (b), another popular way [22, 29, 47] is to perform quality-aware pre-training, by generating numerous distorted images to simulate real-world degradation processes. Subsequently, a model is pretrained to identify the distortion type (and level) of these images using self-supervised learning. Since the pre-trained models have already acquired robust quality-aware representations, they could be easily transferred to the downstream NR-IQA task. However, training from scratch requires a lot of training time and computing resources. For instance, a quality-aware pre-training approach [47] based on MoCo-v2 [7] , takes 75 hours to train ResNet-50 [8] on Im-ageNet (regenerated by adding synthetic distortions to the image), using 8 Nvidia V100 GPUs.In summary, we identify three challenges. Firstly, the limited size of the IQA dataset presents difficulties in acquiring a substantial amount of training data. Secondly, there is a struggle to make the pre-trained model pay more attention to extracting quality-aware features rather than semantic-aware features while maintaining computational efficiency. Thirdly, due to the variability in human preferences, it is difficult for existing IQA methods to generate quality scores consistent with diverse human preferences for different types of visual content.To solve these challenges, we propose Align-IQA, a novel NR-IQA framework aimed at generating visual quality scores aligned with diverse human preferences for various types of visual content. First, we design a customizable quality-aware guidance injection module that integrates specializable quality-aware prior knowledge into general-purpose pre-trained models (e.g., vanilla ViT), where the model is frozen and only a few parameters are introduced for injection. For natural and UGC images, the characteristics of human visual system (HVS) (e.g., visual saliency [33] ) are utilized to guide the pre-trained models in learning quality-aware features. In the case of AIGC images, guidance is provided by the visual-semantic relation information obtained from vision-language models. Second, since there are numerous visual natures to affect human perception of image quality [12] , we further design a multi-scale feature aggregation module. By simulating the multi-scale mechanism in the HVS, the module enables the extraction of a more comprehensive representation of quality-aware features from the human perception perspective. Specifically, we combine depth-wise separable and dilated convolution, to implement multi-scale feature extraction with a small number of model parameters. Finally, we conduct extensive evaluation and analysis to verify the efficiency and effectiveness of the proposed two designs in Align-IQA. The experimental results show that Align-IQA achieves comparable or better performance than SOTA methods. The main contributions of Align-IQA are summarized as follows:• We explore a novel NR-IQA model for generating visual quality scores aligned with diverse human preferences for different types of visual content. By using the proposed customizable quality-aware guidance injection module, our method ensures integrating personalized quality-aware prior knowledge into pre-trained models in the frozen state to obtain various quality-aware features for different types of visual content. • We design a novel multi-scale feature aggregation module.By simulating the multi-scale mechanism in the HVS, the module enables the extraction of a more comprehensive representation of quality-aware features from the human perception perspective. Furthermore, by leveraging depthwise dilated separable convolution, the module allows multiscale features to be obtained with only a small number of model parameters.  
related work: 
 The focus of IQA is on extracting quality-aware features aligned with human preference. Before the rise of deep learning, handcrafted feature engineering dominated the field of NR-IQA. The most common approach for manually extracting quality-aware features is to build NSS models. The assumption underlying this is that the feature distribution of undistorted natural scene images adheres to certain statistical regularities, which are corrupted by various distortions [28] . Therefore, the quality of distorted images can be quantified by modeling distortion-sensitive statistics of natural scenes, including discrete wavelet transform coefficients [34] , locally normalized luminance coefficients [23] , and correlation coefficients of subbands [24] , etc. Besides, some methods [37, 40] have also made beginning efforts to explore the automatic extraction of quality-aware features from distorted images, typically relying on visual codebooks. Recently, a variety of deep learning-based methods have been further developed to better automatically extract quality-aware features, significantly improving performance in evaluating realworld distortions. In the pioneering work [11] , a shallow network of only one convolutional layer is used for NR-IQA. Naturally, later works transition the network architecture from shallow to deep [2, 21] . Because of its non-local self-attention mechanism, ViT was recently adopted to design the NR-IQA model [12, 39] . Concurrently, the CLIP-based method [45] is also sprouting up, based on a language and vision model [27] can exploit the label semantic.In addition to well-designed models for extracting quality-aware features, a few studies pay special attention to solving the main challenge of deep learning-based NR-IQA methods: the limited size of existing IQA datasets. Among them, some [22, 29, 47] attempt to perform quality-aware pre-training via self-supervised learning (e.g., contrastive learning [3] ), based on the large-scale IQA-related dataset created by synthesizing the distorted image. Meanwhile, others [1, 32] resort to directly fine-tuning the model pre-trained The overall architecture of Align-IQA. First, the input image is passed through the patch embedding layer to obtain image embedding. Among these, image embedding is fed into 𝐿 transformer layers. Then, customizable quality-aware prior knowledge is mapped to a feature embedding of 𝐷 dimensions through an additional patch embedding layer, in conjunction with image embedding to the customizable guidance injector (CGI) for generating and injecting the specializable guidance into the transformer blocks for guiding the ViT to focus more on extract quality-aware features rather than semantic-aware features. The multi-scale feature aggregator (MSFA) extracts and integrates multi-scale features from the multi-level outputs of ViT. Finally, the global-local quality-aware features are sent to the dual-branch predictor for final quality prediction.on large-scale non-IQA datasets, aiming to leverage the knowledge from other computer vision tasks.We summarize the differences between Align-IQA and the abovementioned methods. First, different from contrastive learning-based methods [22, 29, 47] , Align-IQA does not require re-training from scratch on plenty of IQA-related images, thus avoiding significant computational overhead for re-training. Second, in contrast to methods [1, 32] , Align-IQA allows the pre-trained model to prioritize the acquisition of quality-aware features over semantic-aware features, boosting the ability of the model to learn quality-aware representations. Furthermore, by introducing customizable guidance, Align-IQA generates visual quality scores consistent with diverse human preferences for different types of visual content. 
proposed method 3.1 overall architecture: 
 The proposed Align-IQA is designed to generate visual quality scores aligned with diverse human preferences for different types of visual content. Specifically, for natural and UGC images, the characteristics of HVS (e.g., visual saliency [33] ) are utilized to guide the pre-trained models in learning quality-aware features. In the case of AIGC images, guidance is provided by the visual-semantic relation information obtained from vision-language models (e.g., [36] ). Align-IQA maintains the transformer encoder of general-purpose pre-trained ViT in a frozen state and contains only a limited set of tunable parameters to learn quality-aware features from the frozen image embedding, guided by customizable quality-aware prior knowledge. The overall architecture of our Align-IQA is illustrated in Fig. 2 .From a hierarchical perspective, we propose the framework of encoder -[customizable guidance injector] -[multi-scale feature aggregator] -dual-branch predictor. The encoder is a vanilla ViT, which consists of a patch embedding layer followed by 𝐿 transformer layers (see Fig. 2(a) ). We design a novel customizable guidance injector, as depicted in Fig. 2(b) , to inject quality-aware prior knowledge into ViT. Additionally, we propose a novel multi-scale feature aggregator, shown in Fig. 2(c ), which extracts multi-scale features from the multi-level outputs of ViT and fuses them to generate global-local quality-aware features. Finally, the global-local quality-aware features are sent to the dual-branch predictor for obtaining the final quality score.Due to the variability in human preferences for different types of visual content, we propose the customizable guidance injector (CGI). The main goal of CGI is to incorporate specializable qualityaware prior knowledge into the input of transformer layers, thereby preserving the original architecture of ViT. Notably, through the implementation of CGI, Align-IQA can effectively guide the generalpurpose pre-trained ViT to focus more on extracting quality-aware features rather than semantic-aware features. As depicted in Fig. 2 , the proposed CGI is integrated into each layer of the backbone network. Formally, providing token sequences {𝐸 𝑙 𝑖𝑚𝑔 , 𝐸 𝑝𝑟𝑖𝑜𝑟 } consist of the frozen image embedding 𝐸 𝑙 𝑖𝑚𝑔 and tunable feature embedding 𝐸 𝑝𝑟𝑖𝑜𝑟 mapped by quality-aware prior knowledge. The designed CGI to generate guidance with these token sequences, the process can be written as:EQUATIONwhere 𝐵 𝑙 denotes the 𝑙-th CGI block. In this way, CGI makes full use of quality-aware prior knowledge to generate effective guidance. The detailed design of CGI is depicted in Fig. 2 (b), our CGI has two input branches for introducing the token sequences of the frozen image embedding 𝐸 𝑙 𝑖𝑚𝑔 and tunable feature embedding 𝐸 𝑝𝑟𝑖𝑜𝑟 , respectively. Specifically, for the 𝑖-th transformer layer of ViT, we take the tunable feature embedding 𝐸 𝑝𝑟𝑖𝑜𝑟 as the query, and the frozen image embedding 𝐸 𝑙 𝑖𝑚𝑔 as the key and value. We use multi-head self-attention to incorporate tunable feature embedding 𝐸 𝑝𝑟𝑖𝑜𝑟 into the frozen image embedding 𝐸 𝑙 𝑖𝑚𝑔 , which can be written as:EQUATIONEQUATIONEQUATIONwhere 𝑛𝑜𝑟𝑚 (•) is LayerNorm, 𝑊 𝑗𝑞 , 𝑊 𝑗𝑘 , 𝑊 𝑗 𝑣 , 𝑊 𝑜 are projection functions, ℎ = 8, ℎ𝑒𝑎𝑑 𝑗 = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝐸 𝑗𝑞 , 𝐸 𝑗𝑘 , 𝐸 𝑗 𝑣 ), 𝑑 𝑘 is the dimension of the key. Furthermore, we apply a learnable parameter 𝛾 𝑖 to balance the tunable guidance embedding 𝑃 𝑙 and the frozen image embedding 𝐸 𝑙 𝑖𝑚𝑔 , which is initialized as 7.68 × 10 -3 . As a result, the feature distribution of 𝐸 𝑙 𝑖𝑚𝑔 is not significantly altered because of the injection of quality-aware prior knowledge, thereby optimizing the utilization of the pre-trained weights of ViT.In the human visual system, there are numerous visual natures to affect human perception of image quality. To better predict quality scores from the human perception perspective, we propose the multi-scale feature aggregator (MSFA). In particular, ViT is divided uniformly into 𝑁 (usually 𝑁 = 4) blocks, each block containing 𝐿/𝑁 transformer layers. After injecting quality-aware prior knowledge into ViT, we obtain four output features, 𝐸 3 𝑖𝑚𝑔 ,𝐸 6 𝑖𝑚𝑔 ,𝐸 9 𝑖𝑚𝑔 ,𝐸 12 𝑖𝑚𝑔 , by processing the frozen image embedding 𝐸 𝑙 𝑖𝑚𝑔 through the transformer layers of blocks 1, 2, 3, and 4. Then, we apply a module that combines depth-wise separable convolution and dilated convolution to extract multi-scale features. In this way, by replacing the standard convolution with depth-wise dilated separable convolutions, our MSFA significantly reduces the number of parameters and computational costs. As shown in Fig. 2 (c ), the proposed MSFA consists of three main components: (i) depth-wise dilated separable convolution operations that capture multi-scale features; (ii) a cross-level aggregation operation for synthesizing cross-level features; (iii) a 1 × 1 convolution operation for the fusion of multi-scale features and cross-level features.Concretely, there is a 1 × 1 standard convolution and three 3 × 3 depth-wise dilated separable convolutions, which capture local features of different receptive fields by setting different dilation rates (𝑟 3 = 3, 𝑟 6 = 2, 𝑟 9 = 1, respectively). By introducing spatial inductive bias towards the enrichment of local information. Multiscale features can be obtained using the following equation:𝐹 𝑗 𝑖𝑚𝑔 =        𝐶𝑜𝑛𝑣 𝑟 𝑗 𝑑𝑤 𝐶𝑜𝑛𝑣 𝑝𝑤 𝐸 𝑗 𝑖𝑚𝑔 , 𝑗 = 3, 6, 9 𝐶𝑜𝑛𝑣 𝑝𝑤 𝐸 𝑗 𝑖𝑚𝑔 , 𝑗 = 12 (7)where 𝐶𝑜𝑛𝑣 𝑝𝑤 indicates a 1 × 1 point-wise convolution for dimensionality reduction, 𝐶𝑜𝑛𝑣 𝑟 𝑑𝑤 indicates a 3 × 3 depth-wise dilated convolution for capturing multi-scale feature, 𝑟 indexes the dilation rate.Furthermore, we utilize a cross-level aggregation operation to fuse cross-level features across multiple levels of ViT. This process is described as follows:EQUATIONEQUATION)where 𝐴𝑣𝑔𝑃𝑜𝑜𝑙 (•) is global average pooling, 𝐶𝑜𝑛𝑣 𝑅 1×1 (•) indicates a 1 × 1 convolution with ReLU activation function, 𝑈 𝑝𝑠𝑎𝑚𝑝𝑙𝑒 (•) indexes the bilinear interpolation.After that, we apply a convolution to coordinate multi-scale branches and the cross-level branch obtaining the integrated globallocal quality-aware feature 𝐹 𝑖𝑚𝑔 . The process is formulated as:EQUATIONF𝑖𝑚𝑔 = 𝐶𝑜𝑛𝑣 1×1 𝐹 𝑖𝑚𝑔 (11) where 𝐶𝑜𝑛𝑣 1×1 (•) indicates a 1 × 1 convolution function to fuse the multi-scale features and cross-level features.When assessing the quality of images, human observers tend to focus on conspicuous regions of the images, e.g., sharp edges. As a result, different image patches usually have diverse quality scores.To tackle the problem of different image patches exerting varying influences on the final quality score of the entire image, Align-IQA adopts the dual-branch structure of patch-weighted quality prediction, as delineated in AHIQ [14] . We obtain the final predicted quality score by the following equation:EQUATION)where 𝑁 denotes the number of patches for a single image, 𝑠 𝑖 denotes the quality score of the 𝑖-th patch, 𝑤 𝑖 denotes the corresponding weight.As shown in Fig. 3 , we visualize the feature maps of vanilla ViT and the proposed Align-IQA. We perform visualization analyses on a fast-fading distorted image and a JPEG compression distorted image, both of which are derived from the LIVE dataset. Compare Align-IQA ViT Align-IQA the outputs from each block in ViT with the corresponding outputs from each block of Align-IQA, the feature maps 𝐹 3 𝑓 𝑓 /𝐹 3 𝑗𝑝𝑒𝑔 to 𝐹 12 𝑓 𝑓 /𝐹 12 𝑗𝑝𝑒𝑔 increasingly concentrate on the target instance and exhibit enhanced visual-linguistic alignment within the semantic space, while the feature maps F 3 𝑓 𝑓 / F 3 𝑗𝑝𝑒𝑔 to F 12 𝑓 𝑓 / F 12 𝑗𝑝𝑒𝑔 do not demonstrate an increasing focus on semantic information, instead, they exhibit a balanced focus on both semantic elements and peripheral details such as edges. It indicates that CGI effectively injects qualityaware prior knowledge into ViT, thereby guiding it to prioritize the acquisition of quality-aware features over semantic-aware features. Impressively, unlike 𝐹 𝑎𝑙𝑙 𝑓 𝑓 and 𝐹 𝑎𝑙𝑙 𝑗𝑝𝑒𝑔 , in F𝑎𝑙𝑙 𝑓 𝑓 and F𝑎𝑙𝑙 𝑗𝑝𝑒𝑔 , regions of interest are more diffuse and broad, extending far beyond the semantic space itself, which indicates that MSFA can effectively capture global and local information, thereby enriching the representation of quality-aware features from the human perception perspective.3 f f F 6 f f F 9 f f F 1 2 f f F all f f F ˆall f f F 1 2 ˆf f F 9 ˆf f F 6 ˆf f F 3 ˆf f F 3 jpeg 
experiments 4.1 experimental setups: 
 4.1.1 Datasets. We evaluate the performance of Align-IQA on eight publicly available IQA datasets, including synthetic datasets such as LIVE [30] , CSIQ [15] , TID2013 [26] , and KADID-10K [18] , authentic datasets like CLIVE [6] and KonIQ-10K [9] , as well as AIGC datasets like AGIQA-1K [46] and AGIQA-3K [16] . LIVE consists of 779 distorted images generated from 29 reference images using five distortion types. CSIQ contains 866 distorted images obtained from 30 reference images. TID2013 consists of 3,000 distorted images sourced from 25 reference images, covering 24 distortion types. KADID-10K contains 10125 images synthetically distorted using 25 different distortions on 81 reference images. CLIVE comprises 1,162 real-world distorted images captured by various mobile devices. KonIQ-10K contains 10,073 images with diverse authentic distortions. AGIQA-1K consists of 1,080 images generated by two text-to-image (T2I) models. AGIQA-3K contains 2,982 AI-generated images produced by six T2I models. For each IQA dataset, 80% of the distorted images contained in it are randomly selected for training and the rest 20% are used to test. For synthetic and authentic datasets, we compare the proposed method with six fully fine-tuned NR-IQA models, including LIQE [45] , MANIQA [39] , MUSIQ [12] , UNIQUE [44] , KonCept [9] , and HyperIQA [31] , two quality-aware pre-trained models, including Re-IQA [29] , and CONTRIQUE [22] , two well-designed CNN-based models, including NSSADNN [38] , and WaDIQaM-NR [2] , as well as two hand-crafted feature-based models, including BIECON [13] , and HOSA [37] . We report the SRCC and PLCC results in Tab. 1. For AIGC datasets, the comparison results are reported in Tab. 2 and Tab. 3. From the results, we draw some conclusions. First, the quality-aware pre-training strategy enables CONTRIQUE and Re-IQA to outperform well-designed CNN-based WaDIQaM-NR, as well as UNIQUE based on a general-purpose pre-trained model. Second, by introducing textual modal information, LIQE performs much better than CONTRIQUE and Re-IQA on CLIVE and KonIQ-10K datasets. Furthermore, Align-IQA outperforms CONTRIQUE, Re-IQA, LIQE, transformer-based MANIQA, and MUSIQ, as well as AIGC-related PSCR [42] and TIER [43] on the various datasets, which verifies the effectiveness of our Align-IQA, injecting quality-aware prior knowledge into the general-purposed pre-trained model and extracting multi-scale quality-aware features. Notably, our Align-IQA uses significantly fewer training parameters (35.43 M) than the latest quality-aware pre-trained models and fully fine-tuned Method AGIQA-1K SRCC PLCC ResNet50 [46] 0.637 0.732 StairIQA [46] 0.550 0.609 MGQA [46] 0.601 0.676 WaDIQaM-NR [2] 0.728 0.779 CONTRIQUE [22] 0.793 0.858 PSCR [42] 0.843 0.840 TIER [43] 0.827 0.830 Align-IQA 0.855 0.890 models, while achieving higher PLCC value on the CSIQ dataset, as shown in Fig. 4 . Additionally, Align-IQA outperforms the previous best results on AIGC datasets, achieving PLCC of 0.890 (+3.73%) and 0.924 (+1.99%) on AGIQA-1K and AGIQA-3K.To demonstrate the generalization ability of our Align-IQA, we conduct cross-dataset tests. Specifically, HOSA [37] and WaDIQaM-NR [2] are selected for comparison, with the SRCC results presented in Tab. 4. Among the synthetic cross-dataset validations, Align-IQA achieves the highest performance. With the help of quality-aware prior knowledge and multi-scale mechanism in the HVS, Align-IQA can accurately evaluate the quality of images with unseen distortion types. Cross-dataset validation shows that our Align-IQA effectively alleviates the problem of limited IQA dataset size.Table 3 : Performance comparisons on the AGIQA-3K dataset.The best and the second-best performance results are marked in boldface.Method AGIQA-3K SRCC PLCC DBCNN [16] 0.821 0.876 CLIPIQA [16] 0.843 0.805 CNNIQA [16] 0.748 0.847 WaDIQaM-NR [2] 0.219 0.393 CONTRIQUE [22] 0.807 0.887 PSCR [42] 0.850 0.906 TIER [43] 0.825 0.882 Align-IQA 0.874 0.924 The results demonstrate that our CGI module effectively incorporates prior knowledge into the general-purpose pre-trained model, thereby guiding it to focus more on extracting quality-aware features rather than semantic-aware features. Moreover, our MSFA model successfully enriches quality-aware features from the human perception perspective, highlighting the significance of integrating multi-scale information for accurate quality assessment. By simulating the multi-scale mechanism in the HVS, our Align-IQA achieves a further improvement in accuracy.The effectiveness of different backbone networks. We conduct comparative experiments with three types of backbone networks and the results are provided in Tab. 6. The backbone networks used for experiments include ViT-Tiny/16, ViT-Small/16, and ViT-Base/16 [5] . For all backbone networks, the input image size is set to 224 × 224, and the shape of the image patch is defined as 16 × 16.It can be found that using ViT-Base/16 works best. This means that using a deeper and wider backbone provides more space to capture richer quality-aware representations, thereby improving performance.The effectiveness of different injection strategies. To evaluate the effectiveness of our CGI module, two additional injection strategies [10] (Fig. 5 The other strategy entails concatenating the input guide and image embedding. It should be noted that the input guide is replaced with the embedding of visual saliency. The performance comparison between these two injection strategies and the CGI module is depicted in Tab. 7. It is observed that our CGI module achieves the best performance.The effectiveness of different strategies for extracting multiscale features. To test the effectiveness of our MSFA in building a feature pyramid using the multi-level output of ViT, an additional strategy [17] (refer to Fig. 7 (a)) is selected for comparison. In this strategy, only the output from the last layer of ViT is utilized to build the feature pyramid. The SRCC and PLCC results are summarized in Table 8 . It is observed that enhanced performance is achieved 𝑗 𝑗𝑝𝑒𝑔 extracted at the same stage are similar. The same observation is noted for VPT-add and VPT-concat. For example, the corresponding lines of (a) and (e) are similar respectively, as do the second, third, and fourth lines of (b) and (f). However, the features F 𝑗 𝑓 𝑓 and F 𝑗 𝑗𝑝𝑒𝑔 extracted by our Align-IQA at the same stage exhibit significant variance for different types of distorted images. This is consistent with each distortion type having its own specific way of affecting image quality, indicating that our Align-IQA can extract discriminative features to accurately describe different types of distortion.(c) (d) (e) (f) (g) (h) 3 f f F 6 f f F 9 f f F 1 2 f f F 1 2 ˆf f F 9 ˆf f F 6 ˆf f F 3 ˆf f F 3 jpeg F 6 jpeg F 6 jpeg F 9 jpeg F 1 2 ˆjpeg F 9 ˆjpeg F 6 ˆjpeg F 3 ˆjpeg F jpeg I f f I 3 f f F 6 f f F 9 f f F 1 2 f f F 1 2 f f F 9 f f F 6 f f F 3 f f 
conclusion: 
 In this work, we propose Align-IQA, a novel NR-IQA framework that aims to generate visual quality scores aligned with human preferences for different types of visual content. First of all, we propose a customizable guidance injection module. By injecting specializable quality-aware prior knowledge into general-purpose pre-trained models, the proposed module guides the acquisition of quality-aware features consistent with diverse human preferences for various types of visual content. Second, we propose a multiscale feature aggregation module. By simulating the multi-scale mechanism in the HVS, the proposed module enables the extraction of a more comprehensive representation of quality-aware features from the human perception perspective. Furthermore, by leveraging depth-wise dilated separable convolution, the proposed module achieves high computation efficiency. Finally, we conduct extensive experiments on eight IQA datasets. The results show that Align-IQA achieves comparable or better performance than SOTA NR-IQA methods. Notably, Align-IQA outperforms the previous best results on AIGC datasets, achieving PLCC of 0.890 (+3.73%) and 0.924 (+1.99%) on AGIQA-1K and AGIQA-3K. Meanwhile, Align-IQA achieves SOTA performance while reducing training parameters by 72.26%. We anticipate that this work motivates more insights on integrating professional prior knowledge into general-purpose pre-trained models, improving the performance and interpretability of downstream IQA tasks.  
 
 Appendix:  


Decision: This paper proposes a quality assessment method able to fit the differences in preferences among uses assessing images through customized guidance. The article describes the proposed method and carries out related experiments. The three reviewers agree on the potential and relevance of the approach and paper. The reviewers see the potential of the approach and its relevance. However, they rise important challenges. These have to do with the added value of the approach, its novelty and clarifications about the writing. The authors clarified most of the issues, but some of them are still open. This is clearly a borderline paper. I really see the potential of the paper, and the novelty of the approach, while not clearly brought forward, is there. Thus, I am inclined to accept it, given that the authors address all the issues risen by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper designs a novel Align-IQA framework to perform IQA tasks on different types of visual content.  Contains two key designs: (1) Customizable quality-aware boot injection module.  By injecting specialized quality-aware prior knowledge into a general pre-trained model, the proposed module guides the acquisition of quality-aware features and allows different adjustments of the features to comply with different human preferences for various types of visual content. (2) Multi-scale feature aggregation module.  By simulating the multi-scale mechanisms in the human visual system, the proposed module is able to comprehensively characterize the human visual system.
strength_weakness: strength_weakness: 1. The guidance injection module proposed in this article can evaluate different types of content, is highly consistent with the human eye's supervisory experience, and has the potential for application in future IQA tasks.

2. The experimental data set is relatively sufficient, and the comparison methods are complete and advanced.

3. The method proposed in this article is used for both traditional visual content and the emerging AIGC Quality Assessment. It can promote further applications of AIGC and is in line with the purpose of this MM conference.
limitations: limitations: 1. The way formulas (2)-(6) are written is a bit confusing. In this part, the author first declares the variables, and then puts all five formulas together, but does not describe the relationship between the variables. Especially in the projection functions section, the block diagram shows that this is an important factor for the effectiveness of Align-IQA, but the author did not explain it. It is recommended to write the formulas one by one.

2. There is room for improvement in the article layout. The authors' typography was not tight enough, resulting in important ablations being placed in the supplementary material. Tables 2-8 can be merged separately to save a certain space.

3. Align-IQA proposed by the author has significant advantages in evaluating AIGC, which is a good point. However, using AGIQA-1K/3K verification is slightly outdated. The development of T2I generative models is extremely rapid, and even the best images in 3K (Jun 2023) are only of average quality compared to the current AIGC. Therefore, the author can verify the performance of Align-IQA on the AIGIQA-20K (Apr 2024) database, thereby better proving its applicability to AIGC.
paper_summary: paper_summary: The authors propose a quality assessment method called Align-IQA, which is able to have a good fit with different preferences through customized guidance. The article describes in detail the principles of the proposed method and carries out related experiments, ultimately re-emphasizing the superiority of the algorithm.
strength_weakness: strength_weakness: This paper is able to utilize guidance information to deal with quality assessment of different preferences is a relatively novel idea. In addition, the work has also designed a series of modules that may have some validity. Another advantage of the model is the low inference overhead.
limitations: limitations: Details from the abstract:

1) The direct use of abbreviations is not recommended for both SOTA and PLCC at L31 and L33, although these terms are not unfamiliar to readers with some experience.

Defects:

2) The Customizable Guidance Injector module is not explained clearly enough. How does it primarily relate to Diverse Human Preferences? From Fig 2 I can only see the source of Eimg, the source of Eprior is not explained clearly enough.
3) I think the experimental setup at L550 is unreasonable and the most fatal flaw of this paper. Within my knowledge, all quality assessment algorithms should be given a fixed quality score for the same input. However, the authors have introduced randomness in their reasoning (despite the averaging operation adopted here), which obviously leads to inconsistent assessment results. I suggest that the authors still use a fixed centercrop strategy instead of randomcrop.
4) Based on the experimental results in Table 1, it is felt that the performance improvement of the proposed method is more limited. Combined with the randomness in 3), I have reason to think that these results are better results after randomizing multiple experiments.
5) The results of the ablation experiments in Table 5 show that neither CGI nor MSFA is significant. This contradiction is even more prominent when compared to the results of the backbone experiment in Table 6. In addition, this performance may also be affected by the randomness in 3), so this performance is clearly unconvincing.


Overall, the Achilles' heel of this article is the introduction of randomness mentioned in 3), which compromises the results of many of the later experiments. The suggestions in 1) to 5）are hopefully something that the authors will seriously think about and improve.
paper_summary: paper_summary: This paper proposes a novel framework that aims to generate visual quality scores aligned with diverse human preferences for different types of visual content, which contains two key designs: (1) A customizable quality-aware guidance injection module. (2) A multi-scale feature aggregation module.
strength_weakness: strength_weakness: Innovative method and clear writing.
limitations: limitations: 1. Please explain in detail how "the characteristics of HVS" and "visual semantic relationship information" are obtained, respectively. Especially in CGI.
2. The synthetic distortion database should be divided into training and testing sets based on reference images.
3. The prediction performance of the proposed method shown in Table 1 is not very outstanding.
4. The comparison methods in Table 4 are a bit lacking, and it is necessary to add results on the synthetic and AIGC databases.
5. Add the results of testing on an authentically distorted database in Section 4.4.1.
6. If the tunable feature embedding of AIGC images is the same as that of authentically distorted images, how does the proposed method perform in predicting the quality of AIGC images?