QPT-V2: Masked Image Modeling Advances Visual Scoring

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 06:22:18

Abstract: Quality assessment and aesthetics assessment aim to evaluate the perceived quality and aesthetics of visual content. Current learning-based methods suffer greatly from the scarcity of labeled data and usually perform sub-optimally in terms of generalization. Although masked image modeling (MIM) has achieved noteworthy advancements across various high-level tasks (i.e., classification, detection etc.). In this work, we take on a novel perspective to investigate its capabilities in terms of quality- and aesthetics-awareness. To this end, we propose Quality- and aesthectics-aware PreTraining (QPT V2), the first pretraining framework based on MIM that offers a unified solution for quality and asthectics assessment. Specifically, QPT V2 incporporates following key designs: To perceive the high-level semantics and fine-grained details, pretraining data is curated. To comprehensively encompass quality- and aesthetics-related factors, degradation is introduced. To capture multi-scale quality and aesthetics information, model structure is modified. Extensive experimental results on 11 downstream benchmarks clearly show the superior performance of QPT V2 in comparison with current state-of-the-art approaches and other pretraining paradigms.

Main paper: 
 introduction: 
 The aims of Image Quality Assessment (IQA), Visual Quality Assessment (VQA), and Image Aesthetics Assessment (IAA) are to appraise the quality and aesthetics of visual content, serving as critical components across a multitude of vision applications including video enhancement, transcoding, and transmission [34, 65, 105] . While being studied separately for a considerable period, these tasks present strong resemblance in various aspects. All these tasks share the same core objective, that is, to mimic the Human Visual System (HVS), so as to generate accurate scores aligned with human perception [27, 82, 83] . Moreover, the proliferation of User-Generated Content (UGC) [73, 77, 110] and AI-Generated Content (AIGC) [5, 93, 102, 103] has become a trend in recent years, which greatly contributed to the exponential growth of image and video data [3] . The complexity and interrelation of quality-related and aestheticsrelated factors in emerging content are unprecedented, and analyzing single factors alone is insufficient to achieve a comprehensive perception of visual content aligned with human perception. In response to the aforementioned resemblance and trend, we refer to IQA, VQA, and IAA jointly as Visual Scoring (VS) for analysis. Facilitated by the advancements of deep neural networks [13, 26, 41, 70, 81] , learning-based methods [28, 43, 44, 72, 112] have surpassed traditional methods [10, 57, 78, 92] based on handcrafted features on multiple VS benchmarks [6, 9, 17, 20, 59, 86, 101] . They acquire features with strong expressiveness via regressing from the Mean Opinion Scores (MOS). However, one of the primary obstacles in solving VS lies in the limited size of labeled datasets [44, 46, 55, 74, 111] . Due to the high cost associated with collecting MOS through extensively annotated subjective studies, the scale of VS datasets is often only a fraction, ranging from one-tenth to even one-hundredth, of other high-level visual task datasets (e.g., object recognition). At all events, the paucity of labeled data restricts the capabilities of data-driven deep learning methods.To tackle this problem, some previous efforts increased data size by patch/frame-level augmentation [4, 35, 36, 46] or mixed-database training [39, 40, 94, 107] . However, the quality and aesthetics scores of local patches often differ from those of the entire content, and subjective differences are observed across datasets, thus hindering the achievement of promising results. On the other hand, a different research line [37, 76, 83, 89, 95] exploits knowledge valuable for VS from datasets and model weights of other domains, by tapping into the power of pretrained vision or vision-language (VL) models [63, 76, 89, 95, 98, 100, 106] . These works attempt to extract knowledge that is more quality-or aesthetic-aware from large-scale datasets by carefully designing pretraining objectives [44, 55] , and are then finetuned on downstream VS tasks. The pretraining objectives of existing works are mainly based on contrastive learning [67, 80] , which can be viewed as a global self-supervised learning (SSL) approach, as it groups similar samples closer and diverse samples far from each other [25, 55, 111] . However, this "sample-level" discernment is insufficient for capturing local distortions and visual attributes [60] . Therefore, exploring more effective pixel-level discrimination may be beneficial for incorporating pretrained priors into downstream VS tasks.Masked Image Modeling (MIM) [24] , which learns representation by pixel-level reconstruction of the masked regions in the input, has demonstrated its impressive ability of semantic-and texture-aware perception in visual tasks [2, 61, 84, 109] . In this paper, we conduct a detailed exploration of MIM, in which we observe MIM can learn both sample-level and pixel-level information of the visual content, showing the potential to serve as a general pretraining recipe to VS tasks. As shown in Fig. 1 , we propose QPT V2, the first pretraining framework based on MIM that offers a unified solution for VS tasks. To enhance the acquisition of prior knowledge by MIM for VS tasks, we propose further improvements and optimizations of the vanilla MIM from the perspectives of data, degradation, and model. Regarding the realm of data, we curate a dataset with high resolution (HR) and high foreground coverage (HFC), thereby aiding the pretext task of MIM. Regarding the realm of degradation, we propose an optimal strategy for applying degradations to the reconstruction target, exploring the type and composition of degradation to acquire prior knowledge of practical scenarios. Regarding the realm of model, we use a drop-in strategy to learn multi-scale representations by adaptively fusing features of different layers. Our main contributions can be summed as follows:• To the best of our knowledge, we are the first to validate the capability of MIM in adeptly unifying downstream visual scoring tasks. We decompose MIM into three crucial components: data, degradation, and model, and individually investigate their respective influences. • We propose QPT V2, which stands as the pioneering MIMbased pretraining framework, offering a unified solution for VS tasks. To enhance the acquisition of prior knowledge through MIM, we make targeted improvements in the aspects of data, degradation, and model. • QPT V2 achieves state-of-the-art (SOTA) results on 11 benchmarks in IQA, VQA, and IAA, surpassing other pretraining paradigms as well. Extensive ablation studies prove the validity of each enhancement of MIM. 
related work 2.1 visual scoring: 
 Visual scoring necessitates precise scoring of visual content in terms of quality (e.g. , IQA, VQA) and aesthetics (e.g. , IAA). In this work, we focus on Non-reference QA (e.g. , NR-IQA and NR-VQA), since the availability of pristine data is too hard in the real world. At the early stage, handcrafted features based on natural scene statistics (NSS) dominate the realm of VS [56, 57, 69] . Later, data-driven methods enhanced the performance significantly with the rise of deep learning [15, 35, 45, 83, 89] . Nonetheless, they rely heavily on label-intensive supervision. Previous works attempt to solve this problem by data augmentation [4, 35, 36, 99] , mixeddatabase training [40, 46, 74, 94, 107] , rank-based learning [47, 50] and general knowledge transfer [15, 83, 89, 108] . Several researches focus on extracting quality or aesthetics information by large-scale pretraining. Among them, CONTRIQUE [55] learns distortion-related information on images with synthetic and realistic distortions based on contrastive learning. Similarly, Re-IQA [66] re-engineers the MoCo-v2 [8] framework and applies intricate data augmentations to learn quality-aware features. Moreover, QPT [111] introduces a diverse array of degradations and composites to mimic real-world distortions, which greatly expands the pretraining data volume. Different from them, we devise a pretraining framework based on MIM to learn effective quality-and aesthetics-related representations.Masked modeling learns representation by reconstructing a masked portion of the input. Driven by the success of BERT [12] in NLP, MIM has become an representative SSL method in computer vision [2, 24, 61, 84, 109] . As a pioneer work, BEiT [2] proposes to reconstruct the features of DALL-E [64] . MAE [24] directly reconstructs raw pixels of the masked areas, which greatly simplifies the whole pretraining pipeline. Some studies prove that pixel-based MIM is biased towards reconstructing low-level details, thus hindering the performance on high-level tasks [2, 51, 60] . As a result, following works introduce more complicated reconstruction target rather than using raw pixels [16, 33, 87, 88] . While previous MIM studies mainly focus on high-level tasks, in this paper, we make the first attempt to adapt MIM to visual scoring.  
methodology: 
 We first revisit MIM concisely in Sec.3.1, and then describe the motivation of QPT V2 in Sec.3.2. Last, the key designs incorporated in QPT V2 are elucidated.There are three major steps in MIM: (1) split the image into visible and masked patches, (2) reconstruct the masked patches and (3) calculate the reconstruction loss. Given the original image I ∈ R 𝐻 ×𝑊 ×3 , where 𝐻 , 𝑊 are the height and width of the image. First, specific degradations A (•) (e.g. , resizing) are applied to the image, generating non-overlapping visible patches I 𝑣 and masked patches I 𝑚 with masking M:EQUATIONSecond, only the visible patches I 𝑣 are fed into the autoencoder G(•) to reconstruct the masked patches Î𝑚 as:EQUATIONThe autoencoder G(•) consists of an encoder F 𝑒 and a decoder F 𝑑 , both are stacked Transformer blocks. Here, a shared learnable mask token e [ M ] functions as the placeholder of masked patches, which are combined with the encoder's output and fed into the decoder. Last, an MSE loss L (•) is computed at masked positions for self-supervision as L = ∥I 𝑚 -Î𝑚 ∥ 2 2 .To accurately score the quality and aesthetics of visual content, a broad range of VS-related factors necessitate examination, namely high-level attributes (e.g. , semantics, composition etc. ) and lowlevel distortions (e.g. , blur, noise etc. ). By analysing the insightful features of MIM, we believe the pretrained models have the potential to be both quality-aware and aesthetics-aware, described next.First, it has been proved that MIM has the ability to comprehend the high-level semantics of the image [2, 24] . During pretraining, the large masking ratio forces the model to reconstruct the masked area provided with a few visible patches. Depicted by Fig. 2 (a) , the pretrained model gives semantically plausible reconstruction even when 90% of the pixels are masked. Second, MIM is proved to be biased towards low-level details when reconstructing [51, 52] the raw pixels. Due to the perfect reconstruction of pixel values, the model focuses on intricate details (e.g. , texture with repeated patterns) besides understanding the content, allowing for a better perception of low-level distortion. To better illustrate the distortionawareness of MIM, we separately apply blurring and noise to the same image. Reconstruction results in Fig. 2 (b) show that the pretrained model can perceive distortions in low and high frequency intervals, respectively.Despite MIM has the potential to encompass VS-related factors comprehensively, unleashing its power on downstream VS tasks still presents a non-trivial endeavor. We dissect the MIM framework and identify three components that contribute to this gap: • Data. ImageNet [11] has become the de-facto pretraining data in MIM studies. The images generally exhibit low resolution and lack intricate details. Over the years, supporting evidence from psychophysical studies has indicated the richness of details (e.g. , spatial complexity) in visual content directly impacts human eye's perception of quality and aesthetics [14, 23, 27, 82] . Thus, pretraining on data lacking details might not be sufficient for model to exploit fine-grained quality and aesthetics information. In all, curating pretraining data tailored for VS is of utmost importance. • Degradation. MIM achieves excellent results in high-level tasks with simple degradations (e.g. , random cropping) [24, 91] . Thus, previous works pay less attention to the degradation design. Yet, simple degradations can only encompass VS-related factors presented in common scenario (e.g. , content editing), overlooking other factors introduced by various visual applications, such as compression, transmission, and unprofessional shooting. Therefore, degradations that cover extensive VS-related factors need to be considered. • Model. HVS assesses quality and aesthetics in a multi-scale fashion [36] . Additionally, numerous previous works have proved the benefits of utilizing multi-scale features in other vision tasks [29, 49] . As a result, to mimic HVS and capture both fine-grained and coarse-grained VS-related factor effectively, the improvement of the model structure is exceedingly crucial.HFC crop LFC crop SA-1B DIV2KDemonstrated in Fig. 3 , pretraining data of QPT V2 is curated from two criteria: high resolution (HR) and high foreground coverage (HFC). As argued above, by reconstructing the rich textures and local structures within the HR images, models are prone to perceive a broad range of quality and aesthetics information during pretraining. In addition, FC is defined as the proportion of foreground region of the entire image. Since foreground region encodes way more semantics and texture than the background, pretraining on HFC images ensures the model's sensitivity to both high-level and low-level visual attributes.Based on the two criteria, multiple datasets with various resolution and FC are investigated. We resort to SA-1B [38] as the pretraining data source for the following reasons. First, SA-1B has an average resolution of approximately 1600×2100, which is significantly higher than that of ImageNet. Second, although widely used datasets (e.g. , DIV2K [1] , UnsplashFull [54] etc. ) in superresolution (SR) task possess higher resolution (>2K), SA-1B exhibits a significantly higher FC. To maintain the resolution of HR images while adapting to the small input size of the model (e.g. , 224×224), degradations A (•) in Equ. 1 typically include random cropping, which further widens the gap between SA-1B and SR datasets in terms of FC. Fig. 4 highlights this difference. Third, SA-1B provides a straightforward criterion, namely the number of objects per image, which allows us to further filter the dataset to get images with higher FC. Eventually, the pretraining dataset for QPT V2 consists of 1.28 million HR images filtered from SA-1B, with each image containing 50 or more objects. The effectiveness of HR & HFC data on downstream VS tasks is validated, we refer the reader to Sec.4.3 for more details.To comprehensively cover the VS-related factors, degradation type and composition are studied. Fig. 5 visualizes all the degradations studied in this work. First, to account for VS-related factors introduced by geometric transformation, resizing is considered. Second, to cover the factors introduced by frequency shift, blurring, sharpening, and gaussian noise are studied. Last, to incorporate the factors introduced by color changing, color jittering and color space transformation (CST) are considered. Following [55] , we employ four color spaces including RGB, LAB, HSV and grayscale. Fig. 6 showcases the completeness of our degradation selection. In terms of degradation composition, two strategies are adpoted. First, we compose degradations sequentially. Second, inspired by recent progress in SR [85, 111] , an advanced composition including shuffling, skipping and high-order operations is used to obtain complex degradations. Random cropping is applied after all the degradations by default. Benefiting from the comprehensiveness of our degradation selection, we discover that CST stands out as the most quality-and aesthetics-aware degradation. Previous NSS-based VS studies have demonstrated that the VS-related information exists in various color spaces [19, 79] and subsequent studies further proposed that the information of different color spaces are complementary to each other [55] . Therefore, we speculate that applying CST to the reconstruction target exposes a richer set of quality and aesthetics factors to the model, improving the data diversity during pretraining. Different from previous pretraining objectives based on contrastive learning, we further reveal the fact that QPT V2 does not benefit from the sequential or advanced composition of degradations. More details of both findings can be found in Sec.4.3.To perceive the quality and aesthetics information at different scales, encoder architecture and multi-scale feature fusion are considered. Regarding the selection of the encoder architecture, the common choices are ViT [13] and hierarchical backbones (e.g. , Swin [53] and HiViT [109] ). Compared to ViT, hierarchical backbones are better at learning multi-scale features by leveraging image-related inductive biases. Thus, a representative hierarchical backbone HiViT is selected as the encoder.There are three stages of different scales in HiViT. Upon that, we devise a fusion module to incorporate the multi-scale features output by different stages. The fusion process is described next. The hierarchical encoder F 𝑒 outputs features at multiple stages during pretraining, shown in Fig. 3 . These features are denoted by 𝑋 = {𝑥 𝑖 } 1≤𝑖 ≤𝑁 , where 𝑁 represents the number of stage. First, 𝑥 𝑖 is processed by a projection layer P 𝑖 (•), which aligns the feature space between outputs of different stages, as:EQUATIONSecond, the projected features of all stages 𝑋 , are integrated by a fusion layer 𝐹 (•) as:EQUATION𝑌 will be fed into the decoder F 𝑑 for pixel reconstruction. Note that the fusion process is only introduced during pretraining, without affecting the finetuning stage. More details of the architecture selection and feature fusion are in Sec.4.3. 
experiments: 
 In this section, experimental setups are first introduced in Sec.4.1. By comparing to existing SOTA methods in Sec.4.2, QPT V2 is evaluated on 11 benchmarks from all three VS tasks. Last, an indepth ablation over QPT V2 is provided in Sec.4.3.Criteria. SRCC (Spearman rank correlation coef.) and PLCC (Pearson linear correlation coef.) are adopted as evaluation criteria for all three tasks, both ranging in [0, 1]. A larger SRCC indicates a better ranking between samples, and a larger PLCC shows a more accurate score prediction.Benchmarks. 11 benchmarks are selected from IQA, VQA, and IAA to comprehensively evaluate the visual scoring ability of QPT V2. For IQA, three synthetically degraded datasets (TID2013 [62] , LIVE [68] , KADID [48] ) and three datasets with real-world distortions (KonIQ10K [32] , CLIVE [18] , FLIVE [98] ) are included. For VQA, we choose three public NR-VQA datasets, including LIVE-VQC [71] , KoNViD-1k [31] , and LSVQ [97] . For IAA, AVA [58] is selected for evaluation. The key designs of QPT V2 are ablated on FLIVE, LIVE-VQC, and AVA. For all the datasets without official splitting, we randomly split them into 80% for training and 20% for testing. The finetuning/evaluation procedure is conducted on 10 different splittings to avoid randomness, and the average SRCC and PLCC is reported.Pretraining details. All the experiments are conducted on 4 NVIDIA V100 GPUs. The pretraining data, degradation and model are specified in Sec.3.3, Sec.3.4, and Sec.3.5, respectively. We randomly mask 75% of the pixels following [24] and the input image size is 224×224. The hyperparameter settings are inherited from [24] .Finetuning strategy. For IQA, we implement the regression head with a simple MLP (e.g. , two linear layers with a GeLU activation in between). Following [74] , we resize the shorter edge of images to 340 while keeping the aspect ratio, then randomly crop sub-images with size 224×224. AdamW is adopted for optimization, with weight decay of 0.01. The initial learning rate is 2e-5 and decayed by cosine annealing without warmup. Pretrained models are finetuned for 200 epochs, and the checkpoint of the last epoch is selected for evaluation. When testing, we take the four corners and the center crops and average their predicted quality scores to obtain the final score.For VQA, we follow the settings in [89] for finetuning. Also, the pretraining weight is inflated to adapt video input, as done in [75] . As for hyperparamters, AdamW is used with weight decay of 0.05 and mini-batch size of 16. The initial learning rate is set to 1e-3 and decay it with cosine annealing strategy. The pretrained models are finetuned for 30 epochs on LSVQ train followed by evaluating on LSVQ test , LSVQ 1080p and two other smaller datasets, LIVE-VQC and KoNViD-1k. We uniformly sample four 32-frame clips from an input video, and average the predicted quality scores as the final results.For IAA, pretrained models are finetuned on AVA 𝑡𝑟𝑎𝑖𝑛 for 60 epochs and then evaluate on AVA 𝑡𝑒𝑠𝑡 , images are resized to 224×224 IQA. We compare our QPT V2 with two groups of IQA methods, including 5 traditional methods and 7 deep learning-based methods. Results in Tab.1 show that QPT V2 achieves superior or comparable performances to current SOTA methods. Previous deep learning-based methods have achieved outstanding performances on three synthetically datasets. Therefore, further improvements on these datasets can be challenging to attain. Still, QPT V2 improves the results on LIVE and TID2013 (e.g. , +1.1% of SRCC on TID2013). Moreover, our method also reaches leading SRCC on FLIVE and CLIVE (+0.4% of SRCC on FLIVE), showcasing its ability to perceive real-world distortions effectively. Besides, Tab.1 includes methods that also harness the power of pretraining by desiging contrastive pretext tasks (e.g. , CONRTIQUE, Re-IQA, and QPT). For example, Re-IQA respectively learns a content-aware encoder on ImageNet-1K and a distortion-aware encoder on 758K distorted images. In comparison, QPT V2 consumes less pretraining data and achieves better performance.VQA. We compare QPT V2 to three traditional methods and six deep learning-based methods. Results given in Tab.2 provide the following conclusions. First, the performances we obtain exceed all the traditional methods that rely on hand-crafted features by a large margin, and beat most data-driven methods on four VQA datasets. Second, under the intra-dataset setting, QPT V2 pushes the SRCC by 0.3% and PLCC by 0.7% on LSVQ 𝑡𝑒𝑠𝑡 , exhibiting accurate quality assessment. Third, under the cross-dataset setting, we surpass the current SOTAs as well (e.g. , with 0.4% and 0.9% gains in SRCC and PLCC on LIVE-VQC), presenting impressive generalization capability.IAA. We select 11 deep learning-based methods for comparison. Tab.3 indicates that our method significantly surpasses previous SOTA results on AVA dataset, reaching 0.865 (+4.3%) of SRCC and Table 3 : Performance of existing SOTA methods and the proposed QPT V2 on AVA dataset. The best and second-best results are bolded and underlined.AVA_test SRCC PLCC NIMA [15] 0.612 0.636 MLSP [30] 0.756 0.757 AFDC [7] 0.649 0.671 MUSIQ [36] 0.726 0.738 MaxViT [76] 0.708 0.745 CLIP-IQA+ [83] 0.619 0.586 Aesthetic Predictor [42] 0.721 0.723 TANet [28] 0.758 0.765 GAT ×3 -GATP [21] 0.762 0.764 LIQE [108] 0.776 0.763 VILA [37] 0.774 0.774 Q-Align [90] 0 QPT V2 vs. other pretext tasks. We compare QPT V2 with four pretext tasks, including QPT, MoCo, ImageNet-1K supervised and train-from-scratch, on three representative VS benchmarks. Note that both supervised training and train-from-scratch use the same encoder backbone as QPT V2, which is HiViT. MoCo and QPT are based on semantic-aware and quality-aware contrastive learning, respectively. The results in Tab.4 verify the superiority of QPT V2. In addition, QPT V2 also achieves better performances than the supervised training and the one without pretrained weights in all three VS tasks. Effectiveness of HR & HFC data. We demonstrate the effectiveness of HR & HFC data in QPT V2 by comparing to models pretrained on data with differenet resolution and FC. Following conclusions can be drawn from Tab.5. First, pretraining on data with both HR and HFC leads to best downstream performances. Second, high resolution matters. When the foreground coverage is generally high, pretraining on HR images yields noticeably superior performance on all three representative VS datasets. e.g. , +2.9% on FLIVE, +2.2% on LIVE-VQC and +10.1% on AVA. Last, HFC is essential. When the resolution is relatively high, HFC data always prevail. e.g. , +4.8% on FLIVE, +2.7% on LIVE-VQC , and +8.1% on AVA.Effectiveness of quality-and aesthetics-aware degradation. Tab.6 displays the downstream performances after applying six different degradations to the reconstructed target in QPT V2. Results obtained without employing any form of degradation serve as the baseline. First, the CST degradation incorporated in QPT V2 performs the best, demonstrating its quality-and aesthecticsawareness. Second, blurring brings a slight improvement in IAA (+0.5% of SRCC and +1.1% of PLCC). Recent MIM studies find that removing the high-frequency components of pixels helps the model to focus on semantics, benefiting downstream high-level tasks [51] . Thus, we attribute the gains to the fact that IAA places greater emphasis on high-level visual attributes compared to IQA and VQA [90] . Last, the geometry-based resizing, frequency-based sharpening and noise, and color jittering impair the downstream performances on all three benchmarks. This suggests that, altering the spatial layout of the reconstruction target, enriching its highfrequency details, corrupting its frequency spectrum, or perturbing its color are all detrimental to model's quality-and aesthecticsawareness.To study the effect of degradation composition, two top-performing degradations in Tab.6, namely, CST and blurring are arranged sequentially. Also, all six degradations are included in an advanced composition protocol discussed in Sec.3.4. The results in Tab.7 indicate two findings. First, CST and blurring cannot synergize when arranged sequentially, which leads to slight inferior results. Second, QPT V2 does not benefit from a complicated degradation space. Above findings are inconsistent with those in contrastive learning, we attribute them to the distinction between two pretraining paradigms.Effectiveness of multi-scale model. Tab.8 validates the effectiveness of the encoder architecture selection. With similar model capacity, hierarchical backbones outperform plain ViT in all three VS tasks. For example, Swin-T reaches 0.868 of SRCC on AVA dataset, 4.2% higher than ViT-S. Since two hierarchical models exhibit similar downstream performances, we opt for HiViT-T as the encoder in QPT V2 for better training efficiency. Though increasing the model capacity might potentially yield better results, we did not use larger models out of tradeoffs, which can be done in future work.To validate the effectiveness of the multi-scale feature fusion strategy proposed in Sec.3.5, we fuse features at different stages, and the downstream results are given in Tab.9. By default, the output of the last stage (stage 3) is always fed to the decoder. Tab.9 indicates that multi-scale feature fusion always provides benefits for VS tasks. Particularly, fusing features from shallow stage (stage 1) yields the most significant gains on three downstream datasets. Due to the inclusion of more low-level details in shallow layer features, we believe that fusing these features assist the model in better perceiving low-level VS-related factors. Additionally, the implementation choices of the projection layer P 𝑖 (•) and the fusion layer 𝐹 (•) specified in Sec.3.5 are discussed in Tab.10. Following conclusions can be drawn: First, a simple linear layer is sufficient to project representation into the same feature space. A more complex MLP (e.g. , Linear-GeLU-Linear structure) cannot bring improvement while introducing non-negligible computational overhead. We think the non-linearity may increase the optimization difficulty as for pretraining. Second, weighted average pooling is better suited for integrating projected features compared to the simple summation.Impact of pretraining data amount . We study the impact of data amount on QPT V2 by using 20%, 50% and 100% percentages of the pretraining data. Given by Tab.11, the performances on three downstream datasets continue to improve as the data amount increases. Surprisingly, we find that even when using only 50% of the data, QPT V2 still achieves comparable performances to SOTA  
conclusion: 
 We propose QPT V2, a novel MIM-based pretraining paradigm crafted for visual scoring tasks, aiming at alleviating the obstacle of insufficient annotated data. To enhance the quality-and aestheticsawareness of the pretraining objective, we provide a meticulous analysis over the vanilla MIM framework and make targeted improvements on three key components: pretraining data, degradation, and model structure. After pretraining, QPT V2 achieves SOTA results on 11 downstream benchmarks in IQA, VQA, and IAA, demonstrating impressive capability and generalization ability. In all, we hope this work will inspire the community to reflect and explore the possibility of different pretraining paradigms in the context of visual scoring. 
 
 Appendix:  


Decision: This submission gets final ratings of 1 accept, 2 weak accept and 1 borderline accept. 
It could be accepted as a poster.

Reviews: review: paper_summary: paper_summary: This submission proposes a novel pretraining framework based on Masked Image Modeling (MIM) designed to improve performance on Visual Scoring (VS) tasks. The framework unifies the tasks of Image Quality Assessment (IQA), Visual Quality Assessment (VQA), and Image Aesthetics Assessment (IAA) under a single methodology. The proposed model incorporates high-resolution and high-foreground coverage data, quality and aesthetics-aware degradations, and a multi-scale model structure.
strength_weakness: strength_weakness: 1) The extensive experiments across 11 benchmarks provide strong evidence of the effectiveness of the proposed method.
2) The paper is well-organized, and the method are clearly explained, which makes the proposed framework easy to follow.
3) The use of MIM in the context of VS tasks is a novel idea. The decomposition of MIM into data, degradation, and model components is well-thought-out.
limitations: limitations: 1) Although there are some ablation studies, more detailed experiments isolating the impact of each component (data curation, degradation types, multi-scale modeling) would strengthen the claims about their contributions.
2) As a conference paper, the number of references is too much and it is advisable to organize them properly.
paper_summary: paper_summary: This work aims at learning quality- and aesthetics-aware representation via Mask Image Modeling and further finetune the encoder on various downstream tasks. The designed scheme showcases its reasonable effectiveness on performance improvements and sufficient ablation studies consolidate conclusions of each modules.
strength_weakness: strength_weakness: 1. This paper has its clear research objection that is to learn quality and aesthetics-aware representation, which is pivotal in IAA and IQA. 

2. Instead of designing modules with huge parameters or large-scale inputs, they think and utilize representation techniques and adapted it into IQA research field, leading a good technical novelty and SOTA performance.

3. This work has clear writing structure and good experiment design. It indicates that the authors have their own thinking and opinions clearly.
limitations: limitations: 1. In MIM, it would be great if the author could have discussions on downstream performance change caused by different mask ratios in the pretraining stage. According to line 553, the author claim that they randomly mask 75% by following prior work which is fine, but would be great to have this in hyper-param study.

2. In section 3.3, the authors claim that QPT-v2 is curated from two criteria, i.e. HR and HFC and also provide reasons of choosing SA-1B as pretraining data source. It would be great if authors can provide any quantitative statistic results here for proving the importance of HR and HFC. According to line 433, saying SA-1B exhibits a significantly higher FC compared with SR task databases. And indeed, I also see table 5 in the ablation study section, which can prove the related conclusions.
paper_summary: paper_summary: This paper proposes a new MIM method towards image quality and aesthetics. The resulting model can be fine-tuned on various visual scoring benchmarks and achieve SOTA results.
strength_weakness: strength_weakness: 1. The paper is well-writen and the motivation is clear.
2. The proposed solutions towards data, degradation and model are reasonable.
3. The Experimental results are strong.
limitations: limitations: The analysis of why different degradation approaches result in different visual scoring results is unclear. A deeper explanation and analysis of the correspondence between degradation and model’s quality- and aesthectics-awareness could make the paper more acceptable.
paper_summary: paper_summary: According to the Call for Submission(https://2024.acmmm.org/regular-papers)
"Submitted papers may consist of up to 8 pages. Up to two additional pages may be added for references. The reference pages must only contain references. Overlength papers will be rejected without review."

The references pages are more than two pages.
strength_weakness: strength_weakness: None
limitations: limitations: None