Don’t Judge Before You CLIP: A Unified Approach for Perceptual Tasks
Authors: Amit Zalcher
navve wasserman
Roman Beliy
Oliver Heinimann
michal Irani
Venue: Accepted by TMLR ()

Abstract: Visual perceptual tasks aim to predict human judgment of images (e.g., emotions invoked by images, image quality assessment). Unlike objective tasks such as object/scene recognition, perceptual tasks rely on {subjective} human assessments, making their data-labeling difficult. The scarcity of such human-annotated data results in small datasets leading to poor generalization. Typically, specialized models were designed for each perceptual task, tailored to its unique characteristics and its own training dataset.  We propose an identical architectural framework for solving multiple different perceptual tasks leveraging CLIP as a prior. Our approach is based on recent cognitive findings which indicate that CLIP correlates well with human judgment. While CLIP was explicitly trained to align images and text, it implicitly also learned human inclinations. We attribute this to the inclusion of human-written image captions in CLIP's training data, which contain not only factual image descriptions, but inevitably also human sentiments and emotions. This makes CLIP a particularly strong prior for perceptual tasks.  Accordingly, we suggest that minimal adaptation of CLIP  suffices for solving a variety of perceptual tasks. Our simple unified framework employs a lightweight adaptation to fine-tune CLIP to each task, without requiring any task-specific architectural changes. We evaluate our approach on three tasks: (i) Image Memorability Prediction, (ii) No-reference Image Quality Assessment, and (iii) Visual Emotion Analysis. Our model achieves state-of-the-art results on all three tasks, while demonstrating improved generalization across different datasets.

Published in Transactions on Machine Learning Research (07 (025))
Don't Judge Before You CLIP: A Unified Approach for Perceptual Tasks
Amit Zalcher*
omtt.zacherr@usssmmnn..cc
Deportment of Computer Science and Applied Mathematics The Weizmann Institute of Science
Navve Wasscrman"
Deportment of Computer Science and Applied Mathematics The Weizmann Institute of Science
Roman Beliy
Department of Computer Science and Applied Mathematics The Weizmann Instátute of Science
Oliver Heinimann
Department of Computer Science and Applied Mathematics The Weizmann Instilute of Science
Michal Irani
Department of Computer Science and Applied Mathematics The Weiemann Institute of Science
Reviewed OpenReview: https: opper/eveee.....ee...m.mm net/ per/formm? id=wo[YYikkbu
Abstract
Visual perceptual tasks aim to predict human judgment of images (c.g., emotions invoked by images, image quality assessment). Unlike objective tasks such object/scent recognition, perceptual tasks rely on subjective human assessments, making their data-labeling difficult. The scarcity of such Human--nnntatee data results in small datasets leading to poor general- ization. Typically, specialized models were designed for each perceptual task, tailored to its unique characteristics and its own training dataset. We propose an identical architectural framework for solving multiple different percoptual tasks leveraging CLIP as prior. Our approach is based on recent cognitive findings which indicate that CLIP correlates well with human judgment. While CLIP was explicitly trained to align images and text, it implicitly also learned human inclinations. We attribute this to the inclusion of human-written image captions in CLIP's training data, which contain not only factual image descriptions, but inevitably also human sentiments and emotions. This makes CLIP a particularly strong prior for perceptual tasks. Accordingly, we suggest that minimal adaptation of CLIP suf- fiees for solving a variety of perceptual tasks. Our simple unified framework employs a lightweight adaptation to fine-tune CLIP to each task, without requiring any lask-specific architectural changes. We evaluate approach three tasks: (i) Image Memorability Prediction, (ii) No-reference Image Quality Assessment, and (iii) Visual Emotion Analysis. Our model achieves state-of-the-art results on all three tasks, while demonstrating improved generalization across different datasels. See our Project page for demos and models.
Equal contribution

---

Published in Transactions on Machine Learning Research (07/2025)
(a) Image Perceptual Properties
(c) Our Approach
(d) SOTA Results
Image Quality
KonlQ-10k
CB
0.932
(PLCC)
IQA
Image Memorability
Ours
0.945
Memorability
Emotional Reaction
Fine-Tuning Per Task
LAMEM
(SRCC)
CB
0.720
(b) Human-Written Image Captions
A Unified
Ours
0.741
"A special and unusual fruit arrangement."
"Pure joy, two bananas time!"
Framework
'An extremely blurry car."
Leveraging CLIP
EmoSet-8
Emotions
CB
0.819
(ACC)
Ours
0.853
Task Perceptual Score
CB Current Best
Figure 1: Our Framework: (a) Perceptual tasks rely on subjective human judgment. (b) Illustraiion of CLIP's traiming samples, which includes human-wrillen caplions. These humaan-generuted annotations contain not only fac- tual image descriptions, but inevitably also human sentiments, preferences and emotions. This suggests that CLIP can serve as prior for perceptual tasks. (c) Our approach leverages CLIP's s prior knowledge to address multiple per- veptual tasks with misiimal lask-specifie adaptation (d) We achieve state-of-lhe-ar performance across three distinct perceptual tasks. CB refers to the current best method in each task (see Tables 1 15 for numerical scores).
1 Introduction
Visual perceptual tasks have been long-standing research area Marr 2010 Leeds et al. 2013 incorpo- rating both psychological aspects Giisson
2014 Lu et al. 2011 Shemyakina et al. 2024 ) and practical
applications like education (Wang] 2021
Duivenboden
2023 and advertising (Picters et al. 2010 Bi ct al.
2024 Sample et al., 2020 Negin & Tantawi 2015). These include predicting human-perceived image qual-
ity (Khosla et al. 2015 Ma et al. 2017 Zhao et al. 2023 emotional responses to images (Peng et al. 2016
Yang et al. 2021ab, and image memorability (Isola et al. 2011b Squsalii-oussaini al., 2018). Unlike objec- live tasks such as object or scene recognition, perceptual tasks rely on subjective human assessments, making its data-labeling difficult. First, capturing genuine human reactions requires carefully designed experiments, such using precise survey questions to distinguish subtle emotional states evoked by images. Additionally, variability in perception of different people nccessitates collecting data from multiple participants to ensure
robust and reliable results (Kret & De Gelder 2012 Isola et al. 2011b Fang et. al. 2020). The scarcity of such human- annotated data results in small datasets which restricts the capabilities and generalization of predictive models.
Typically, specialized models were designed for each perceptual task, tailored to its unique characteristics and its own training dataset. For instance, emotion recognition models (Xu et al. 2022 2024b) often integrate
psychological insights, while image quality assessment models (Xu et al.] 2024a Qin el al. 2023 focus on multi-level image features. However, fully understanding the specific characteristics of human perception for each perceptual task is ongoing rescarch and remains debatable (Kramer et al. 2023
Chandler 2013
Zhang et al. 2024 Wang al. 2023a)
Since human-labeled data is limited, a strong percepiual prior is needed. In this paper we propose unified approach, named PerceptCLIP, for solving multiple different perceptual tasks using identical architectural frameworks. Our approach is based recent cognitive findings (Shoham et al. 2024] which indicate that CLIP correlates well with human judgment. While CLIP was explicitly trained to align images and text, it has also implicitly learned human preferences. We attribule this to the nature of CLIP's training data, which includes human-written image captions that consist not only factual image descriptions, but inevitably also subjective aspects such as human sentiments, preferences and emotions. This makes CLIP particularly strong prior for perceptual tasks. Accordingly, we suggest that minimal adaptation of CLIP, rather than extensive fine-tuning, is sufficient to achieve strong performance on perceptual tasks.
2

---

Published in Transactions on Machine Learning Research (07/2025)
A few previous methods employed CLIP for specific perceptual tasks. However, some fully fine-tuned most of its components (Zhang et al. 2023
Martin-Fernández et al. 2023), thus erasing its strong pre-trained knowledge and risking dataset-overiit Others relied on tuning only CLIP's text-prompts (Deng et al, 2022
2024 Wang et al. 2023b Xu et al. 2024b), which is too restrictive, preventing proper adaptation of relevant layers. Our approach strikes balance belween preserving CLIP's strong ability to caplure diverse perceptual aftributes, while allowing sufficient flexibility for task-specific adaptation.
Specifically, simple unified architectural framework employs lightweight adaptation to fine-tune CLIP to each task, without requiring any task-specific architectural changes. We apply LoRA (Low-Rand) Adap- tation) (Hu et al.) 2021) on CLIP visual encoder, selectively fine-tuning only the attention layers, followed by an MLP head. This approach preserves CLIP's perceptual priors while enabling task-specific refinement.
We evaluate our approach on three tasks: (i) Image Memorability Prediction, (ii) No-reference Image Quality Assessment, and (iii) Visual Emotion Analysis. Our model achieves state-of-the-art results on all three tasks, while demonstrating improved generalization across different datasets. Our results show that CLIP already holds rich perceptual knowledge and, with efficient tuning, surpasses previous task-specific methods without extensive manual adjustments or domain exportise. Furthermore, we demonstrate the benefits of joint training on multiple datasets of the same lask, by using different MLP head per dataset, with joint finetuned CLIP backbone for all datasets. This significantly enhances model performance small datasets.
Our contributions are as follows:
We propose simple unified fromework that leverages CLIP's pre-trained knowledge, enabling effec- live adaptation to multiple perceptual tasks and eliminating the need for lask-specific architectural adjustments.
Our models achieve state-of thetthertttrttte art performance across different perceptual tasks: image memo- rability prediction, no-reference image quality assessment, and visual emotion analysis, showcasing both superior results, as well better generalization across different datasets.
We demonstrate that joint training on multiple different datasets of the same task, leads to enhanced model performance on small datasets.
We highlight that strong, reprodueible baselines using simple framework can outperform complex, task-specific architectures, setting solid foundation for future progress.
2 Related Work
No-Reference Image Quality Assessment (IQA). No-reference IQA involves predicting the percep- tual quality of an image (according to human judgment) without reference image. Early methods predom-
inantly used CNN-based architectures (Kang et al.] 2014 Ma et al. 2017 Su et al. 2020: Network Saha
ct al.
2023) or hybrid CNN-transformer models (You & Korhonen 2021 Xu et al. 2024a These approaches
aimed to capture both local and global image features, essential lor accurate quality assessment (Qin el al.
2023
More recent transformer-based models have demonstrated further improvement (Qin et al. 2023
Ke et al. 2021. A notable advancement in IQA has been the integration of large-scale pretrained mod- els (Zhang et al. 2023 Xu al.] (2024a] Wang et al. (023b)) While CLIP has been utilized as a pretrained backbone (Zhang al. 2023 Wang al. 2023bb, existing approaches either fully fine-tune it optimize only text prompts, failing to effectively adapt CLIP to the task and fully leverage its strong prior. The current stale-of-the-art (Xu el al.] 2024a) uses both a pretrained VIT and ResNet trained on ImageNet, adapting them to the IQA task. Our method surpasses these approaches while having significantly fewer trainable paramoters.
Image Memorability Prediction. Early works [Isola et al., 2011a b) demonstrated that image memo- rability (i.c., the likelihood of an image to be remembered) is an inherent property of an image, influenced primarily by its content and structure. Those initial approaches relied on global image descriptors and color histograms, later evolving to leverage features extracted from fine-tuned CNNs (Khosla al. 2015).
3

---

Published in Transactions on Machine Learning Research (07/2025)
which significantly improved prediction performance. Subsequent methods (Squalli-ouusaaii al. 2018 Leonardi et. al.
incorporated additional semantic features from image captioning systems or soft at- tention mechanisms. Residual networks (ResNets) further enhanced memorability estimation, with studies like Fajtl et al. 2018 integrating RcsNet50 and LSTM for regression. recent work (Hagen & Espeseth, 2023) explored using Vision Transformer (ViT) (Dosovitskiy el al. 2020), which performed well but stiill fell short of (Squali-Houssaini al. (2018). Our approach achieves superior performance compared to the above methods leveraging CLIP perceptual prior.
2019:
Visual Emotion Analysis. The problem of assessing the emotional reaction and sentiment of people towards images has evolved through various approaches. Early works (Yang al.) 2018: Rao et al. 2019
used pre-trained CNN backbones for feature extraction, while graph-based methods (Yang al. 2021a
Wu et al. 2021 Yang et al. 2021b) emphasized object-graph relationships and emotion--nnaanced features, highlighting the importance of contextual interactions. Xu ct al. (2022) introduced hicrarchical emotion modeling, leveraging psychological theories for improved emotion representation. CLIP has recently been explored for emotion recognition (Deng et al., 2022 2024; Xu et all 2024b Yao et al.) 2024), mainly through prompt-based learning while some further use insights from psychological research. Our method demonstrates improved results, without relying on task-specific knowledge.
Unlike previous methods that rely task-specific architectures, use single architecture across tasks, eliminating the need for extensive modifications or domain expertise. As described above, few works use CLIP for specific perceptual tasks. However, they either trained all of its parameters, risked the loss of its priors or used restrictive approach by applying only prompt tuning. Tn contrast, our approach strikes a balance between preserving CLIP's perceptual prior and allowing task-specific adaptation, achieving state- of-thc-art results.
3 A Unified Framework for Perceptual Tasks
We propose unified framework that adapts effectively to diverse perceptual tasks while maintaining simple, consistent architecture (see fig. Our approach uses the CLIP vision encoder (Radford et al. 2021), b, followed by an MLP head for task-specific predictions. To preserve CLIP's pretrained knowledge, we apply LoRA (Hu) et al. 2021 to the attention weights, enabling lightweight task-specific adaptation with minimal additional parameters. We use the same architecture and set of hyperparameters across all tasks, reducing the need for extensive task-specific adjustments. In section 5 evaluate approach three distinct perceptual tasks: Visual Emotion Analysis, Memorability Prediction, and Image Quality Assessment.
A Unified Architecture Across Perceptual Tasks Finetuned Per Task and Dataset
LORA
Clip Vision Encoder
MLP
Prediction
Figure 2: Undfied Framework for Perceptual Tasks: Leveraging the CLIP vision encoder, following MLP hend, our architecture maintains simple, shared structure across diverse perceptual tasks. With lightweight LoRA adoptation, it fine-tunes efficiently for each task independently, effecttvely exploiting CLJP'ss prior perceptual knowl- edge.
4

---

Published in Transactions on Machine Learning Research (07 (025))
3.1 Architecture and Training
To enable elficient adaplation while preserving CLIP's perceplual knowledge, fine-tune only the query (q), key (k), and value (v) attention layers of the vision transformer using LoRA, with r=16 (decomposition rank) and a=8 (scaling factor). This significantly reduces the number of trainable parameters while allowing task-specific adaptations, resulting in less than 3M trainable parameters.
Model Training We follow consistent training strategy across all tasks. The LoRA weights and the MLP are optimized using the AdamW optimizer with weight decay of 1c-4. We explore different learning rates (5e-5, Te-4, 5e-4, 1e-3) and implement early stopping with a patience of 12 epochs and maximum of 40 epochs. The final model is selected based on the highest validation score. Before being input into the model, all images are normalized using the CLIP preprocessor. Task specific losses and augmentations are explained in section 4 and Appendix A.1
3.2 Multi Dataset Training
While our method performs well, generalizing from training to testing can still be challenging with small datasets. We propose adapting the same CLIP vision encoder using LoRA across multiple datasots within the same task, allowing the model to leverage larger datasets and improve performance on smaller ones (see section 5).
Our approach uses single CLIP vision encoder with LoRA weights, along with dataset-specifie MLP heads. The training procedure follows a two-stage process. In the first stage, the entire model is trained together, with proportional sampling ensuring that each dataset contributes to a batch in proportion to its size. The model then optimized by averaging dataset-specific losses. In the second stage, only the MLP heads are fino-tuned towards the target dataset while the CLIP encoder remains frozen. This allows our model adapt to each dataset with its unique characleristies without altering the shared represenlations provided by the CLIP encoder (which is shared by all datasets).
4 Tasks And Datasets
In this section, we present the three perceptual tasks which were used in our work (fig. 3) along with their dataset details and loss functions (with further details in Appondix A.1
4.1 No-Reference Image Quality Assessment
Image Quality Assessment (IQA) measures how humans perceive image quality, considering factors like sharpness, noise, and distortions, and is important for many visual tasks (e.g. image generation and en- hancement). is formulated as a regression problem, predicting a continuous quality score that aligns with human judgment (see Fig. 3R). We use diverse benchmark datasets with both authentic distortions (e.g., motion blur, overexposure) and synthetic distortions (c.g., mamually added noise and color quantization) of varying lypes and intensities. Each dataset. is annotated via crowd-sourced quality ratings, with mean opinion scores (MOS) reflecting human perception.
We used four authentic distortion benchmark datasets: LIVEC (Ghadiyaram Bovik 2015 (1,162 im- ages) and SPAQ (Fang al. 2020 (11,125 images), both focused on real images captured by mobile devices; KonIQ- 10k (Hosu et al. 2020) (10,073 images), which includes diverse real-world scenes; and FLIVE (Ying ee al.) 2020 comprising 39,810 images from social media and streaming platforms. We further used three
synthetic distortions benchmark datasets, which include distortions such as compression artifacts, noise, and color quantization. Each dataset consists of reference images and corresponding controlled degradations:
TID2013 (Ponomarenko et al. 2013) contains 25 reference images and 3,000 distorted images, LIVE (Sheikh) et al. 2006) includes 29 reference images and 779 distorted images, and KADID-10K (Lin et al. 2019) provides 81 reference images and 10,125 distorted images.
We follow the same training procedure as Xu al. (2024a), including the loss function, augmentations, loss function based the Pearson Linear Cor-
and data splits (see Appendix A.1. Specifically,
5

---

Published in Transactions on Machine Learning Research (07 (025))
a) Task 1: Image Quality Assessment Instruction: Assess the image quality scale of 1-100
Score: 99
Score: 66
Score: 40
b) Task 2: Image Memorability Prediction
Instruction: Click if you have seen the image before Time
c) Task 3: Visual Emotion Analysis
Instruction: Does image evoke the given emotion?
you feel <emotion> when you see this image?
Yes
No
Emotions: Excitement, Contentment, Fear, Anger, Sadness,
Figure 3: Visual Perceptual Tasks.
relation Coefficient (PICC).. Given a batch of m images, with predicted quality scores ÿ = U1, U2. Um and corresponding ground-truth labels y = 12..92.--. Mm. the PLCC-inducod loss is defined as: LPLCC = f (11 - PLCC(J, y)) where PLCC( v) represents the Pearson correlation coefficient between the predicted and ground-truth scores. During inference, we average the score predictions from 15 randomly cropped versions of each image. The data split 10 times, with cach split using 80% for training and 20% for testing. From the training set, 10% is further reserved for validation.
4.2 Image Memorability Prediction
This task aims to predict how well an image is remembered. is formulated as a regression problem, where the goal is estimate a memorability score reflecting the likelihood of the image being remembered (see Fig. 3)).
We train and evaluate our model two benchmark datasets. The LaMem dataset (Kholla. et al. 2015 contains a large-scale collection of ~60,000 images from diverse sources, including scenes, objects, and art. The Things memorability dataset (Kramer et al. 2023 comprises of 26,107 object images covering 1,854 distinct concepts. The LaMem dataset split into five parts: 75% for training, 5% for validation, and 20% for testing. For the Things datasot, we use the entire dataset for generalization assessment. The loss used for training is the mean squared error (MSE) loss the memorability score.
4.3 Visual Emotion Analysis
Visual Emotion Analysis (VEA) is a classification task focused on predicting the emotional response that an image likely evoke in viewers (see Fig. 3+).
We experiment with prominent benchmark datasets, both supporting binary emotion classification (positive vs. negative) and multi-class classification of more nuanced emotions, carefully labeled by human annotators. EmoSet (Yang et al. 2023). the largest existing visual emotion dataset, contains 118,102 images from social networks and artistic sources, labeled into 8 balanced emotions: Awe, Excitement, Amuscment, and Contentment (positive) / Anger, Sadness, Disgust, and Fear (negative). EmotionROT (Peng el al. 2016) includes 1,980 Flickr annotated images, categorized into 6 emotions: Joy, Surprise (positive) / Anger, Disgust, Fear, and Sadness (negative). Following Yang et al. (2023), EmoSet split into 80% training, 15% test, and 5% validation, while EmotionROI is randomly divided into splits with 75% training, 20% test, and 5% validation. Our model is trained using the cross-entropy loss.

---

Published in Transactions on Machine Learning Research (07/ 2205)
Table 1: Model Performance on 1QA Benchmarrks. Models are evaluated on seven common Image Quality Assessment benchmarks, reporting the median SRCC (Spearman's Ronk Correlation) and PLCC (Pearson Linear Corvelation) across 10 splits, along with the wumber of trainable parameters. The table shows that our model, Per- ceptCLIP, outperforms all leading IQA-dedicated methods, achivving the best performance, on six out of seven datasets whille using significantly fewer trainable parometers.
LIVEC
KonIQ-10k
SPAQ
FLIVE
LIVE
TID2013
KADID-IOK
Method
SRCC
PI.CC
SRCC
PL.C0
SRCC
PLOC
SROC
PLCC
SRCC
PLOC
SRCC
PLCCC
SRCC
PL.CC
HyperIQA (27h))
0.882
0.917
0.915
0.602
0.962
0.840
0.852
MUSIQ (27M)
0,702
0.746
0.916
0.928
0.918
0.921
0.506
0.661
0:940
0.911
0.773
0.815
0.875
0.872
TREs (152M)
0.816
0.877
0.915
0.928
0.511
0.625
0.969
0.968
0.863
0.883
0.859
0.858
DEIQT (24M)
0.875
0.894
0.921
0.934
0.919
0.923
0.571
0.663
0.980
0.982
0.892
0.908
0,589
0,88T
LIQE (151M)
0.901
0.910
0.919
0.908
0.970
0.961
0.930
0.931
Re-QQA (21M)
0.851
0.923
0.918
0.925
0.845
0.733
0.970
0.804
0.872
0.885
QPT-ResNet50 (24M)
0.895
0.914
0.927
0.911
0.9 0.925
0.928
0.575
0.675
LoDa (9M)
0.876
0.899
0.932
0.944
0.925
0.928
0.578
0.679
0.975
0.979
0.869
0.901
0.931
0.936
PLIP (Ours) (3M)
0.908
0.923
0.915
0.954
0.930
0.933
0.590
0.689
0.980
0.982
0.887
0.908
0.952
0.956
5 Experimental Results
In this section, we present the results of our proposed approach for training models on three different perceptual tasks. Our trained models achieved state-of-the-ar (SOTA) results with significant improvements across all three tasks. We first present the results for cach task, comparing them against the best task-spocific methods. We then demonstrate the effectiveness of training a model with multiple datasets for a given task, leading to substantial performance gains smaller datasets.
5.1 Image Quality Assessment
We train and evaluato our model on 7 prominent datascts, covering both authentic and synthotic distortions, using the standard 10 splits provided by Xu et al. (2024a). Our approach is evaluated using the median Spear- man's Rank Correlation Coefficient (SRCC) and median Pearson's Linear Correlation Coefficient (PLCC)
with a 4-parameter logistic regression correction, as in Xu et al. 2024a (see Appendix
A.1
Table D summarizes our model's performance in predicting image quality scores compared to leading IQA-
dedicated methods (Su et al.] 2020 Ke et al. 2021 Kramer et al. 2023 Golestanch et al. 2022 Qin et al. 2023 Zhang et al.) 2023 Saha et al. 2023 Zhao et al.) 2023] Xu et al. 2024a), along with the number of trainable parameters for each model. Our model achieves the best results on 5 out of 7 datasets in both SRCC and PL.CC. Our results are even more impressive when considering the number of trainable parameters. Our model consist approximately 3M trainable parameters, compared to 9M to 152M in other models, highlighting the strength of CLIP's perceptual prior.
Furthermore, as shown in Table 2 our method not only sets new state-of-the-art results but also demon- strates strong generalization capabilities across different datasets (i.c., training one dataset, but testing another). This highlights the robust of our model and its ability to learn meaningful perceptual features, rather than just overfitting dataset-specific patterns. Notably, the improvement in generalization is significantly larger than the improvement in within-dataset evaluation. For instance, we achieve a 1.4% relative SRCC improvement compared to LoDA when training and testing on KonIQ, while in a general- ization setting, when training on FLIVE and testing on KonIQ, the improvement increases to 6.4% relative SRCC. Our lightweight adaptation ensures that we retain CLIP's strong human-like perceptual priors when fine-tuning for specific tasks.
5.2 Image Memorability Prediction
We train and evaluate our model on the LaMem dataset using the standard 5-fold validation strategy. To assess generalization, we evaluate our model on the THINGS memorability datasct, as done in Neodell & Bainbridge/ (2022): Hagen & Espeseth| (2023). Performance is reported using the mean SRCC and averaged mean squared error (MSE) across 5 folds. Table 3 presents our model's performance LaMem compared to
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 2: Cross-Datased Generalization in IQA. SRCC generolization results for training on one dataset and testing on onother. PerceptCLIP signifieantly outperforms all previous models.
Train dataset
FLIVE
LIVEC
KonIQ
Test dataset
KonIQ
LIVEC
KonIQ
LIVEC
HyperIQA
0.758
0.735
0.772
0.785
TReS
0.713
0.740
0.733
0.786
DEIQT
0.733
0.781
0.744
0.794
QPT-ResNet50
0.749
0.821
LoDa
0.763
0,805
0.745
0.811
PerceptCLIP (Ours)
0.812
0.825
0.794
0.875
loading memorablliy--ddiccaeee methods Khosla et al. 2015 Fajtl et al. 2018 Leonardi et al. 2019 Hagen
& Espeseth 2023 Squalli- Houssiii et al. 2018). Our model significantly surpasses all others, achieving new state-of-the-art results on both metrics. Specifically, it improves SRCC by 2.9% relative to the second-best model (Squalli- Hooualli-Huusaaiii Houssaini et al. 2018) and reduces MSE by 9.2% relative to the previous best-performing
model (Hagen & Espesoth! 2023). ). Furthermore, our model demonstrates significantly better gencralization
capabilities. When trained LaMem and evaluated on THINGS, it achieves 12% relative improvement
over the best previously reported result. (Hagen & Espeseth
2023 shown in the Table A.1. Results
another small image memorability dataset, SUN (Isola et al. 2011b be seen in Table A.2 Altogether,
this demonstrates the power of CLIP as a prior for the image memorability task.
5.3 Visual Emotion Prediction
We train and evaluate on EamoSet and EmotionROT datasets, using the standard splits as described Section 4.3 and report emotion classification accuracy for 2, 6, and emotion categories, depending on the compares our model's performance with various existing methods (Rao et al. 2019 Xu
dataset. Table 4
et al. 2022 Yang et al. 2021b; Deng et al. 2022 2024 Xu et al. 2024b) designed to classify the emotions evoked by images. The results demonstrate that our model achieves statc-of-tho-art performance on both benchmarks, excelling in both binary emotion classification (positive vs. negative) and multi-class emotion classification (8 categories for EmoSet, and for EmotionROI).
Notably, the improvement in multi-class emotion classification more significant, as binary classification already exhibits high performance across models. Our model achieves an approximately 4% relative accu- racy improvement for both EmoSet and EmotionROI in multi-class emotion classification. These results further strengthen the effectiveness of our approach. Additional comparisons and metrics can be found in Appendix A.2 in Table A.3
Table 3: Model Performance on Image Memo- rahility. Average SRCC and MSE across 5 splits of the LAMEM dataset are reported, showing that our model, PerceptCLIP, surpasses all previous methods.
Table Model Performance on Emotion Classi- fication. Accurucy results for binary and multi-class emotion clossification on the EmotionHtO (mean over five splits) and Emooet datasets. Our model achieves state-of-the-art performance for both datasets.
LaMem
Method
EmoSet
EmotionROI
SRCC
MSFJ
Method
8
2
6
2
MemNett
0.640
Unknown
MRCNN
0.7539
0.9228
AMNet
0.677
0.0082
MDAN
0.7575
0.9371
0.6166
Leonardi et al
0.687
0.0079
StimliVEA
0.7840
0.9458
0.6162
ViTMem
0.711
0.0076
PT-DPC
0,7713
0.9216
0.6970
0.8855
Squalli-Husssain Houssaini et al
0.720
0.0092
SimEmotion
0.7906
0.9428
0.7054
0.9040
MVP
0,8192
0.9644
0.7189
0.9255
PerceptCLIP (Ours)
0.741
0.0069
PerceptCIII (Ours)
0.8528
0.9780
0.7485
0.9288
8

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 5: Mulli- Dataset Training Improves Performance. The table shows the benefits of training on multisple datasets within the same lask, demonstrating significantly improved performance on smaller datasets. Results for all datasets (shown the Appendia) show that these multi-dataset models achieve state-of Mate-of-the-art thehhar results across all benchmarks.
IQA
VEA
Memorability Prediction
Method
LIVEC
TID2013
EmoROI-6
THINGS
SRCCI
PLCC1
SRCC1
PLCC
ACC1
SRCCI
MSEL
Previous SOTA
0.904
0.910
0.892
0.908
0.7189
PerceptCIIP (Ours)
0.908
0.923
0.887
0.908
0.7485
0.452
0.0058
PerC Multi-Dataset (Ours)
0.922
0.933
0.900
0.915
0.7591
0.454
0.0054
5.4 Multi-Dataset Training
We present the results of our multi-dataset training, where single model is trained on multiple datasots of the same lask. In this selup, the LoRA finetuning of CLIP's atlention layers is shared across datasets, while the final added MLP heads are dataset-specific.
Training on multiple datasets provides additional data, significantly improving performance on smaller datasets (see Table 5) while maintaining comparable results on larger datasets (see Tables A.4 to A.6 We present the results of four models, each trained on a specifie set of datasets: (1) authentic IQA datasets (LIVEC, KonIQ-1Ok, SPAQ), (2) synthetic IQA datasets (KADID-IOK, LIVE, TID-2013), (3) memorability datasets (THINGS, LaMem), and (4) emotion datasets (EmotionROL, EmoSet). For example, in the IQA task we achieve 1.5% relative improvement plcs), when training together with the larger KonIQ and SPAQ datascts, compared to training on LIVEC alone. These results emphasize the power of using CLIP as prior with minimal adaptation, it allows us to leverage CLIP's perceptual knowledge, which is likely well-aligned with the shared properties across different datasets. When comparing our multi-dataset training against previous models on these datasets, the improvement is even more significant (e.g., a relative improvement of 5.6% over the previous SOTA on EmoROT-6). Our multi-dataset models achieve improved SOTA results across all datasets and tasks.
the small LIVEC dataset (which contains only 1,126 exam-
6 Interpretability and Visualizations
We visualize our model's attention heads to gain deeper insight into its decision decision-making process. Specifically, we focus on analyzing attention across all layers to identify the heads most critical for prediction. To achiove this, we employ an automated allention masking approach to systematically determine the importance of individual heads. First,, an image is passed through the model to obtain a baseline prediction. Then, the same image is processed again, but this time the CLS token's attention for the target head is replaced with a uniform, equal-weighted map. By comparing the resulting prediction with the baseline (i.e., before and after masking), we can determine whether the head's attention is significant for prediction. A significant drop in performance indicates that the head is important. Repeating this process over many images and averaging the prediction differences allows us to quantify the importance of each head.
For the most influential (automatically-ddeeece differences. A few such images are presented in fig. 4 along with differences between attention heatmaps of PerceptCIIP and the original CLIP. This comparison highlights the shift in altention, revealing how our fine-tuned model reallocates focus to different image regions, revealing the features it considers relevant for perceptual property prediction.
heads, we select images that exhibit the largest prediction
Figure 4 displays visual examples for both emotion and memorabilily predictions. The atlention maps show that the model attends to semantically relevant regions for each emotion. For instance, in the image classified as fcar-inducing, the model focuses on blood, a strong visual cue for fear. For anger, it attends to fire, while for excitement, it highlights the girl with an open mouth and a 'like' hand gesture (see more examples in Figures A.1 and A.2). For memorability, the model mainly focuses on the most. distinctive and recognizable
9

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Emotions
Memorability
Contentment
Fear
Excitement Amusement
Sadness
Anger
Figure 4: Attention Shift Toward Perceptual Cues. We present images along with the differences in their attention maps betwven our PercepICLIP model and pretrained CLIP vision encoder (displaying results from critical attention heads that most influence the perceptual predictions). This highlights the shift in attention, revealing how our model reallocates focus to perceptually meaningful regions.
objects in the image. For example, in image containing multiple elements, the model attends to chimneys with smoke, which may contribute significantly to memorability. Altogether, these demonstrate that fine-tuned model successfully learns to focus on perceptually meaningful image features.
7 Ablation Study
In this section, conduct an ablation study to analyze the impact of different design choices in our approach.
Training Strategy. We aim to investigate how to elfectively leverage CLIPP's perceptual prior while adapt- ing it to different tasks. We compare three different training strategies: (1) full fine-tuning of CLIP's vision encoder, (2) freezing CLIP and training only the MLP head, and (3) our chosen approach finc-tuning CLIP by using LoRA adaptation (of the allention heads). As shown in Table 6 full fine-tuning significantly degrades the performance. This is likely due to catastrophic forgetting, where the model loses its strong pre-trained prior and fails to generalize effectively. While training only the MLP (with frozen CLIP weights) yields better results than fully fine-tuning CLIP, it still falls short compared to using LoRA. Using LoRA adaptation provides the best. balance, allowing the model fine-tune efficiently while preserving CI.IP's strong perceptual prior. Ablation on our two-step multi-dataset training can be found in the Appendix A.3
Table 6: Impact of Training Strategies. We compare three training strategies for the CLIP visual encoder-full full fine-tuning freezing, and LoRA adaptation. The LoRA-based approach (which used our framework) outperforms both full fune-luning and freezing, striking the best balance between adaplation and preserving CLIP's perceplual prior.
IQA
VEA
Memorability
Training
KonIQ-10k
EmoSet-8
LaMem
Procedure
SRCCT
PLCC1
ACC
SRCCT
MSEJ
Full Fine-tuning CLIP
0.888
0.904
0.715
0.618
0.0466
Frozen CLIP
0.894
0.913
0.845
0.691
0.0081
LoRA for CLIP
0.945
0.954
0.853
0.741
0.0069
10

---

Published in Transactions on Machine Learning Research (07/)025)
MLP vs. Linear Layer We explored the impact of different head architectures by comparing a simple linear layer with the MLP head used in our main model. As shown in Table 7 the choice of head has a minor effect on performance in the IQA and VEA tasks, with both architectures performing competitively. However, the MLP head demonstrates better performance on the memorability task. In all tasks, both heads consistently achieve SOTA results using our approach.
Table 7: Effect of Different Heads on Performance. We compared the performance of a simple linear layer and the MLP head used in our main model. The choice of head has a minor effect on performance in the IQA and VEA tasks. However, the MLP head shows better performance on the memorability task.
IQA
VEA
Memorability
Method
KonIQ-101 10k
EmoSet-8
LaMem
SRCCI
PLCC1
ACC1
SRCC
MSEL
Linear Layer
0.943
0.953
0,8526
0.734
0,0077
MLP (Ours)
0.945
0.954
0.8528
0.741
0.0069
Backbone Choice. To assess the role of different vision backbones as perceptual priors, we experimented with 3 different pre-trained backbone models (MAE, DINOv2, and CLIP, all using ViT-L/14), while keeping the rest of the architecture and LoRA configuration the same. As shown in Table [A.8] while our approach performs well also with DINOv2, CLIP achieves the best results across all three tasks. This provides further evidence that CLIIP effectively captures perceptual properties, making it a strong prior for perceptual tasks.
Effect of VIT Size. To assess the impact of vision transformer size on performance, we compared three CLIP vision encoder variants: VIT-B/16 B/16, VIT-B/32, and ViT-L/14. The results, summarized in Table A.7 indicate that ViT-B/16 outperforms ViT-B/32 suggesting that a finer patch resolution contributes to better perceptual understanding. Additionally, ViT-L/14 achieves the best results across all tasks, reinforcing our choice to use it as our backbone.
8 Conclusion
We propose a simple unified framework for visual perceptual tasks, which leverages the implicit yet rich perceptual knowledge of CLIP. The human-annotated image captions in CLIP's training data contain also human sentiments and emotions, which is what probably makes CLIP particularly strong prior for hu- man judgment. We apply minimal task-specific adaptations, aiming to balance preserving CLIP's strong perceptual prior, while allowing necessary task-specific flexibility. Remarkably, this lightweight yet effec- tive framework produces models that achieve state-of-the-art performance on 3 important perceptual tasks, as well as demonstrates impressive cross-dataset generalization capabilities. This suggests that complex, task-specific designs and knowledge are not necessarily required for modeling perceptual tasks.
Acknowledgment:
This project was funded by the European Union (ERC, MindReading, 101142115).
References
Jochen Antkowiak, T Jamal Baina, France Vittorio Baroncini, Noel Chateau, France FranceTelecom, Antonio Claudio França Pessoa, F Stephanie Colonnese, Italy Laura Contin, Jorge Caviedes, and France Philips. Final report from the vidco quality experts group on the validation of objective models of vidco quality assessment march 2000. Final report from the video qualily experts group on the validation of objective models of video quality assessment march, 10, 2000.
Sheng Bi, Andrew Porkins, Huan Chen, and Benjamin Phifer. Does your boss prefer iced tea advertisements? the effect of feeling powerful on consumers' preference for cold advertisements. Journal of Advertising, 53 (3):377-30 -3677 2024.
11

---

Published in Transactions on Machine Learning Research (07/2025)
Damon M Chandler. Seven challenges in image quality assessment: past, present, and future research. International Scholarly Research Notices, (3900)))55 2013.
Simuo Deng, Ge Shi, Lifang Wu, Lehao Xing, Wenjin Hu, Heng Zhang, and Ye Xiang. Simemotion: A simple knowledgeable prompt tuning method for image emotion classification. In International Conference on Dulabase Systems for Advanced Applications, pp. 222 229. Springer, 2022.
Sinuo Deng, Lifang Wu, Ge Shi, Lehao Xing, Meng Jian, Ye Xiang, and Ruihai Dong. Learning compose diversified prompts for image emotion classification. Computational Visual Media, pp. 1 1-15, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprinl arXiv:2010.11 119299 2020.
Merel van Duivenboden. The Effect of Images and their Memorability on Foreign Vocabulary Learning. PhD thosis, 2023.
Jiri Fajtl, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. Amnet: Memorability estimation with attention. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6363-6372, 2018.
Yuriing Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smart- phone photography. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- lion, pp. 3677-377770 3686, 2020.
Deepti Ghadiyaram and Alan C Bovik. Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions Image Processing, 2571):372 387, 2015.
James J Gibson. The ecological approach to visual perception: classic edition. Psychology press, 2014.
Lore Goetschalekx and Johan Wagemans, Memcat: new category-based image set quantified on memora- bility. PeerJ, 7:e8169, 2019.
Alireza Golestaneh, Saba Dadsetan, and Kris M Kitani. No-reference image quality assessment via trans- formers, relative ranking, and self-consistency. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1230, 2022.
Thomas Hagen and Thomas Espeseth. Image memorability prediction with vision transformers. arXio preprint xrX:e:2301.08//1 2023.
Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 29:4041 4056, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. Understanding the intrinsic memorability of images. Advances in neural information processing systems, 24, 2011a.
Phillip Isola, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. What makes an image memorable? Tn CVPR 2011, pp. 145-152. IEEE, 2011b.
Le Kang, Peng Ye, Yi Li, and David Docrmann. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733- 1740, 2014.
12

---

Published in Transactions on Machine Learning Research (07/2025)
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVI international conference on computer vision, pp. 5148-5151 2021.
Aditya Khosla, Akhil S Raju, Antonio Torralba, and Aude Oliva. Understanding and predicting image memorability large scale. In Proceedings of the IEEE international conference computer vision, pp. 2390-2398, 2015.
Max A Kramer, Martin N Hebart, Chris I Baker, and Wilma A Bainbridge. The features underlying the memorability of objects. Science advances, 9 (11))xeadd2981, 2023.
Mariska E Kret and Beatrice De Gelder. review on sex differences processing emotional signals. Neuropsychologio, 50(():1211-122) 2012.
Daniel D Leeds, Darren A Seibert, John A Pyles, and Michael J Tarr. Comparing visual representations across human fmri and computational vision. Journal of vision, 13(13):25 25, 2013.
Marco Leonardi, Luigi Celona, Paolo Napoletano, Simone Bianco, Raimondo Schettini, Franco Manessi, and Alessandro Rozza. Image memorability using diverse visual features and soft attention. In Image Analysis and Processing-ICIAF ICIAP 2010: 20th International Conference, Trento, Italy, September 9-13, 2019, Proceedings, Part II 20, pp. 171 180. Springer, 2019.
Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-10k: A large-scale artificially distorted iqa database. In 2019 Eleventh International Conference on Qualily of Mullimedia Experience (QoMEX), pp. 1-3. IEEE, 2019.
Zhong-Lin Lu, Tianmiao Hua, Chang-Bing Huang, Yifeng Zhou, and Barbara Anne Dosher. Visual percep- tual learning. Neurobiology of learning and memory, 95(2):145 151, 2011.
Kede Ma, Wentao Liu, Kai Zhang, Zhengfang Duanmu, Zhou Wang, and Wangmeng Zuo. End-to-end blind image quality assessment using deep neural networks. IEEE Transactions on Image Processing, 27(3): 2017.
David Marr. Vision: A compulational investigation into the luman representation and processing of visual information. MIT press, 2010.
Iván Martin--ernández,, Ricardo Kleinlein, Cristina Luna-Jiménez, Manuel Gil-Martín, and Fernando Gernández-Marttnez. Video memorability prediction from jointly-learnt semantic and visual features. In Proceedings of the 20th international conference on content-based multimedia indering, pp. 178 182, 2023.
Coen D Needell and Wilma A Bainbridge. Embracing new techniques in deep learning for estimating image memorability. Computational Brain & Behavior, 5(2):168 184, 2022.
Eiman Negm and Passent Tantawi. Investigating the impact of visual design on consumers' perceptions towards advertising. International Journal of Scientific and Research Publications, 5(4):1 9, 2015.
A Deep Bilinear Convolutional Neural Network. Blind image quality assessment using a deep bilinear convolutional neural network.
Kuan-Chuan Peng, Amir Sadovnik, Andrew Gallagher, and Tsuhan Chen. Where do emotions come from? predicting the emotion stimuli map. In 2016 IEEE international conference on image processing (ICIP), pp. IEEE, 2016.
Rik Pieters, Michel Wedel, and Rajeev Batra. The stopping power of advertising: Measures and effects of visual complexity. Journal of marketing, 74(5):48 60, 2010.
Nikolay Ponomarenko, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Lina Jin, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Foderica Battisti, al. Color image databasc tid2013: Peculiarities and preliminary results. In Europeon workshop on visual information processing (EUVIP), pp. 106-111. TEEE, 2013.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Guanyi Qin, Runze Hu, Yutao Liu, Xiawu Zheng, Haotian Liu, Xiu Li, and Yan Zhang. Data-efficient image quality assessment with attention-panel decoder. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 2091-2100, 2100, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Tianrong Rao, Xiaoxu Li, Haimin Zhang, and Min Xu. Multi-level region-based convolutional neural network for image emotion classification. Neurocompuling, 333:429 439, 2019.
Avinab Saha, Sandeep Mishra, and Alan C Bovik. Re-iqa: Unsupervised learning for image quality assessment in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5846 5855, 2023.
Kevin L Sample, Henrik vedt, and S Adam Brasel. Components of visual perception in marketing contexts: A conceptual framework and review. Journal of the Academy of Marketing Science, 48:405 421, 2020.
Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on image processing, 15(11):3440 3451, 2006.
NV Shemyakina, Zh V Nagornova, AV Grokhotova, VA Galkin, VA Vasenkina, SV Biryukova, and Yu G Potapov. study of oeg characteristics of aesthotic perception and assessment of painting works during museum visits: Neuroaesthetic study. Human Physiology, 50(4):345 357, 2024.
Adva Shoham, Idan Daniel Grosbard, Or Patashnik, Daniel Cohen Or, and Galit Yovel. Using deep neural networks to disentangle visual and semantic information in human perception and memory. Nature Human Behaviour, pp. 16, 2024.
Hammad Squalli-Houssaiie Ngoc QK Duong, Marquant Gwenaëlle, and Claire-Hélène Demarly. Deep learn- ing for predicting image memorability In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2371 2375. IEEE, 2018.
Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3667-3676, 2020.
James Z Wang, Sicheng Zhao, Chenyan Wu, Reginald B Adams, Michelle G Newman, Tal Shafir, and Rachelle Tsachor. Unlocking the emotional world of visual media: An overview of the science, research, and impact of understanding emotion. Proceedings of the IEFF, 111(10):1236-11 2023a.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intellience,, volume 37, pp. 2555 2563, 2023b.
Shunye Wang. Online learning behavior analysis based on image emotion recognition. Thaitement du Signal, 38(3), 2021.
Lifang Wu, Heng Zhang, Sinuo Deng, Ge Shi, and Xu Liu. Discovering sentimental interaction via graph convolutional network for visual sentiment prediction. Applied Sciences, 11(4):1404, 2021.
Kangmin Xu, Liang Liao, Jing Xiao, Chaofeng Chen, Haoning Wu, Qiong Yan, and Weisi Lin. Boosting image quality assessment through efficient transformer adaptation with local feature cuhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2662-2672, 2024a.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Liwen Xu, Zhengtao Wang, Bin Wu, and Simon Lui. Mdan: Multi-level dependent attention network for visual emotion analysis. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 9479 9488, 2022.
Qinfu Xu, Yiwei Wei, Shaozu Yuan, Jie Wu, Leiquan Wang, and Chunlei Wu. Learning emotional prompt fcatures with multiple views for visual emotion analysis. Information Fusion, 108:102366, 2024b.
Jingyuan Yang, Xinbo Gao, Leida Li, Xiumei Wang, and Jinshan Ding. Solver: Scene-object interrelated visual emotion reasoning network. IEEE Transactions on Image Processing, 00:8686-8701, 2021a.
Jingyuan Yang, Jie Li, Xiumei Wang, Yuxuan Ding, and Xinbo Gao. Stimuli-aware visual emotion analysis. IEEE Transaclions on Image Processing, 30:7432 7445, 2021b.
Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Emoset: A large-scale visual emotion dataset with rich attributes. In Proceedings of the IEER/CVF International Conference on Computer Vision, pp. 20383 20394, 2023.
Jufeng Yang, Dongyu She, Yu-Kun Lai, Paul L Rosin, and Ming-Hsuu Hsuan Yang. Weakly supervised coupled networks for visual sentiment analysis. In Proccedings of the IEEE conference on computer vision and pollern recognilion, pp. 7584 7592, 2018.
Yanyin Yao, Xue Mei, Jianping Xu, Zhichuang Sun, Cheng Zeng, and Yuming Chen. Vlm-emo: Context- aware emotion classification with clip. In 2024 5th International Seminar on Artificial Intelligence, Net- working and Information Technology (AINTT), pp. 1615-1620. IEEE, 2024.
Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3575-3585, 2020.
Junyong You and Jari Korhonen. Transformer for image quality assessment. In 2021 IEEE international conference on image processing (ICIP), pp. 1389-1393 IEEE, 2021.
Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14071 14081, 2023.
Yiming Zhang, Ying Hu, Xiongkuo Min, Yan Zhou, and Guangtao Zhai. fmrr exploration of visual quality assessment. arXiv preprint ar 1111024404.1166, 2024.
Kai Zhao, Kun Yuan, Ming Sun, Mading Li, and Xing Wen. Quality-aware pre-trained models for blind image quality assessment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22302 22313, 2023.
15

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
A Appendix
A.1 Further Technical Details
Data Augmentations. In our training we used a common data angmentation techniques which are de- tailed below. Image Quality Assessment: Following Xu et al. (2024]] during training, each image is augmented into 3 to 10 replicas, depending on the dataset. Each replica undergoes independent random horizontal and vertical flips (p 0.5) and random cropping to 224x224. Predicting Image Memorabil- ity: Before leeding the images into the model, we resize the shortest side of each image to 224 pixels while maintaining the aspect ratio. A center crop of size 224x224 is then applied. Visual Emotion Analysis: We apply data augmentation inspired by Yang et al. (2023), including random resized cropping 224x224 and horizontal flipping (p = 0.5). During inference, images are resized so that their shorter side is 224 pixels while preserving the aspect ratio, followed by center crop.
Training Runtime Training on an NVIDIA L40S GPU takes up to a day for the largest dataset (e.g.. EmoSet with approximately 100,000 images) and only few hours for smaller datasets when run for epochs. In practice, early stopping significantly reduces the overall training time.
4-parameter logistic regression correction. For the Image quality assessment task, we report Pear- son's Linear Correlation Coefficient (PLCC) with a 4-parameter logistic regression correction, proposed by Antkowiak et al. (2000) and following Xu et al. 20244).. The 4-parameter logistic function is defined as:
82
V' B2 +
#-88 1
A4
where y/ is the transformed prediction, is the original model output, and 81, 82, B3, and B4 are the four parameters. This function is fitted to the model predictions using non-lincar regression, optimizing its parameters to best match the subjective scores. Once the predictions are adjusted using this transformation, PLCC is computed to measure the correlation between the corrected predictions and human ratings.
A.2
Additional Results
A.2.1 Image Memorability Additional Results
Table A.1 iilustrates the generalization capability of our model compared to two other models on the memo- rability task. ResMen[Neodell & Bainbridge (2022) and ViTMen Hagen & Esposeth (2023) were trained using combined dataset of LaMem (58,741 images) Khosla al. (2015) and MemCal (10,000 images) Goetschal- ckx & Wagemans (2019), while our model was trained exclusively the LaMem dataset. All models were evaluated on the Things memorability dataset. Our model outperforms the others, demonstrating superior generalization capabilities.
Table A.l: Generalization Results on Things Dataset.
THINGS
Method
SRCC
resMem
0.22
ViTMem
0.30
PerceptCLIP (Ours)
0.34
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table A.2: Model Performance on SUN Image Memorability dataset. Average SRCC and MSE
across 25 splits of the SUN dataset reported, showing that our model, PerceptCIIP, surpasses all previous methods.
SUN
Method
SRCC1
MSE
MemNet
0.610
unknown
A.MNet.
0.649
0.011
PerceptCLIP (Ours)
0.708
0.014
A.2.2 Visual Emotion Analysis Additional Metrics
In addition to the results presented in the paper, provide additional evaluation metrics in table A.3 comparing our model with two leading existing models. These metrics include: Mean Average Precision (mAP): Measures the area under the precision-recall curve, assessing the model's ability to rank positive examples higher than negatives across different thresholds. Macro F1 Score (m-F1): Computes the score for each class independently and averages them, treating all classes equally regardless of dataset imbalance. Weighted F1 Score (w-F1): The harmonic mean of precision and recall, weighted by class support, ensuring that both high precision and recall contribute fairly across imbalanced datasets.
By incorporating these diverse metrics, ensure more comprehensive assessment of model performance beyond simple accuracy. Our model consistently outperforms the others across all metrics, demonstrating its robustness and effectiveness in handling complex classification tasks.
Table A.3: Additional Metrics for the Emotion Recognition Task: We report Accurecy, mean Average Pre- cision (mAP), macro-F1 (m-F1), and weighted-Fi (w-F1) on the EmoSet and EmotionROI datasets. Our approach oulperforms cxisting mothods, achieving the best resulls across all metrics.
EmoSet-8
EmotionRO1-6
Method
Accuracy
mAP
m-F1
w-F1
Accuracy
mAP
m-F1
w-F1
SimEmotion
0.7906
0.7387
0.7514
0.7894
0.7054
0.7054
0.7082
0.7027
MVP
0.8192
0.8003
0.8059
0.8047
0.7189
0.7116
0.7194
0.7148
Ours
0.8528
0.9336
0.8592
0.8526
0.7485
0.8081
0.7432
0.7445
A.2.3
We conducted joint training on multiple datasets for the same task, where the CLIP LoRA adaptation is shared, and each dataset has its own MLP. We present the results of these models on all datasets in tables A.4 to A.6 These results show a significant improvement on smaller datasets, leveraging multi-dataset training with our approach, while achieving comparable results on larger datasets. Overall, our models trained on multiple datasets achieve state-of-the-art results across all tasks and datasets.
Table A.4: Multi- Dataset us. Single-Datac Dataset Results on IQA: We compare models trained on individual dalasels with those trained on meulliple dalasels. Mulli-dulasel models were trained either wuthentic distortions (LIVEC, KonIQ-10k, SPAQ) or synthetic distortions (LIVE, TID2019, KADID-10K). Multi- Dalaset models improves performance smaller datasets while maintaining comparable results larger ones.
LIVEC
KonIQ-10k
SPAQ
LIVE
TID2013
KADID-10K
Method
SRCC
PLCC
SRCC
PLOC
SRCC
PLCC
SRCC
PLOC
SRCC
PLCC
SRCC
PLOC
PerceptCLIP
0.908
0.923
0.945
0.954
0.930
0.933
0.980
0.982
0.887
0.908
0.932
0.956
PerceptCIIP Multi-Dataset.
0.922
0.933
0.944
0.953
0.930
0.934
0.980
0.983
0.900
0.915
0.953
0.956
17

---

Published in Transactions on Machine Learning Research (07 2025)
Table A.5: Mulli-Datased vs. Single-Dataset Train- ing for Memorability Prediction: The malli-dataset ap- proach improves performance on THINGS while maintain- ing comporable results on LaMem.
Table A.6: Multi-Datasee Dataset us. Single-Der Dataset
Training for Emotion Recognition: Trašning Emotion together with EmoSet-8 improves perfor- monce EmotionRO1-6 while maintaining comparable results to single dataset training on EmoSet-8.
LaMem
THINGS
Method
SRCC
MSE
SRCC
MSE
Method
EmoSet-8
RmotionROI-6
PerceptCLie PeII Multi-Dataset
0.741
0.0069
0.452
0.0058
PeIC
0.8528
0.7485
0,740
0.0060
0.454
0.0054
PerCeptCIIP Multi-Datasct
0.8511
0.7591
A.3 Ablation Study
In this section, we present additional details from our ablation studies, including comprehensive tables, well as further ablations of the multi-dataset training procedure.
Different CLIP VIT Sizes. Table A.7 Shows the results of using different VIT sizes for the CLIP vision encoder backbone in our architecture.
Table A.7: Performance of Different CLIP Variants: We trained our model using different sizes of ViT the CLIP vision encoder. The results show that ViT-B/16 outperforms ViT-B/32, suggesting that finer potch resolution improves perceptual understanding. VCT-L/14 achieves the best overall performance, reinforcing its use our backbone model.
IQA KonIQ-108
VEA EmoSet-8
Memorability LaMem
Method
SRCCT
PLCC1
ACC1
SRCC1
MSEJ
ViT-B/32
0.932
0.944
0.8303
0.724
0.0073
VIT-B /16
0.938
0.949
0,8422
0.731
0,0071
VIT-L/14 14
0.945
0.954
0.8528
0.741
0.0069
Different Vision Backbones. Table A.8
shows the effect of different vision backbones performance
(including TULIP model)
Table A.8: Efffec of Vision Backbones on Performance: We trained model with different backbones,
applying LoRA to the k and a layers. CLIP outperforms MAE and DINOv2 across all tasks, demonstrating its stronger perceptual prior.
IQA KonlQ- 10k
VEA EmoSet-8
Memorability LaMem
Backbone
SRCC
PLCC
ACC*
SRCC1
MSEJ
MAE
0.921
0.935
0.7681
0.699
0.0078
D2
0.937
0.950
0.8252
0.729
0.0071
TULIP
0.942
0.950
0.841
0.682
0.0076
CLIP
0.945
0.954
0.8528
0.741
0.0069
Multi-Dataset Training In our multi-dataset setting, we use two-step training procedure, where first train the shared CLIP LoRA weights and the MLP for cach datasct together with all datascts. In the sccond step, each dataset's MLP is trained separately while the CLIP LoRA weights are fixed and not optimized. We have conducted an ablation study in table [A.9 to demonstrate that the performance improvement is not solcly due to the two-phase training procedure. We compare three approaches: (1) training separate model for each dataset using our single-dataset approach, (2) training a separate model for each dataset using two-phase procedure (first training the entire model, then fine-tuning only the ML. while keeping
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
CLIP frozen), and (3) training single model on multiple datasets using our multi-dataset approach with the two-phase training. The multi-datasel approach leads to better performance, while the two-phase training with a single dataset yields comparable results to the single-dataset approach.
Table A.9: Ablation Study for Multi-Dataset Training.
IQA
VEA
Memorability prediction
Method
LIVEC
TID2013
EmoRO1-6
THINGS
SRCCI
PLCC1
SRCCt
PLCC1
ACC1
SRCC1
MSEJ
PerceptCIII Single-Datasct
0.908
0.923
0.887
0.908
0.7485
0.452
0.0058
PereeptCIIIP 2-Phase Training Singlo-Datasee
0.907
0.923
0.889
0.909
0.7500
0.451
0.0055
PerceptCLIP 2-Phase Training Aklli-Dataaee
0.922
0.933
0.900
0.915
0.7591
0.454
0,0054
LoRA Parameters We have systematically experimented with different values r and a on our main model using the KonlQ-10k dataset from the IQA task (see tables A.10 and A.11). As shown those tables, these parameters have minimal impact on model performance.
Table A.10: SRCC Performance Across LoRA Configurations: Spearman Rank Correlation Coef- ficient (SRCC) values for different combinations of LoRA rank (r) and scaling factor (a) the kon10k dataset within the IQA task.
r
4
8
16
0.945
0.944
0.944
16
0.944
0.945
0.945
32
0.945
0.944
0.944
Table A.11: PLCC Performance Across LoRA Configurations: Pearson Linear Correlation Coefficient (PLCC) values for different combinations of LoRA rank (r) and scaling factor (a) the konl0k datasct within the IQA task.
4
8
16
8
0.954
0.954
0.954
16
0.954
0.954
0.954
32
0.954
0.953
0.953
19

---

Published in Transactions on Machine Learning Research (07 (025)
CLIP Attention
PerceptCLIP Attention
CLIP Attention
PerceptCLIP Attention
Image
Image
Excitement
Amusement
Contentment
Awe
Figure A.l: Attention Mask Visualization Positive Emotions. We present images alongside their corre- sponding attention maps from the pretrained CLIP vision encoder and our PerceptCLIP model (displaying results from critical attention heads that most inftuence the perceptual predictions). This highlights the shift in attention, reveating how our model reallocates focus to perceptually meaningful regions.
CLIP
PerceptCLIP Attention
CLIP Attention
PerceptCLIP Attention
Image
Image
Attention
Fear
Sadness
Disgust
Anger
Figure A.2: Attention Mask Visualization Negative Emotions. We present images alongside their corre- sponding atlention maps from the pretrained CLIP vision encoder and PerceptCLIP model (displaying results from critical attention heads that most influence the perceptual predictions). This highlights the shift in attention, revealing how our model reallocates focus perceptually meaningful regions.
20

---

Published in Transactions on Machine Learning Research (07 (075)
A.4 Error Analysis and Model Limitations
Visual Emotion Analysis. Figure A.3 presents the conlusion matrix for PLIcs classification per- formance on EmoSet. Most misclassifications occurred between semantically similar emotions (e.g., amuse- ment vs. contentment, fear vs. disguss)-conuusiionn confusions that are also common among human annotators (e.g.. we fear bugs or feel disgusted by them?).
A notable pross-valence confusion was observed between sadness (a negative emotion) and contentment (a positive one). As shown in Figure A.4 many of these images feature animals, suggesting a potential model bias associating animals with positive emotions. Another frequent confusion occurred between anger and excitement, often in images showing people with open mouths-suce
during protests or singing (see Figure A.5 These findings highlight the subjectivity of emotion perception and point to a possible model bias toward interpreting open-mouthed expressions as excitement.
Memorability Prediction. FigureA.66 presents a histogram comparing the ground-truth and predicted memorability scores on the LaMEM test set (split 1). We observed that PerceptCLIP's predictions are more concentrated around the mean. To further explore this behavior, visualized the images with the largest prediction errors at both extremes of the memorability spectrum, shown in fig. A.7 However, these examples did not reveal any consistent or interpretable visual patterns, making it difficult to pinpoint specific failure mode.
Image Quality Assessment. Figure A.8 shows examples of images with the highest prediction errors in the IQA task for the KonIQ10k dataset. We applied z-score normalization to both the predicted and ground- truth scores before identifying these cases. The failure cases exhibited only subtle patterns, potentially involving complex images that require high-level understanding.
Confusion Matrix
fear
1664
144
67
36
20
22
12
2500
disgust
91
4
tadneess
anger
72
19
1562
10
32
10
innussmeett
True
20
exctement
3000
179
2708
10
17
11
45
55
1968
148
contentment
210
105
1961
fear
disqust
sadness
anger
amusement
excitement
contentment
Labels
Figure A.3: Visual Emotion Analysis Confusion Malriz. PerceptCLIP's emolion classificalion performance on the EmoSet lest set across emotion categories. Rows represent ground-trui Iruth labels; colamns represend predicted labels.
21

---

Published in Transactions on Machine Learning Research (07 (025))25)
Ground-Truth: Sadness Predicted: Contentment
Figure A.4: Enamples of Sadness Misclassified as Contentment. Images where the ground-truth label is sadness but PerceptCLIP predicted contentment. Mony of these examples feature animals, suggesting that the model may associate animal imagery with positive affect, leading to cross-valence misclassifiaation
Ground-Truth: Anger Predicted: Excitement
WORLD NAKED BIKE RIDE
Figure A.5: Examples of Anger Misclassified Excilement. Images where the ground-trulh label is anger but PerceptCIII predicted excitement. Many of these ezamples depict people with open mouths, such as during protests or intense singing, suggesting a potential model bias toward interpreting such expressions excitement. This highlights the subjectivity of emotion perception and the inftuence of content-spedd specific visual cues.
22

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Distribution of True vs. Predicted Scores
1400
True Scores
1200
Predicted Scores
1000
Count
800
600
400
200
0.0
0.2
0.4
0.6
0.8
1.0
Memorability Score
Figure A.6: Distribution of Ground- Trulhd es. Predicted Memorabillty Scores. Hislogram comparing ground- trulh and predicted memorabilly scores on the LaMem lest set splill. The predicted scores are more concentrated around the center of the scale, indicating that the model tends underestimate highly memorable images and over- estimate less memorable ones.
High Memorability, Highest Error
Low Memorability, Highest Error
0.67
P:
0.95
0.95
0.71
0.772
0.95 0.72
0.73
7.955
P. 0.73
0.0.6
T: 0.35
T:
0.34
0041
Figure A.7: Ezamples of Images with Extreme Prediction Errors. Visualization of images from the LaMem test set with the largest prediction errors at both the low and high ends of the memorability spectrum. Each image is annotated with its ground-trute (T:) and predicted (P:) memorability scores.
23

---

Published in Transactions on Machine Learning Research (07/2025)
Top Failure Cases (Predicted vs True IQA Scores)
Main the intan lit- Ftcl
T:-1.6
F-1.05 0.05
-3
-2.19
T:10.11
Tc-1.41
0.0.0
T0996
0.88
-3..47
P:.0.85
11.5
0.47
P: -0.77
P1.1
0.10
Figure A.8: High-Erro Examples 1QA. Visualization of images from the KonlQ10k test with the largest differences between predicted and ground-truth quality scores. Both predicted and ground- scores were standardized using z-score normalization. Each image is annotated with ground- bruth (T:) and predicted (P:) quality scores.
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
B Broader Impact Statement
This work introduces PLIP framework for visual perceptual tasks such as image memorability, image quality, and visual emotion analysis, built upon CLIP. While PerceptCLIP leverages CLIP's alignment with human judgment, it may inherit its pretraining data, which consists of large-scale internet content, that can contain social and cultural biases. As a result, PerceptCLIP may reflect such biases in its predictions. Additionally, while our model predicts general perceptual responses based on crowd-sourced annotations, future extensions involving lensionlization applications to personal content. could raise ethical and privacy concerns.
25


Reviews:
[{"review_id": "eH8Ze674At", "reviewer": "Reviewer_vAhY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\nThe paper presents a single, general solution for multiple perceptual tasks by re-using the same CLIP-based model structure with minimal changes. The use of LoRA to fine-tune only CLIP’s attention layers means very few parameters are learned for each task.\n\nThe proposed method achieves strong results that meet or exceed SOTA on all three evaluated tasks. These performance gains, combined with the cross-dataset generalization tests, demonstrate that the CLIP-based solution is not only competitive but more robust across data variations than specialized models.\n\nThe model does not rely on any task-specific feature engineering, auxiliary networks, or human-provided insights. The paper’s clarity in describing this unified framework is good, and the rationale is intuitively well-motivated.\n\n[Weaknesses]\nThe central idea of leveraging a large pre-trained model (CLIP) for downstream tasks with lightweight fine-tuning is incremental and not particularly novel. LoRA and related parameter-efficient fine-tuning techniques are established methods, and the decision to tune only attention layers is more an implementation detail than a deep insight.\n\nThe paper’s title and narrative emphasize a unified approach for perceptual tasks, implying a general solution across many tasks. However, the experimental validation is limited to three tasks, and the approach treats each task separately with separate fine-tuning and heads. As a result, the claim of a general unified solution feels somewhat overstated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a simple unified architecture that leverages the pre-trained CLIP model as a foundation for various visual perceptual tasks like Image Memorability Prediction, No-Reference Image Quality Assessment, and Visual Emotion Analysis. Across all three tasks, the proposed approach reportedly achieves SOTA results, outperforming specialized prior methods on standard benchmarks. An important claimed benefit is improved generalization: because perceptual data is scarce and often overfits, the authors test cross-dataset transfer and report substantially higher correlations than past models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should explicitly clarify in the introduction and title what “unified” entails in this work. It would help to state that the framework is unified at the architectural level but that separate fine-tuning is done per task.\n\nTo give a more nuanced picture, the authors should add some analysis of where PerceptCLIP falls short. For instance, are there certain kinds of images where the model’s predictions disagree with human judgments?", "broader_impact_concerns": "CLIP was trained on a vast amount of internet data, which is known to include various social and cultural biases. If PerceptCLIP inherits these biases, its predictions of human judgments may be skewed or unfair for certain groups.\n\nVisual emotion analysis in particular can be sensitive. If a model can predict emotional reaction to images, it might be one step away from trying to infer a viewer’s emotional state from their images or social media content.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u09tbyoCQF", "reviewer": "Reviewer_rTfe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the paper in general, and I find the proposed approach simple and concise, yet quite powerful, which is a strong mixture in my opinion. Here's the list of strengths:\n- The proposed approach is universal across several tasks (up to the MLP head, which is task-specific);\n- PerceptCLIP utilises CLIP's pre-trained knowledge with little added trainable parameters, which makes it efficient compared to many existing approaches;\n- Three considered tasks are different in nature, e.g. image memorability prediction should be related to perceiving semantic features of the image, image quality assessment mostly related to the fine-grained non-semantic details of the image, and visual emotion analysis is related to the specific information on the image, i.e. emotions;\n- Extensive experiments show great performance across multiple datasets on all the three tasks and good generalisation across datasets. This suggests that the model does learn robust perceptual features rather than overfitting to individual datasets. \n- Good ablation study. Essential cases of using fully-frozen CLIP, fully fine-tuned CLIP and fine-tuned CLIP using proposed LoRA approach are considered, which clearly shows the advantage of the proposed LoRA-based training. Ablation on the choice of the backbone model supports the choice of the CLIP, and different version of the CLIP backbone are considered. Ablation on attention heads interpretability is really nice, as it shows that the model did actually learn meaningful features that we would like it to learn, and it's performance is likely connected to the learned features, and not some other hidden reason.\n\nWeaknesses (all except the first one is more like \"what would be nice to have\"):\n- The paper has a good ablation on many factors, but the ablation on LoRA hyperparameters is missing. The authors use r=16 and α=8, but it would be beneficial to see how robust the model is to these parameters, and, maybe, what differences in attention maps it exhibits based on the values of these parameters. Moreover, there is an ablation on the choice of backbone (MAE, DINOv2, and CLIP), and though  they are all using ViT-L/14, different choices of r and α might be the best for different backbones.\n- There're some more modern CLIP-like architectures, namely SigLIP, SigLIP2, TULIP. Would be interesting to see if their prior knowledge is better for this task\n- I am a little concerned about CLIP having some perceptual biases, e.g. biased judgement on emotions of different people. It would be beneficial to explore that, although I am not sure how important that is for such kind of papers.\n- There is only one dataset used for computing metrics on the task of image memorability. I am not sure if there exist more, but in case if not, it might be good to perform a user study to show generalisability of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel unified approach called PerceptCLIP for several image perceptual tasks, namely image memorability prediction, no-reference image quality assessment, and visual emotion analysis. The proposed approach leverages prior knowledge of the pre-trained CLIP model, which was indicated to have a good correlation with human judgement. To solve each of the perceptual tasks, the pre-trained CLIP model is fine-tuned using lightweight LoRA adapters on the weights of the attention layer (namely, key (K), query (Q), and value(V)), and the per-task trainable MLP head is added on top of CLIP embeddings, which outputs the final prediction to the task. Experimental results show that PerceptCLIP achieves state-of-the-art results on all the three tasks across multiple benchmark datasets, outperforming task-specific models that are often more complex and parameter-heavy. Moreover, the model demonstrates strong generalization, outperforming existing methods in cross-dataset evaluations, and benefits from joint training on multiple datasets from the same task. This suggests that the proposed model learns robust perceptual features rather than overfitting to individual datasets. Finally, the paper presents rich ablation study, showing benefits of utilising CLIP over other visual encoders and justifying the proposed tuning setup. \nTo sum up, PerceptCLIP proposes a unified framework for several perceptual tasks, that shows state-of-the-art results with good generalisation across multiple tasks and datasets, and benefits from universality of the architecture across these tasks, efficient training,  and interpretability of the learned features.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would only strongly request ablation on LoRA hyperparameters, i.e. r and α, at least for one (which is used in main experiments) version of CLIP backbone. Other things mentioned in the \"weaknesses\" section would be beneficial, but not crucial in my opinion.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uLkXlkeXtY", "reviewer": "Reviewer_2MdG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The proposed method is technically sound, simple yet effective\n- The experiments are comprehensive, verifying the effectiveness of three tasks.\n- Some experiments, such as dataset scaling, can provide some insight\n\nWeakness\n- The proposed method lacks technical contribution and novelty, as CLIP has already been utilized in the studied tasks in this paper, including IQA [1], emotion classification [2], and Image Memorability Prediction [3]. So, the observation about using CLIP for perception tasks is not new, and the method design using LORA to tackle these tasks is also not new. The authors also need to compare with these state-of-the-art.\n\n[1] Wang, Jianyi, Kelvin CK Chan, and Chen Change Loy. \"Exploring clip for assessing the look and feel of images.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 2. 2023.\n\n[2] Yao, Yanyin, et al. \"VLM-EMO: Context-Aware Emotion Classification with CLIP.\" 2024 5th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT). IEEE, 2024.\n\n[3] Martín-Fernández, Iván, et al. \"Video memorability prediction from jointly-learnt semantic and visual features.\" Proceedings of the 20th international conference on content-based multimedia indexing. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a unified framework built on CLIP representations for visual perception tasks, which introduces tunable LORA and MLP layers.  The authors argue that CLIP can be repurposed for various perceptual tasks without requiring bespoke architectural changes. \n\nThrough comprehensive experiments, the proposed models are demonstrated to achieve state-of-the-art results on three distinct benchmarks: image memorability prediction, no-reference image quality assessment, and visual emotion analysis. \n\nThe authors demonstrate that joint training across heterogeneous datasets for the same task yields meaningful gains, especially when specific datasets are small. This empirical finding reinforces the value of multi-dataset learning and suggests practical strategies for alleviating data scarcity in perceptual modeling.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Clarify the difference with other works, novelty, and technical contributions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)