Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 13:19:38

Abstract: Visual emotion recognition (VER) is a longstanding field that has garnered increasing attention with the advancement of deep neural networks. Although recent studies have achieved notable improvements by leveraging the knowledge embedded within pre-trained visual models, the lack of direct association between factual-level features and emotional categories, called the ''affective gap'', limits the applicability of pre-training knowledge for VER tasks. On the contrary, the explicit emotional expression and high information density in textual modality eliminate the ''affective gap''. Therefore, we propose borrowing the knowledge from the pre-trained textual model to enhance the emotional perception of pre-trained visual models. We focus on the factual and emotional connections between images and texts in noisy social media data, and propose Partitioned Adaptive Contrastive Learning (PACL) to leverage these connections. Specifically, we manage to separate different types of samples and devise distinct contrastive learning strategies for each type. By dynamically constructing negative and positive pairs, we fully exploit the potential of noisy samples. Through comprehensive experiments, we demonstrate that bridging the ''affective gap'' significantly improves the performance of various pre-trained visual models in downstream emotion-related tasks.

Main paper: 
 related works 2.1 visual emotion recognition: 
 Vision plays a crucial role in human emotional perception, and research on VER has been conducted for over two decades [64] . Early studies mainly focus on designing hand-crafted features based on psychology and art theories, including low-level elements [48] , midlevel principals [54, 60] , and high-level adjective-noun pairs [1] . Later on, the rapid advancements in deep learning reveal the superiority of learning-based features [29, 53] , catalyzing the emergence of numerous methods built upon it [43, 52] . As their performance gradually approaches the bottleneck, researchers shift focus towards perceiving emotions at multiple regions and scales [44, 51] .More recently, various correlative subfields witness increased attention, such as personalized prediction [47, 63] , domain adaptation [59, 65] and zero-shot learning [49, 55] .The rapid development of VER in recent years is inseparable from the emergence of pre-trained models. The non-trivial knowledge embedded in these models has significantly aided them in perceiving emotions [58] . However, the misalignment between pre-training objectives and downstream VER tasks leads to the "affective gap"-the absence of intermediate emotional-level features between factual-level features and emotional categories. To tackle this problem, Feng et al. [10] imitate human's visual sentiment perception mechanism and design six pre-training tasks to guide the model in learning factual-level and emotional-level features from scratch. Although achieving impressive performance gains for several backbones, they expand the original network size by threefold and demand extensive annotated data from diverse datasets, imposing significant computational costs on the training process. In contrast to them, we enhance the emotional perception of pre-trained visual models on top of the original factual-level knowledge, significantly reducing the demand for data.Language and vision have their respective pros and cons in different aspects. In recent years, many studies have explored harnessing the advantages of language to enhance visual models. Out of them, CLIP [28] demonstrates the efficacy of text supervision in guiding the acquisition of robust and transferable visual representations. Following works are proposed by considering such as fine-grained alignment [3, 30, 66] , additional supervision [7, 66] , external knowledge [11, 33] and non-contrastive objective [67] .Inspired by their success, EmotionCLIP [58] transfers this paradigm to VER, by guiding the visual model in learning emotion-level features with a subject-aware context encoding and a sentimentguided contrastive learning. It incorporates the knowledge of a pre-trained textual model during training, similar to ours, but only employs it for computing relationships between samples. Additionally, their approach requires substantial video and text data strictly correlated in both factual and emotional aspects. However, they haven't released their training data yet, and to our knowledge, no publicly available dataset meets this requirement. Consequently, subsequent works following them entail not only significant training costs, akin to [10] , but also necessitate additional expenses for data collection. In comparison, our comprehensive utilization of pre-training textual knowledge and the noisy image-text pairs mitigates the demands on both the quantity and quality of training data and enables stronger performance on downstream tasks. 
method: 
 The pipeline of our proposed PACL is demonstrated in Fig. 2 . It comprises three stages: Dataset Partition (Section 3.1), Unimodal Sample Cluster (Section 3.2) and Adaptive Contrastive Learning (Section 3.3). We provide a detailed explanation of each stage below. We employ CLIP [28] as the grounding evaluator of the factual connection, with its visual and textual encoder denoted by 𝐶 𝑣 (•) and 𝐶 𝑡 (•). Given a sample (𝑣 𝑖 , 𝑡 𝑖 ), we consider it is factual-matched if its CLIP-similarity score is larger than a threshold 𝜎:EQUATIONOtherwise, we consider it factual-mismatched. We employ DeepSentiBank [2] , denoted by 𝐷 (•) and pre-trained BERTweet [26] , denoted by 𝐵(•), as the grounding evaluator of the emotional connection. For sample (𝑣 𝑖 , 𝑡 𝑖 ), we first transform its image 𝑣 𝑖 into the top-3 adjective-noun pairs 𝐷 (𝑣 𝑖 ). Then, we determine whether it is emotional-matched based on a calculation similar to the above:EQUATIONFollowing this, we divide the dataset into four partitions: 1. strong-coupled samples that are factual-matched and emotionalmatched; 2. partial-coupled samples that are emotional-matched but factual-mismatched; 3. partial-coupled samples that are factual-matched but emotional-mismatched; 4. weak-coupled samples that are factual-mismatched and emotional-mismatched. We My dog turns 16 years old today! Obviously, she's proud of her age.Here's a scene that makes me smile. It's a small window to my world.My bear line art which I did today. I'm pretty happy with him.Oh, the lies you told, across the wire, but I still really feel it, don't know why… display a sample from each partition in Fig. 3 for intuitive understanding. It should be noted that we employ CLIP, DeepSentiBank, and BERTweet to simulate the partition conducted by humans, which can be replaced with other off-the-shelf tools.To optimize the construction of sample pairs in the subsequent contrastive learning, we utilize the knowledge embedded in the pre-trained visual and textual models to explore the unimodal intersample relationships from both factual and emotional perspectives. We adopt the network architecture of CLIP, as shown in Fig. 4 . In this stage, none of the gradients are involved in the calculation.For the visual encoder, we initialize it with parameters from a pre-trained visual model, denoted by 𝑉 𝜃 (•), which serves as the enhanced target of PACL. Leveraging its factual-level knowledge, we encode factual features for all images, clusters, where 𝐾 is a hyperparameter describing the cluster granularity, and each image is assigned a cluster label. We denote the label of image 𝑣 𝑖 as its factual pseudo-label 𝑓 𝑖 . Thus, we obtain the factual relationship between images 𝑣 𝑖 , 𝑣 𝑗 : they are factually similar if 𝑓 𝑖 = 𝑓 𝑗 , and dissimilar if 𝑓 𝑖 ≠ 𝑓 𝑗 .{𝑣 𝑖 |𝑖 = 1, 2, • • • , 𝑁 },For the textual encoder, we initialize it with parameters from SKEP [36] , denoted by 𝑇 𝜉 (•). SKEP is an unsupervised sentiment knowledge-enhanced pre-trained model focusing more on emotional features. Leveraging this characteristic, we encode emotional features for all texts, {𝑡 𝑖 |𝑖 = 1, 2, • • • , 𝑁 }, and perform k-means clustering that also yields 𝐾 clusters. Similarly, we denote the cluster label of text 𝑡 𝑖 as its emotional pseudo-label 𝑒 𝑖 .With the outcomes of the two preparation stages, we devise an adaptive contrastive learning that constructs positive and negative pairs under different strategies for each dataset partition. In this stage, we treat SKEP as the knowledge source and keep the parameters of the textual encoder frozen. We update the parameters of the visual encoder through contrastive learning to guide it in learning the generalizable factual-level and emotional-level knowledge from textual modality.Preliminary. We employ contrastive learning between images and texts multiple times. Denoting the features of (𝑣 𝑖 , 𝑡 𝑖 ) as 𝑧 𝑣 𝑖 and 𝑧 𝑡 𝑖 , here we provide a general form of contrastive loss:EQUATIONEQUATION𝑝 + (•), 𝑝 -(•) represent the positive and negative samples of the anchor. 𝜏 is the temperature hyperparameter. This loss comprises two components, with either image 𝑣 𝑖 or text 𝑡 𝑖 serving as the anchor. Taking the example of image 𝑣 𝑖 , the objective of Eq. ( 3) is to bring it closer to 𝑝 + (𝑣 𝑖 ) while pushing it away from 𝑝 -(𝑣 𝑖 ).Strong-coupled Samples. We first conduct contrastive learning solely on this partition to facilitate the alignment between the feature spaces of visual encoder 𝑉 𝜃 (•) and textual encoder 𝑇 𝜉 (•).Since samples are both factual-matched and emotional-matched, we treat the images and texts from the same samples as positive pairs, and those from different samples as negative pairs. It enables the model to assign higher cosine similarity scores to image-text pairs with strong factual and emotional connections. Denoting the projector of the visual branch as 𝑞 𝑣 (•) and it of the textual branch as 𝑞 𝑡 (•), the features of image 𝑣 𝑖 and text 𝑡 𝑖 are computed as:EQUATIONBy substituting Eq. ( 4) into Eq. ( 3), we obtain the contrastive loss of this dataset partition:L 1 = L (𝑝 + (𝑣 𝑖 ) = 𝑝 + (𝑡 𝑖 ) = {𝑖}, 𝑝 -(𝑣 𝑖 ) = 𝑝 -(𝑡 𝑖 ) = { 𝑗 | 𝑗 ≠ 𝑖}).(5)Partial-coupled Samples. These samples are from two dataset partitions. We discuss them together due to their common properties, that their images and texts have strong connections only in one of the factual and emotional aspects. To leverage these connections, we construct positive pairs in the same way as strongcoupled samples. However, since partial-coupled samples no longer guarantee the advantages of positive pairs over negative pairs in both factual and emotional aspects, the former construction of negative pairs is sub-optimal.For the partition where samples are emotional-matched but factual-mismatched, we take (𝑣 𝑖 , 𝑡 𝑖 ) as an example. Compared to 𝑣 𝑖 , the image 𝑣 𝑗 from another sample may demonstrate a stronger factual connection with 𝑡 𝑖 , and in this case, treating (𝑣 𝑖 , 𝑡 𝑖 ) as a positive pair while (𝑣 𝑗 , 𝑡 𝑖 ) as a negative pair compromises the factual-level learning of the visual model. Therefore, we devise a mapping 𝑚 𝑓 (•) that searches for images that might possess strong factual connections with text 𝑡 𝑖 to avoid inappropriate negative pairs:EQUATIONIt first finds the most matching image, denoted by 𝑣 𝑙 , based on the cosine similarity scores and then selects images with factual pseudo-label same as 𝑓 𝑙 . We also define an inversion of 𝑚 𝑓 (•) that searches for texts based on image 𝑣 𝑖 :EQUATIONSubsequently, the negative samples of 𝑣 𝑖 are all other texts with the exclusion of 𝑚 -1 𝑓 (𝑣 𝑖 ), and the negative samples of 𝑡 𝑖 are all other images with the exclusion of 𝑚 𝑓 (𝑡 𝑖 ). Thereby, the contrastive loss of this partition is:EQUATION)For the partition where samples are factual-matched but emotionmismatched, we devise a symmetric 𝑚 𝑒 (•) that searches for texts that possess strong emotional connections with image 𝑣 𝑖 :EQUATIONand also an inversion of 𝑚 𝑒 (•) that searches for images based on text 𝑡 𝑖 :EQUATIONSimilarly, the negative samples of 𝑣 𝑖 (𝑡 𝑖 ) are all other texts (images) with the exclusion of 𝑚 𝑒 (𝑣 𝑖 ) (𝑚 -1 𝑒 (𝑡 𝑖 )). The contrastive loss of this partition is:EQUATION3.3.4 Weak-coupled Samples. Since samples of this partition are factual-mismatched and emotional-mismatched, constructing positive pairs as before would disrupt the semantic relationships learned by the model [19] and lead to degenerations in downstream performance. Therefore, we reconstruct positive pairs by leveraging the learned correlation between images and texts in previous partitions. Taking sample (𝑣 𝑖 , 𝑡 𝑖 ) as an example, we select positive samples according to the cosine similarity scores:EQUATIONEQUATION)For negative pairs, we follow the strategies adopted for partialcoupled samples. We simultaneously apply 𝑚 𝑓 (•) and 𝑚 𝑒 (•) to filter out potential false negative pairs with either strong factual or emotional connections. As a result, the contrastive loss is:EQUATION3.3.5 Total Loss. We devise a progressive learning schedule to guide the model to learn from strong-coupled samples to weakcoupled samples step by step. In the first one-third of epochs, we only incorporate the strong-coupled samples into training:EQUATIONIn the second one-third of epochs, we incorporate both the strongcoupled and partial-coupled samples into training:EQUATIONIn the last one-third of epochs, we incorporate all samples into training:EQUATION 
experiment 4.1 datasets: 
 We conduct training of PACL on TumEmo [45] . It contains 200k image-text pairs crawled from social media. Its samples are annotated based on their emotions. Since we emphasize leveraging the factual and emotional connections between images and texts, we avoid using knowledge from the annotations.In the downstream testing, we adopt six publicly available VER datasets that are annotated according to diverse emotion theories. FI [53] is labeled according to Mikels' psychological study [23] . Emotion-6 and UnbiasedEmo [24] contain 6 common emotion categories. WebEmo [24] is labeled according to Parrot's hierarchical emotion model [27] , which contains 2 basic categories at the first level, 7 categories at the second level and 25 fine-grained categories at the third level. Emotic [16] comprises 26 fine-grained emotion categories, with each image assigned multiple emotion labels, and 10-scale VAD (Valence-Arousal-Dominance) [32] ratings. IAPS [18] is labeled by 9-scale VAD ratings.All of our implementations are based on Pytorch. We employ three visual models pre-trained on ImageNet [4] , including ResNet-50 [13] , ViT-base [8] , Swin-base [21] , and one visual model pre-trained by CLIP [28] (ViT-clip) as the target models for enhancement. Most of our experiments adopt SKEP [36] as the pre-trained textual model. In Section 5, we also evaluate the enhancement effect of other textual models [6, 20, 31] . In the first stage of PACL, we set the partition threshold 𝜎 to 0.7. In the second stage, we set cluster granularity 𝐾 to 2. In the third stage, we set the contrastive temperature 𝜏 to 0.07, the initial learning rate of ResNet-50 to 1e-3, ViT-base and ViT-clip to 1e-5, Swin-base to 2e-5, projection layers 𝑝 𝑣 (•), 𝑝 𝑡 (•) to 1e-3. We train the model for 30 epochs with a minibatch size of 64. We adopt AdamW optimizer with a cosine schedule to decay the learning rate. In the downstream testing, we adopt the original partition for training, validation, and testing if provided. Otherwise, we randomly divide the dataset in a ratio of 8:1:1. All the experiments are conducted on four NVIDIA 4090 GPUs.Two other works also focus on enhancing pre-trained visual models in perceiving emotions. Feng et al. [10] factorize human perception into three steps and devise a pre-training method to imitate it. Their pre-training tasks include colorization, super resolution, scene recognition, jigsaw puzzles, ANP prediction, and image captioning, requiring 10.5M images with annotations. We reproduce it under all of our settings for comparison. EmotionCLIP [58] adopts visionlanguage pre-training to extract emotional representations from verbal and nonverbal communication. It leverages the pre-trained knowledge from CLIP [28] and DistilBert [31] . It is trained on 1M video-text pairs with the annotation of sentiment score and human bounding boxes. Since the training data is not publicly available yet, we compare PACL with it under pre-trained ViT-clip, which is the only model released. PACL is our proposed model, it leverages the pre-trained knowledge from SKEP [36] and is trained on 200K image-text pairs without any additional annotations. PACL has the loosest requirements for training data among them.We adopt four kinds of VER downstream tasks. 1. In single-label learning, models are required to classify each image into a single emotional category. 2. In hierarchical-label learning, models are required to predict categories for three emotion levels simultaneously. We report accuracy (Acc) and weighted-F1 (F1) for the first two tasks. 3. In multi-label learning, models are required to predict multiple emotion categories for each image. We report the mean of average precision (mAP) and area under roc curve (AUC) of categories. 4. In VAD learning, models are required to predict Table 1 : Emotional enhancement of visual models on single-label learning datasets including FI [53] , Emotion-6 [24] , Unbi-asedEmo [24] , hierarchical-label learning dataset WebEmo [24] , multi-label learning dataset Emotic [16] , and VAD learning datasets including Emotic [16] and IAPS [18] . Accuracy (Acc↑), weighted-F1 (F1↑), mean of average precision (mAP↑), area under roc curve (AUC↑), mean of mean square error (MSE↓), mean of R square (𝑅 2 ↑) are reported for these datasets. Acc and F1 are followed by the number of total categories, and MSE is multiplied by 10 2 . The best results are marked in bold. the continuous values in the 'Valence', 'Arousal', and 'Dominance' dimensions. We report the mean of mean squared error (MSE) and R squared (𝑅 2 ) of the three dimensions. Besides, we conduct experiments under two evaluation protocols: linear evaluation and fine-tuning. In both protocols, we set the learning rate of the linear classification head to 1e-2 and train for 40 epochs. For parameters of the backbone network, we keep them frozen under linear evaluation and set the initial learning rate of ResNet-50 to 1e-4, ViT-base, Swin-base, and ViT-clip to 1e-5 under fine-tuning. The other settings of downstream testing are aligned with Section 4.2.The experimental results are presented in Table 1 . We can observe that PACL consistently improves all the pre-trained visual models on all tasks by a large margin. In contrast, the effectiveness of Feng et al. is limited under linear evaluation, especially in hierarchical-label learning. It highlights the advantages of harnessing textual knowledge in perceiving emotions at multiple levels. Additionally, since they start the pre-training from scratch, they can not leverage the generalizable factual knowledge in the pre-trained visual models. For instance, PACL achieves significant performance gains by starting from ViT-clip instead of ViT-base, underscoring the effectiveness and scalability of bridging the "affective gap". Emo-tionCLIP, on the other hand, also undergoes poor performance on some tasks, specifically, multi-label learning. We attribute this to the misalignment between its pre-training and testing. Although it explicitly models the dependencies between context and humans, it may be misled by the massive communications between humans in training data and learns emotional representations heavily dependent on humans. However, the multi-label learning dataset Emotic emphasizes the emotion perception from both context and humans, resulting in the undesirable performance of EmotionCLIP. On the contrary, PACL leverages diverse image-text pairs and the unified factual and emotional features embedded in texts, naturally encouraging visual models to perceive emotions in different regions and scales, thereby promoting universal enhancements on all tasks.To further demonstrate the necessity and effectiveness of bridging the "affective gap", we present a brief view of the current SOTA methods in two downstream tasks. 4.5.1 Classification on FI, WebEmo. Following previous methods [40] , we group eight categories of FI into two coarse-grained parent categories. We report the classification accuracy of SOTA methods on FI and WebEmo at multiple emotion levels in Table 2 . It should be noted that the models are tested on each level separately, different from the settings in hierarchical-label learning in 4.4. As shown in Table 2 , ViT-clip enhanced by PACL outperforms other Table 2 : Accuracy of SOTA methods in classification at different emotion levels on FI [53] and WebEmo [24] . The best results are marked in bold, and the second-best results are underlined.Model FI WebEmo Acc-2 Acc-8 Acc-2 Acc-6 Acc-25Sentibank [1] 56.47 44.49 ---DeepSentibank [2] 64.39 53.16 ---PCNN [52] 75. 34 methods in most cases. Additionally, since our enhanced model does not incorporate additional parameters besides the backbone network, it can be combined with other methods for further improvements. By adding a polarity-consistent regression loss [62] , our enhanced model achieves the best performances across all scenarios. These results validate the efficacy and scalability of PACL. 4.5.2 Zero-shot on FI, Emotion-6. Since we adopt the network architecture of PACL the same as CLIP [28] , our enhanced models are naturally capable of conducting zero-shot emotion classification by leveraging the learned emotional connections between images and texts. We report the zero-shot classification accuracy of the enhanced ResNet-50 and the SOTA methods in Table 3 . Similar to CLIP, our enhanced model is also sensitive to text prompts. Therefore, we test its performances under both regular and manually designed prompts. From the results, we observe that the enhanced ResNet-50 significantly outperforms CLIP by learning from much fewer image-text pairs. We attribute this to the difference between the types of training data. PACL learns from social media posts, which contain generally strong emotional connections between images and texts, as well as more frequent occurrences of textual emotional expressions. Compared with other methods, the enhanced ResNet-50 achieves the second-best performance on both datasets. Although zero-shot emotion classification is not our primary goal, the satisfactory performance achieved by the enhanced ResNet-50 provides us with the intuitive advantages of learning from social media posts. 
analysis: 
 In this section, we validate the effectiveness of each component of PACL. All the experiments are conducted by adopting pre-trained ResNet-50 as the backbone under linear evaluation.We conduct ablation experiments in Table 4 to explore the influences of different dataset partitions and sample usages. Firstly, we do not apply partition strategies and conduct contrastive learning by treating all the samples as strong-coupled. It results in minor enhancements for the pre-trained model since TumEmo contains certain noisy samples with either weak factual or emotional connections. In these cases, the vanilla contrastive learning is sub-optimal. Next, we apply partition based only on factual connections, treating factual-matched samples as strong-coupled and factual-mismatched samples as partial-coupled. PACL enhances the pre-trained model by a large margin with solely factual-matched samples and further improves its performance by utilizing factual-mismatched samples. We also apply partition based only on emotional connections with the same settings, which brings a similar trend. By dynamically constructing contrastive pairs, we succeed in harnessing these noisy samples. Notably, although the downstream task is emotion-related, applying partition based on either emotional or factual connections can both lead to significant enhancements, demonstrating the cruciality of proper handling of both factual-mismatched and emotional-mismatched samples. Finally, by simultaneously applying factual and emotional partition as PACL, the pre-trained model progressively learns from factual and emotional connections of different kinds of data, achieving the best result. The above results indicate the effectiveness of PACL in fully exploiting the dataset. To probe the influence of hyperparameters, we conduct the experiments in Table 5 .Partition Threshold 𝜎. It controls the precision and size of each partition. By adopting a small threshold, the strong-coupled samples can not guarantee the generally strong factual and emotional connections between images and texts. It disrupts the alignment between the visual and textual feature spaces [19] . On the other hand, a large threshold limits the number of strong-coupled samples, which is also not conducive to the alignment. Consequently, we adopt a medium threshold that provides sufficient high-quality strong-coupled samples. Specifically, 22% samples are strong-coupled, 51% samples are partial-coupled, and 27% samples are weak-coupled under 𝜎 = 0.7.Cluster Granularity 𝐾. It balances the distances between intra-cluster and inter-cluster samples, thereby influencing the reconstruction of negative pairs for partial-coupled and weak-coupled samples. Fine-grained clustering guarantees the correctness of filtered negative pairs, while coarse-grained clustering emphasizes the correctness of the remaining. According to the experimental results, we adopt a small 𝐾. This indicates that the advantages of filtering out additional true negative pairs outweigh the potential drawbacks of ignoring false negative pairs. Interestingly, the enhanced model achieves the best results at 𝐾 = 6 on Emotion-6. We attribute this to the alignment in the granularities of clustering and downstream classification. However, we still adopt 𝐾 = 2 due to its general advantages across all datasets. [24] and UnbiasedEmo [24] under text supervisions from different pre-trained textual models. Specifically, we employ SKEP [36] , DistilBert [31] , Bert-base [6] , RoBerta-base [20] , Bert-large [6] and ReBerta-large [20] .Contrastive Temperature 𝜏. It controls the strength of penalties on negative samples [38] . As shown in the results, PACL is not sensitive to its selection. The enhanced model achieves slightly superior results under smaller temperatures compared to larger selections. Therefore, we adopt 𝜏 = 0.07 according to results and related works [12, 28] .We evaluate the generalizability of PACL by employing the knowledge from various pre-trained textual models to enhance the visual model. As shown in Fig. 5 , different pre-trained textual models bring consistent performance improvements on both Emotion-6 and Un-biasedEmo. It validates the universal advantages of textual models in encoding emotional-level features compared to visual models. Additionally, we can observe a positive correlation between the representative capability of the textual model and its enhancement effectiveness through PACL on the visual model. It demonstrates the scalability of PACL relative to the textual models. With the evolution of large language models in recent years, the potential of PACL can be further exploited. 
conclusion: 
 In this paper, we focus on bridging the visual "affective gap" of the pre-trained visual models. It arises from the misalignment between the objectives of pre-training and downstream tasks. Inspired by the advantages of language, we propose Partitioned Adaptive Contrastive Learning (PACL) that enhances visual models by leveraging the unified factual-level and emotional-level features from the pre-trained textual model. PACL learns from the factual and emotional connections from noisy image-text pairs collected from social media by dataset partition, unimodal sample cluster, and adaptive contrastive learning. Through extensive experiments, we demonstrate the effectiveness and scalability of PACL in enhancing the emotional perception of visual models, validating the importance of bridging the "affective gap".  
: 
 Figure 1 : Illustration of the "affective gap" in the pre-trained visual model and how our proposed method bridges it using knowledge from the pre-trained textual model. cues is progressively emerging as a pivotal challenge on the path toward the next generation of artificial general intelligence [22, 50] . Therefore, it has drawn increasing attention in recent years [9, 24] and exhibited broad applications in opinion mining [37] , business intelligence [14] , and autonomous driving [17] .Prior studies in VER have greatly enhanced their performance by leveraging the powerful generalizable knowledge of pre-trained models [61] . However, a misalignment exists between the objectives of their pre-training and downstream tasks, leading to the "affective gap" phenomenon, as depicted in Fig. 1 . Specifically, the commonly adopted pre-training tasks, such as classification on ImageNet [4] and CLIP [28] -type vision-language pretraining, encourage models to acquire non-trivial factual-level features under semantic guidance. Yet, these features lack a direct association with the emotional categories, leading to the incomplete applicability of pre-training knowledge for downstream VER tasks. To bridge the "affective gap", the pre-trained model needs the capability of encoding emotional-level features, which serves as an intermediate link between factual-level features and emotion categories.An intuitive way to acquire such capability is learning from large-scale emotional annotated datasets, while it would encounter two primary challenges in practice. Firstly, the perception subjectivity of emotion [25, 63] results in high annotation costs, restricting the scale of high-quality datasets. Secondly, the sparse information density of visual data [3, 39] causes a dispersed distribution of emotional information, requiring models to extract features across multiple regions and scales, thereby increasing the demand for computational resources. However, language has natural advantages in these two aspects compared with vision, as it explicitly conveys emotions through specific words and possesses higher and more consistent information density [34] . These advantages allow pre-trained textual models to simultaneously learn generalizable factual-level and emotional-level features from large corpora in unsupervised manners, eliminating the "affective gap" in textual modality.Inspired by this discovery, we propose to bridge the visual "affective gap" by enhancing pre-trained visual models with the unified factual-level and emotional-level features from pre-trained textual models, as illustrated in Fig. 1 . To transfer this knowledge from texts to images, we focus on image-text datasets where samples generally possess relatively strong factual and emotional connections between their images and texts. Specifically, we adopt TumEmo [45] , an image-text dataset sourced from social media. Due to the inherent factual inconsistencies and emotional discrepancies of usergenerated posts [46] , the dataset inevitably incorporates certain noisy samples that are factual-mismatched, emotional-mismatched samples, or both. Given such characteristics, we propose Partitioned Adaptive Contrastive Learning (PACL) to leverage these connections within all samples. It comprises three stages. Firstly, we quantitatively assess the factual and emotional connections between the image and text of each sample, dividing the dataset into separate partitions. It allows us to formulate distinct learning strategies for different types of data. Secondly, we cluster on two unimodal subsets of the dataset, revealing the inter-sample relationships from both factual and emotional aspects. Thirdly, we design an adaptive contrastive strategy by utilizing the outcomes from the preceding stages. We dynamically filter out false negative pairs and reconstruct false positive pairs for different dataset partitions, thereby ensuring the consistent superiority of positive pairs over negative pairs, both factually and emotionally.We follow the network architecture of CLIP [28] , with its textual encoder initialized by SKEP [36] and its visual encoder initialized by the to-be-enhanced visual model. In this manner, the enhanced visual model can encode emotional-level features from an image while retaining the original perception of factual-level features. It effectively bridges the "affective gap" in visual models and significantly improves their performance in downstream VER tasks.The overall contributions of this paper are three-fold:• We harness the advantages of language over vision. As far as we know, this is the first attempt to bridge the "affective gap" in pre-trained visual models with pre-trained textual knowledge. • We propose a method called PACL. It leverages the factual and emotional connections within noisy image-text pairs by dynamically constructing contrastive pairs for different types of samples. • We conduct extensive experiments on six VER benchmarks and four kinds of downstream tasks with four pre-trained visual models. The results demonstrate the consistent improvements achieved by our method, underscoring the effectiveness and necessity of bridging the "affective gap". 
 
 Appendix:  


Decision: We would like to thank the authors for answering the questions by the reviewers in the rebuttal. Reviewers have considered your statements and most reviewers felt that their questions/concerns have been addressed properly. As such, we are happy to accept the paper.

Reviews: review: paper_summary: paper_summary: proposed Partitioned Adaptive Contrastive Learning (PACL) for Visual emotion recognition.
strength_weakness: strength_weakness: managed to separate different types of samples and devised distinct contrastive learning strategies for each type.
By dynamically constructing negative and positive pairs, exploited the potential of noisy samples.
limitations: limitations: 1. Analyzing computational complexity.
2. Provide reproduction code.
3. Where there are correlations between emotions, visualize the distribution of emotions and validate the model's ability to discriminate emotions.
paper_summary: paper_summary: This paper presents Partitioned Adaptive Comparison Learning (PACL), which bridges the visual “affective gap” of pre-trained visual models by augmenting them with unified factual-level and emotional-level features from the pre-trained textual model. PACL learns from the factual and emotional connections from noisy image-text pairs collected from social media by dataset partition, unimodal sample cluster, and adaptive contrastive learning.
strength_weakness: strength_weakness: 1.	This paper is well-written and easy to follow.
2.	Experiments are also solid, providing extensive comparisons as well as ablation studies.
limitations: limitations: 1.	Regarding the proposed progressive learning methodology that divides the entire training process equally into three parts, is there a basis for such a division? Has a more optimal training approach been explored?
2.	The clustering method in Stage 2 uses consistent clustering granularity for factual and emotional, and is puzzled as to whether the granularity must be aligned between the two. It is recommended to add more comparative experiments to verify the authors' settings in the article
3.	It would be helpful if the papers could elaborate on limitations suggesting future directions and potential avenues for future research.
4.	While the paper is generally well-written, it would greatly benefit from a careful proofreading to address the presence of some typographical errors in the manuscript, e.g. the lack of a period after Equation (10).
paper_summary: paper_summary: This paper identifies the affective gap as the lack of direct association between factual-level features and emotional categories in visual models. The authors propose a contrastive learning-based method that uses noisy image-text pairs to create dynamic positive and negative pairs for contrastive learning. This approach separates different types of samples and applies distinct strategies to each, fully exploiting the potential of noisy data. Experiments are conducted following [10].
strength_weakness: strength_weakness: 1.	The idea of introducing image-text data is interesting and will benefit the community.
2.	Extensive experiments demonstrate the effectiveness of proposed method.
3.	Ablation studies verify that each component is an optimal solution.
limitations: limitations: 1.	Affective gap is a long-standing problem in visual emotion recognition, and any progress in this area would be highly valuable. The authors have some efforts to introduce textual knowledge to narrow this gap. However, the paper lacks a clear and precise definition of the "affective gap," making it challenging to assess the extent to which it has been bridged. It would significantly strengthen the paper to provide both quantitative and qualitative metrics (rather than a single sample of image-text pair) that can be used to evaluate the gap.
2.	The results of the basic backbone are somewhat unusual. If the reviewer correctly understands the experimental setting, the accuracy of the fine-tuned ResNet50 on the FI dataset (65.14%) is lower than those reported in other works (67.00% from [10] and 67.53% from [A, B]). This discrepancy suggests that the proposed PACL method offers only a limited performance improvement (+0.11%). Additionally, the performance of [10] reported in this work is inconsistent with the results from the original paper. Could the authors provide more detailed information about their implementation? Clarifying these details would help in understanding the experimental conditions and ensuring the reproducibility of the results. This additional information is crucial for accurately assessing the effectiveness of the proposed method.
3.	The paper lacks some critical experimental details, and no code has been provided, which raises concerns about the reproducibility of the results.

[A] EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes, ICCV23

[B] Stimuli-Aware Visual Emotion Analysis, TIP21

Although I appreciate the interesting idea of this work, but there are issues about some of inconsistent performance that can not align with previous works and some bold statement that is not verified by evidence. This make the reviewer leans to reject.
paper_summary: paper_summary: This paper proposes a Partitioned Adaptive Contrastive Learning (PACL) method to leverage the factual and emotional connections between images and texts in noisy social media data. By dynamically constructing negative and positive pairs, the model fully exploits the potential of noisy samples. Experiments on six VER benchmarks and four kinds of downstream tasks with four pre-trained visual models demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The studied topic, i.e., bridging the “affective gap” in pre-trained visual models with pre-trained textual knowledge, is interesting.
2. The experiments are convincing. The ablation studies on different components are well conducted to demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. Figures 2 and 3 require a clear indication of the dotted and solid lines.
2. More explanation is needed on how the combination of "ViT-clip + PACL + PDANet" in Table 2 implements and contributes to performance improvements.
3. Why the proposed ResNet-50 + PACL underperforms against Ye et al. [49] in zero-shot classification, as shown in Table 3?