# Image Sentiment Analysis via Active Sample Refinement and Cluster Correlation Mining

Authors: Hongbin Zhang; Haowei Shi; Jingyi Hou; Qipeng Xiong; Donghong Ji
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Training an effective image sentiment analysis model using high-quality samples and the implicit cross-modal semantics among heterogeneous features is still challenging. To address this problem, we propose an active sample refinement (ASR) strategy to obtain sufficient high-quality images with definite sentiment semantics. We mine the cluster correlation among the heterogeneous SENet features. Discriminative cross-modal semantics is generated to train an effective but robust image classifier. Ensemble learning is employed to further boost performance. Our method outperforms other competitive baselines, demonstrating its effectiveness and robustness. Meanwhile, the ASR strategy is a useful supplement to the current data augmentation method.


---

## Body

## 1. Introduction

In early research studies, text-based sentiment analysis [1–4] has been proposed to predict the implicit sentiment of texts. These methods have demonstrated evident progress in sentiment analysis. However, visual information accounts for about 80% of all perception in the human brain. With the rapid development of social media (e.g., WeChat, Twitter, and microblog), an increasing number of people would like to use all sorts of photographs to express their private emotions. Hence, besides texts, image (or any kind of visual information) is another important complementary way to characterize human emotions. Accurate assessment of the implicit emotions in images, called image sentiment analysis, has become an active and hot research topic in the field of computer vision (CV). More attractively, this research topic has many practical applications, including in education, entertainment, psychotherapy, and advertisements [5, 6]. As shown in Figure 1, we can make real-time fashion tendency prediction (left) or public opinion monitoring (right) based on the results of image sentiment analysis. In particular, if we want to know whether a commodity is popular, we typically look at how much the item has been purchased. Meanwhile, the proportion of positive comments is another important factor. As we know, there are many images uploaded by users in these comments. Accurate analysis of these images can evaluate the positive rating and then make fashion trendency prediction. For public opinion monitoring, we can make the real-time monitoring of the images transferred on the social platforms. If some images are detected to be offensive or negative, then we need to focus on this user and take the necessary processing step. Therefore, learning the visual sentimental information has the following advantages. (1) It helps explain human emotions in terms of the visual signal rather than textual signal. (2) It helps produce more personalized predictions that are consistent with human’s real preferences. Owing to these advantages, its interpretability, and numerous practical applications, image sentiment analysis has attracted more and more attention from academia and industry.Figure 1
Practical applications of image sentiment analysis.However, owing to the subjective differences among different annotators, numerous semantically ambiguous images remain in the existing benchmark datasets. As we know, image sentiment annotation is more difficult than other traditional CV tasks because the emotions hidden in images are not independent in the aspect of semantics. Contrarily, those concrete objects (e.g., person, cat, dog, bike, and bird) are independent [7]. Moreover, few studies have attempted to exploit the implicit correlation among the heterogeneous features from a homologous neural network. To address these problems, we propose a simple yet effective model for image sentiment analysis, which can not only adaptively augment the original public datasets but also make full use of the implicit complementary information among the heterogeneous SENet features. All these ideas can boost the final classification performance and strengthen the practicality of our model.Conceptually and empirically, the main contributions of this study are summarized as follows:(1)
We proposed an active sample refinement (ASR) strategy that can adaptively “generate” (not really generating but mining) high-quality images with definite sentiment semantics, which builds a firm data foundation for model training. More importantly, the ASR strategy is a useful supplement to the current data augmentation method. It can be seamlessly incorporated into the state-of-the-art baselines to achieve better performance.(2)
We created a set of more discriminant but robust features by fully mining the implicit cluster correlation among the heterogeneous SENet features. These features can more accurately and comprehensively characterize the key sentimental semantics in images.(3)
We conducted extensive experiments using two benchmark datasets. The results demonstrate the superior performance of our model over other state-of-the-art baselines. The code for our method is available athttps://github.com/Danicaghost/SHW.git.


## 2. Related Works

Image sentiment analysis is still a challenging CV task, which makes its automated classification trickier [8, 9]. Initially, many hand-crafted image features that focus on low-level visual information, including color, texture, and shape, were used to perform image sentiment analysis. For instance, Machajdik and Hanbury [10] used a set of color and textural features to represent the visual sentiment content in images. Lu et al. [11] investigated how the shape characteristics in images affect human emotions. Ko and Kim [12] utilized both color and scale-invariant feature transform (SIFT) features to train a pLSA-based model for image sentiment analysis. These hand-crafted features [10–12] usually require manual interventions and are insufficient to characterize the deep-level sentiment semantics in images. Moreover, any single image feature is hard to integrally describe an image. Hence, Zhang et al. [13] proposed a feature mid-fusion algorithm called gene selection XGBoost (GS-XGB) to mine the implicit correlation among a group of image features [14], which obtained satisfactory performance of image sentiment analysis. However, the feature mid-fusion algorithm requires numerous features and is intricate to realize. Furthermore, it separates the inherent dependencies between different classes. Unlike feature mid-fusion, Zhang et al. [14] employed discriminant correlation analysis (DCA) [15] to complete feature early-fusion and mine the correlation among a set of heterogeneous features. The corresponding ME2M model [14] is effective and efficient. But the feature dimension after the DCA operation is too low, which loses some important information. The above studies [10–14] are fully supervised, which need massive accurate sentiment annotations.Recently, convolutional neural networks (CNNs) have demonstrated the ability to learn more representative features. Some state-of-the-art methods employed CNN to complete image sentiment analysis. Das et al. [16] proposed a deep learning model including an attention mechanism for focusing local regions and determining the required sentiment. She et al. [17] proposed a model called weakly supervised coupled network (WSCNet) that aims to automatically select relevant regions to reduce the burden of sentiment annotation and improve performance. Zhu et al. [18] combined a weakly supervised learning strategy with a CNN model to complete end-to-end image sentiment prediction. To align image regions for gaining spatial invariance and learning strongly localized features, Durand et al. [19] designed a new method called weakly supervised learning of deep ConvNets for image classification, pointwise localization, and segmentation (WILDCAT). Evidently, these weakly supervised methods [16–19] are dedicated to easing the annotation burden. Similarly, Sun et al. [20] discovered the affective regions via a deep network. Rao et al. [21] designed a multilevel region-based framework to explore the sentiment response of local regions. In [22], a model called MldrNet, which learns multilevel deep representations, was proposed for image sentiment classification. Furthermore, Wu et al. [23] developed a multiattention model for jointly discovering and localizing multiple relevant local regions that gave predicted attributes. Although the sentiment annotation burden has been alleviated to some extent, the corresponding classification performance of most weakly supervised methods [17–23] is unsatisfactory. Unlike the above studies, Simonyan and Zisserman [24] proposed SmileyNet using a novel sentiment-aligned image embedding to leverage the intricate relationship between emojis and images in large-scale available social media data. However, the SmileyNet model leverages additional emoji information, which may be scarce in most public available datasets.Hence, how to make full use of the existing benchmark datasets (this helps alleviate sentiment annotation burden from another perspective) and reduce the impact of ambiguous annotations have become a hot concern. Recently, some researchers focused on utilizing state-of-the-art domain adaptation methods to address the ambiguous annotation problem caused by subjective differences. For example, Zhu et al. [25] translated each image from a source domain to a target domain in the absence of paired examples. Zhao et al. [26] proposed CycleEmotionGAN for image sentiment classification by adapting source domain images to have distributions like those of the target images by enforcing emotional semantic consistency. Lin et al. [27] proposed a multisource sentiment generative adversarial network (MSGAN), which uses a unified sentimental latent space to handle data from multiple domains. These methods [25–27] can make full use of the existing data resources. However, the GAN-based domain adaption methods [26, 27] are difficult to train and require additional computing resources. In contrast to these works [25–27], Zhang et al. [14] designed a static sample refinement method that is relatively easy to reproduce and obtain high-quality images. Valuable knowledge can be mined from the refined samples and adaptively merged into the classification model, which helps boost the final classification performance. Unlike those GAN-based methods, the goal of the refinement-based method is to fully refine the original samples. Additionally, the principle of the GAN-based and refinement-based methods has a large difference. The GAN-based methods train a network to minimize the generated image and real image. However, the refinement-based model usually employs the traditional machine learning method, and it refines samples based on the diverse predictions of different classifiers. Hence, our method belongs to the refinement-based method. Certainly, the refinement-based method can also be combined with the GAN-based method to further improve the image quality.Recently, some researchers advocated combining more than one modality to infer perceived sentiment. Compared to a single modality, multimodalities have richer semantic information. Mittal et al. [28] proposed a M3ER method, which combines the cues from multiple co-occurring modalities such as face, text, and speech. Lu et al. [29] borrowed the idea of bidirectional encoder representations from transformers (BERTs) and proposed a model for learning task-agnostic joint representations of image content and natural language, namely, short for vision-and-language BERT (ViLBERT). The ViLBERT model can process both visual and textual inputs in the separate streams that interact through co-attentional transformer layers. Yang et al. [30] proposed a multimodal sentiment analysis model based on the multiview attentional network (MVAN), which utilizes a memory network to obtain the deep semantic information of the image-text pair. These multimodal methods have some evident limitations. First, these multimodal datasets are hard to collect, especially for the audio modality. Second, how to fuse the hidden information between mutimodalities is still a big challenge. Hence, we perform our method on a single image modal.Summarily, the above methods have achieved evident progress in the field of image sentiment analysis. Early works [16–23] focused on the analysis of local regions in images rather than the whole image. Most of them require massive high-quality data to train a robust model. Nevertheless, the corresponding performance is unsatisfactory. Although some domain adaption methods [25–27] have been proposed to alleviate this problem, the scarcity of high-quality images with definite sentiment semantics becomes more and more evident. Hence, some state-of-the-art data augmentation methods, including DADA [31] and AutoAugment [32], have been proposed to generate more high-quality samples. However, they need massive computing resources. On the other hand, few studies have attempted to exploit the implicit correlation among the heterogeneous features from a homologous neural network, such as the state-of-the-art SENet modules [33]. Thus, unlike recent works, our model is an organic combination of the feature early fusion and ASR. To a certain extent, our model can alleviate the issue of the lack of high-quality images. Meanwhile, our model tries to fully mine the cluster correlation among the heterogeneous SENet features. Hence, our method does not generate any new samples and is easier to reproduce.


## 3. Method

An overview of the proposed model is shown in Figure2. First, we perform the ASR strategy to adaptively and gradually augment the original datasets. The existing high-quality images in a dataset (D5) are used to train classifiers, and then, the semantically ambiguous images are predicted by the well-trained classifier. If the predicted results are consistent with the corresponding annotations, the images are chosen to augment the original high-quality dataset. For the high-quality dataset Dfine generated by the ASR strategy, the heterogeneous SE-ResNet features are extracted for the subsequent correlation mining. Because these features are derived from a homologous neural network, we attempt to mine the implicit but effective cross-modal semantics (CMS) among the heterogeneous SENet features by performing cluster correlation mining (CCM). Subsequently, several classifiers are trained to predict the sentiment of the images. Finally, an ensemble learning strategy is proposed to further improve the final performance. Through ensemble learning, the predictions of different classifications are integrated together to get more objective results and further improve the final classification performance. Hence, in this section, we first present our ASR strategy. Next, we show how to perform CCM among the heterogeneous SENet features. Finally, the training procedure is described.Figure 2
The pipeline of our model. We use the fine-grained Flickr and Instagram (FI) dataset to illustrate our model. High-quality images are obtained through ASR. Then, we extract a set of heterogeneous SENet features and perform CCM among these features to obtain more discriminative CMS. A simple voting strategy is designed to boost the final performance.


### 3.1. Active Sample Refinement

As described earlier, image sentiment analysis requires massive high-quality images with definite sentiment semantics. Based on the concept of active learning [34], we propose the ASR strategy to augment the original dataset. Active learning is a type of semi-supervised learning that is usually used to annotate unlabeled samples and improve the performance of classifiers [35]. In this study, we adopt active learning to retrieve high-quality images from the original dataset. Unlike other GAN-based methods [26, 27], we will not generate any new images and only make full use of the original benchmark datasets. Hence, our method can mitigate the absence of images with definite sentiment semantics and lay a firm data foundation for model training. The proposed ASR strategy is divided into two progressive steps: coarse-grained sample refinement and fine-grained sample refinement. The technology flowchat of the ASR strategy is shown in Figure 3.Figure 3
The technology flowchart of the ASR strategy.As shown in Figure3, the ASR strategy consists of two stages: coarse-grained sample refinement and fine-grained sample refinement. The left part illustrates the coarse-grained refinement procedure, while the right part illustrates the fine-grained refinement procedure. First, the classifiers trained by high-quality images are utilized to predict the semantically ambiguous images, and the coarse-grained refinement is completed by comparing the predicted label with the ground truth. So we obtain the Dcoarse dataset. Then, the active learning idea is used for the subsequent fine-grained sample refinement. In this procedure, we use one classifier to predict the data in Dcoarse each time, and the refined images will be directly added into D5. We loop the previous step to realize the gradual screening of images based on active learning (please see the blue arrow). Finally, the final Dfine dataset is obtained by intersection or union of the images generated by all the classifiers.Step 1:Coarse-grained sample refinement. First, we obtain a data subset called D3-5 from the original dataset D3 (D3 represents that at least three Amazon Mechanical Turk (AMT) workers gave the same sentiment label to an image. Similarly, D5 represents that all five AMT workers gave the same sentiment label to an image. We remove the D5 data from the D3 dataset and obtain D3-5). Hence, D3-5 can be regarded as the pending data for ASR. Our ultimate purpose is to achieve high-quality images from D3-5 and use these samples to augment the original D5 dataset. Therefore, we consider D5 as the training set (because any sample in D5 is high-quality) and D3-5 as the testing set. Second, to promote real-time efficiency, similar to reference [14], we utilize the cross-modal information between SIFT and VGG19 to efficiently characterize the D3-5 and D5 datasets. Third, to avoid selective bias, we perform image sentiment analysis using a group of heterogeneous classifiers (nine classifiers for Twitter I and five classifiers for FI). Finally, if all the predictions are consistent with the real label, the corresponding testing sample in D3-5 will be merged into Dcoarse. Therefore, Dcoarse is a data subset obtained from D3-5. It contains lots of images with relatively definite sentiment semantics. This builds a solid foundation for the subsequent fine-grained sample refinement.Step 2:Fine-grained sample refinement. We attempt to refine further to obtain high-quality images with definite sentiment semantics based on the Dcoarse subset. Therefore, we consider D5 as the training set and Dcoarse as the testing set, and the active learning strategy is seamlessly incorporated into the fine-grained sample refinement procedure. The fine-grained sample refinement process requires multiple iterations to gradually absorb effective knowledge into our model. In iterations, each classifier predicts every sample in Dcoarse. Then, we use the ranked batch mode queries [36] to rank the sample. The rank score function is expressed as(1)score=∂1−Φx,D5+1−∂Ux,∂=D5D3−5+D5,where x denotes an image from Dcoarse, Ux is the uncertainty of the predictions for x, and Φ is the Euclidean function used to calculate the distance between two images. High-scoring samples are placed at the top of a list. We select the top three samples and compare the predicted label of each sample with its real label. The sample will be chosen if it is completely consistent. Thus, we augment the original training dataset by adding this new sample. Otherwise, the sample will be removed from the list. After multiple iterations, much more valuable knowledge from these high-quality samples is adaptively and gradually merged into the ASR algorithm, making it more efficient, robust, and capable of autonomous learning. To prevent selective bias, we employ a group of heterogeneous classifiers to complete this fine-grained sample refinement procedure (two classifiers for Twitter I and four classifiers for FI (please refer to Figure 5). Hence, each classifier outputs a set of candidate high-quality images from its own perspective, and then, we perform union processing on these sets to build Dfine, which is the final augmentation data of the original D5 dataset. Each sample in Dfine has definite sentiment semantics. Hence, the augmented data builds a firm foundation for model training. Unlike those GAN-based methods, we only make full use of the existing benchmark datasets. We do not generate any new samples. So, it is easier to reproduce the ASR strategy.


### 3.2. Cluster Correlation Mining

Robust but effective features help improve the final accuracy. However, a single image feature is insufficient for image sentiment analysis. Hence, we first extract a set of heterogeneous yet effective features from the SENet modules, namely, SE-ResNet50, SE-ResNet101, SE-ResNet152, SE-ResNeXT-50, and SE-ResNeXT101 (why do we make this choice? please refer to Table5). Then, we mine the implicit cluster correlation among these SENet features because they all derive from a homologous neural network. We believe that some valuable information hides in these heterogeneous features. Meanwhile, we try to mine the inherent dependencies between different classes. Finally, we use this clustering correlation to generate CMS for characterizing the core sentimental semantics.Letxic denote the i-th sample in the c-th class, and yic denotes the i-th label in the c-th class. X and Y denote two heterogeneous SENet features for training, respectively, whereas X∗ and Y∗ represent the corresponding features for testing, respectively. The cluster correlation between X and Y is calculated as follows:(2)ρ=maxw,vw′VXYvw′VXXwv′VYYv,where w and v are the two projection matrices of X and Y, respectively. The covariance matrices, namely, Vxy, Vxx, and Vyy, are defined as follows:(3)VXY=1N∑c=1C∑i=1Xc∑j=1Ycxicyjc′,VXX=1N∑c=1C∑i=1XcYcxicxic′,VYY=1N∑c=1C∑j=1YcXcyjcyjc′,where N=∑c=1CXcYc. We aim to maximize the correlation between the projections of X and Y on w and v. After we obtain w and v, we transform X, X∗, Y, and Y∗ into Xα, Xα∗, Yα, and Yα∗, respectively, as follows:(4)Xα=w′X,Xα∗=w′X∗Yα=v′Y,Yα∗=v′Y∗.Using the formula (4), we construct U and V, respectively, as follows:(5)U=XαXα∗,V=YαYα∗.Finally, we concatenateU with V and generate CMS among heterogeneous SENet features as follows. The implicit and valuable cluster correlation among the SENet features is obtained. The inherent dependencies between different classes are mined in turn.(6)F=UV=XαYαXα∗Yα∗.


### 3.3. Training Procedure

Based on the augmented data “D5 + Dfine,” we used four general classifiers, namely, logistic regression (LR), support-vector machine (SVM), adaptive boosting (AdaBoost), and extreme gradient boosting (XGBoost), to train the proposed model. Then, ensemble learning (hard voting or soft voting) is performed on the top three classification results. This helps further improve the final classification performance.


## 3.1. Active Sample Refinement

As described earlier, image sentiment analysis requires massive high-quality images with definite sentiment semantics. Based on the concept of active learning [34], we propose the ASR strategy to augment the original dataset. Active learning is a type of semi-supervised learning that is usually used to annotate unlabeled samples and improve the performance of classifiers [35]. In this study, we adopt active learning to retrieve high-quality images from the original dataset. Unlike other GAN-based methods [26, 27], we will not generate any new images and only make full use of the original benchmark datasets. Hence, our method can mitigate the absence of images with definite sentiment semantics and lay a firm data foundation for model training. The proposed ASR strategy is divided into two progressive steps: coarse-grained sample refinement and fine-grained sample refinement. The technology flowchat of the ASR strategy is shown in Figure 3.Figure 3
The technology flowchart of the ASR strategy.As shown in Figure3, the ASR strategy consists of two stages: coarse-grained sample refinement and fine-grained sample refinement. The left part illustrates the coarse-grained refinement procedure, while the right part illustrates the fine-grained refinement procedure. First, the classifiers trained by high-quality images are utilized to predict the semantically ambiguous images, and the coarse-grained refinement is completed by comparing the predicted label with the ground truth. So we obtain the Dcoarse dataset. Then, the active learning idea is used for the subsequent fine-grained sample refinement. In this procedure, we use one classifier to predict the data in Dcoarse each time, and the refined images will be directly added into D5. We loop the previous step to realize the gradual screening of images based on active learning (please see the blue arrow). Finally, the final Dfine dataset is obtained by intersection or union of the images generated by all the classifiers.Step 1:Coarse-grained sample refinement. First, we obtain a data subset called D3-5 from the original dataset D3 (D3 represents that at least three Amazon Mechanical Turk (AMT) workers gave the same sentiment label to an image. Similarly, D5 represents that all five AMT workers gave the same sentiment label to an image. We remove the D5 data from the D3 dataset and obtain D3-5). Hence, D3-5 can be regarded as the pending data for ASR. Our ultimate purpose is to achieve high-quality images from D3-5 and use these samples to augment the original D5 dataset. Therefore, we consider D5 as the training set (because any sample in D5 is high-quality) and D3-5 as the testing set. Second, to promote real-time efficiency, similar to reference [14], we utilize the cross-modal information between SIFT and VGG19 to efficiently characterize the D3-5 and D5 datasets. Third, to avoid selective bias, we perform image sentiment analysis using a group of heterogeneous classifiers (nine classifiers for Twitter I and five classifiers for FI). Finally, if all the predictions are consistent with the real label, the corresponding testing sample in D3-5 will be merged into Dcoarse. Therefore, Dcoarse is a data subset obtained from D3-5. It contains lots of images with relatively definite sentiment semantics. This builds a solid foundation for the subsequent fine-grained sample refinement.Step 2:Fine-grained sample refinement. We attempt to refine further to obtain high-quality images with definite sentiment semantics based on the Dcoarse subset. Therefore, we consider D5 as the training set and Dcoarse as the testing set, and the active learning strategy is seamlessly incorporated into the fine-grained sample refinement procedure. The fine-grained sample refinement process requires multiple iterations to gradually absorb effective knowledge into our model. In iterations, each classifier predicts every sample in Dcoarse. Then, we use the ranked batch mode queries [36] to rank the sample. The rank score function is expressed as(1)score=∂1−Φx,D5+1−∂Ux,∂=D5D3−5+D5,where x denotes an image from Dcoarse, Ux is the uncertainty of the predictions for x, and Φ is the Euclidean function used to calculate the distance between two images. High-scoring samples are placed at the top of a list. We select the top three samples and compare the predicted label of each sample with its real label. The sample will be chosen if it is completely consistent. Thus, we augment the original training dataset by adding this new sample. Otherwise, the sample will be removed from the list. After multiple iterations, much more valuable knowledge from these high-quality samples is adaptively and gradually merged into the ASR algorithm, making it more efficient, robust, and capable of autonomous learning. To prevent selective bias, we employ a group of heterogeneous classifiers to complete this fine-grained sample refinement procedure (two classifiers for Twitter I and four classifiers for FI (please refer to Figure 5). Hence, each classifier outputs a set of candidate high-quality images from its own perspective, and then, we perform union processing on these sets to build Dfine, which is the final augmentation data of the original D5 dataset. Each sample in Dfine has definite sentiment semantics. Hence, the augmented data builds a firm foundation for model training. Unlike those GAN-based methods, we only make full use of the existing benchmark datasets. We do not generate any new samples. So, it is easier to reproduce the ASR strategy.


## 3.2. Cluster Correlation Mining

Robust but effective features help improve the final accuracy. However, a single image feature is insufficient for image sentiment analysis. Hence, we first extract a set of heterogeneous yet effective features from the SENet modules, namely, SE-ResNet50, SE-ResNet101, SE-ResNet152, SE-ResNeXT-50, and SE-ResNeXT101 (why do we make this choice? please refer to Table5). Then, we mine the implicit cluster correlation among these SENet features because they all derive from a homologous neural network. We believe that some valuable information hides in these heterogeneous features. Meanwhile, we try to mine the inherent dependencies between different classes. Finally, we use this clustering correlation to generate CMS for characterizing the core sentimental semantics.Letxic denote the i-th sample in the c-th class, and yic denotes the i-th label in the c-th class. X and Y denote two heterogeneous SENet features for training, respectively, whereas X∗ and Y∗ represent the corresponding features for testing, respectively. The cluster correlation between X and Y is calculated as follows:(2)ρ=maxw,vw′VXYvw′VXXwv′VYYv,where w and v are the two projection matrices of X and Y, respectively. The covariance matrices, namely, Vxy, Vxx, and Vyy, are defined as follows:(3)VXY=1N∑c=1C∑i=1Xc∑j=1Ycxicyjc′,VXX=1N∑c=1C∑i=1XcYcxicxic′,VYY=1N∑c=1C∑j=1YcXcyjcyjc′,where N=∑c=1CXcYc. We aim to maximize the correlation between the projections of X and Y on w and v. After we obtain w and v, we transform X, X∗, Y, and Y∗ into Xα, Xα∗, Yα, and Yα∗, respectively, as follows:(4)Xα=w′X,Xα∗=w′X∗Yα=v′Y,Yα∗=v′Y∗.Using the formula (4), we construct U and V, respectively, as follows:(5)U=XαXα∗,V=YαYα∗.Finally, we concatenateU with V and generate CMS among heterogeneous SENet features as follows. The implicit and valuable cluster correlation among the SENet features is obtained. The inherent dependencies between different classes are mined in turn.(6)F=UV=XαYαXα∗Yα∗.


## 3.3. Training Procedure

Based on the augmented data “D5 + Dfine,” we used four general classifiers, namely, logistic regression (LR), support-vector machine (SVM), adaptive boosting (AdaBoost), and extreme gradient boosting (XGBoost), to train the proposed model. Then, ensemble learning (hard voting or soft voting) is performed on the top three classification results. This helps further improve the final classification performance.


## 4. Experiments and Discussion

In this section, we first compare our model against state-of-the-art methods on Twitter I [37] and FI [38]. The Twitter I dataset is collected from social websites and labeled with two sentiment polarity categories (positive and negative) by AMT participants. Like D5 or D3 (please refer to Section 3.1), D4 indicates that at least four AMT workers gave the same sentiment label to the same image. The Twitter I dataset consists of 1,269 images. It is a coarse-grained dataset. Unlike Twitter I, the FI dataset is collected by querying eight sentiment categories, namely, anger, amusement, awe, contentment, disgust, excitement, fear, and sadness, as keywords from social websites. It consists of 23,308 images, each with at least three agreements. The FI is a fine-grained dataset. Making accurate sentiment predictions on this fine-grained dataset is still a big challenge. In our experiments, each benchmark dataset is randomly divided into 70% for training and 30% for testing. The details of the two datasets are provided in Table 1. We also exhibit the number of refined images in this table.Table 1
The Twitter I and FI datasets with refined data.
DatasetD5D4D3DcoarseDfineTwitter I8821116126912274FI523812644215081214610Second, we evaluate the effectiveness and robustness of the proposed ASR and CCM strategies, respectively. We want to know whether the ASR strategy is a general but effective data augmentation method and the CMSs generated by the CCM strategy are effective for image sentiment classification.Third, we compare the computational efficiency against some baseline models. Finally, we show the corresponding details of parameter tuning.In this study, we use a relatively high-performance laptop with the following hardware configuration: Intel (R) Core (TM) i7-8550, MX150, and 8 GB RAM, to implement our experiments.


### 4.1. Comparisons with State-of-the-Arts

In this section, we compare our model with the following state-of-the-art methods:(1)
Classical fine-tuned deep learning models, namely, VGG16 [39], AlexNet [40], CAM-Res101 [41], and Res101 [42].(2)
State-of-the-art feature fusion methods, namely, DCA [15], canonical correlation analysis (CCA) [43], gradKCCA [44], GS-SVM [13], GS-LR [13], and GS-XGB [13].(3)
Recently proposed baseline methods for image sentiment analysis, namely, ResNet-MldrNet [24], SPN [18], WSCNet [17], WILDCAT [19], ME2M (M) [27], SR-w-DCA [27], CycleGAN [25], CycleEmotionGAN [26], MSGAN [27], Sun’s model [20], Rao’s model [21], and SmileyNet [24].We also created two variants of our model by using different correlation mining methods, namely, DCA [15] and CCA [43]. We wanted to demonstrate the generalization of the proposed idea. So, we used “OursCCA” and “OursDCA” to represent the two variants. Meanwhile, in order to demonstrate that Dfine contains more high-quality image samples than Dcoarse, we also perform our model on Dcoarse and compare the performance on Dcoarse with that on Dfine. Hence, “OursCCM (coarse)” means that we perform our model on Dcoarse. The corresponding experimental results are shown in Table 2. Herein, “D5” represents that we only used the D5 data to evaluate the classification model. “D5 + Dfine” means that we executed each model on the augmented dataset. Others only used D3 to evaluate each model. Hence, we trained all the methods under the same setting. Moreover, in the “Type” column, “D” means deep learning-based method. “F” means feature fusion method. “B” means state-of-the-art baseline. “O” means our proposed model.Table 2
Performance comparisons. The best value of each dataset is shown as89.90. Unit: %.
DatasetMethodTypeAccuracyMethodTypeAccuracyTwitter IVGG16D76.75WILDCAT (D5+Dfine)B73.85VGG16 (D5)D86.56WSCNetB84.25VGG16 (D5+Dfine)D82.69WSCNet (D5)B87.62AlexNetD73.24WSCNet (D5+Dfine)B89.40CAM-Res101D82.67MSGANB63.58GS-SVM (D5)F88.72CycleGANB61.59GS-LR (D5)F87.22CycleEmotionGANB62.38GS-XGB (D5)F86.47SmileyNetB89.16GS-SVM (D5+Dfine)F82.04Sun’s modelB88.91GS-LR (D5+Dfine)F84.51ME2M (M)B87.15GS-XGB (D5+Dfine)F83.10SPNB81.67CCA (D5)F80.08OursCCAO85.71gradKCCA (D5)F77.07OursCCA (coarse)O77.81DCA (D5)F87.59OursDCAO82.23WILDCATB79.53OursCCM (coarse)O89.07WILDCAT (D5)B71.43OursCCMO89.90FIRes101D66.16WILDCAT (D5+Dfine)B72.23Res101 (D5)D75.61WSCNetB70.07Res101 (D5+Dfine)D78.11WSCNet (D5)B72.22AlexNetD58.13WSCNet (D5+Dfine)B74.72CAM-Res101D68.54MSGANB70.63GS-SVM (D5)F73.52CycleGANB63.87GS-LR (D5)F73.59CycleEmotionGANB67.78GS-XGB (D5)F72.89MldrNetB67.75GS-SVM (D5+Dfine)F74.79Rao’s modelB75.46GS-LR (D5+Dfine)F75.30SR-w-DCAB75.72GS-XGB (D5+Dfine)F75.98SPNB66.57CCA (D5)F50.29OursCCAO61.89gradKCCA (D5)F61.84OursDCAO75.93DCA (D5)F73.71OursDCA (coarse)O73.86WILDCATB67.03OursCCM (coarse)O79.16WILDCAT (D5)B70.09OursCCMO80.32First, our model outperforms the classical fine-tuned deep learning models. For example, compared to the most powerful VGG16 and Res101 models, our model improves 3.34% on the Twitter I dataset and 2.21% on the FI dataset, respectively. As we know, these classical deep learning models use very complex network structures, which make them more prone to overfitting when high-quality images are very scarce. This phenomenon is more evident on the FI dataset because this fine-grained dataset needs more high-quality samples to train a robust model that can discriminate more sentiment categories. Contrarily, our model handles the above problem well. It uses the ASR strategy to obtain sufficient high-quality images (please refer to Table1), which can augment the original benchmark datasets and build a firm data foundation for the subsequent classification. Meanwhile, the new CMS fits the number of training samples well, which helps reduce the risk of overfitting to a certain degree. More importantly, owing to a simpler structure (please refer to Figure 3), our model is relatively easier to reproduce. Moreover, we found another interesting phenomenon. The ResNet 101 model obtains about 2.50% performance improvement on the FI dataset when the augmented “D5 + Dfine” data are used. Contrarily, the VGG 16 model gets some performance decline when the augmented “D5 + Dfine” data are used. We guess the proposed ASR strategy enriches the original benchmark dataset from the fine-grained sentiment perspective. This offers sufficient training samples for the fine-tuned deep learning models. However, for the coarse-grained Twitter I dataset, the refined samples tend to be consistent with the existing samples in sentimental semantics, which has a little positive influence on performance improvement. Hence, more robust performance improvements can be observed on the fine-grained FI dataset.Second, our model beats the state-of-the-art feature fusion methods. For example, compared to “GS-XGB (D5 + Dfine),” a 4.34% performance improvement is observed on the FI dataset because the ASR strategy can enrich the original dataset from the fine-grained sentiment perspective. Additionally, our model improves 6.61% compared to the DCA model on the FI dataset. This is another evidence of the effectiveness of our ASR strategy. We also found that the performance of CCA is lower than that of DCA on each dataset. Evidently, the corresponding sample distribution of DCA is better than that of CCA (please see Figure 7). The proposed ASR strategy plays an important role in our model. It offers us plentiful high-quality image samples (Table 1). Meanwhile, our model performs CCM, which is a relatively simple feature early-fusion strategy only considering interclass dependence. Contrarily, the GS-based methods require numerous features and need to perform a very complex feature mid-fusion procedure. Thus, our model is easier to reproduce, demonstrating its practicality. Similarly, obvious performance improvements can be observed on the FI dataset when the “D5 + Dfine” data are used. Our ASR strategy offers more valuable samples for the fine-grained FI dataset. So, more robust performance improvements can be observed on this dataset.Last, our model exhibits superior performance over the state-of-the-art baselines. The ME2M (M) and SR-w-DCA models perform static sample refinement. Unlike them, our model performs ASR to adaptively and gradually augment the original datasets, which can bring us more valuable samples. The SPN and WILDCAT methods focus on obtaining the local regions with very strong sentiment tendency, which mostly relies on additional manual annotations. The corresponding performance is unsatisfactory. However, the proposed model only uses the whole image and does not need any additional annotations. This helps alleviate the sentiment annotation burden. The SmileyNet leverages additional private emoji information, whereas our model only uses the publicly available images (D3-5). Compared with the domain adaptation-based methods (MSGAN and CycleGAN), our model exhibits outstanding performance. These domain adaptation-based approaches should face a significant gap between the source and target domains. Hence, it is hard to train them. Contrarily, owing to a simple but clear structure, it is easier to train the proposed model. Moreover, we observe that the CCM strategy is effective and robust, which beats other correlation analysis methods by a large margin. More importantly, larger performance improvements can be observed on the fine-grained FI dataset, demonstrating that our model is highly practical. In our daily life, we usually present more fine-grained emotions rather than “like” or “dislike.” Finally, to our surprise, we found that the refined samples can improve the final performance of the state-of-the-art baselines (please compare “WSCNet (D5 + Dfine)” and “WSCNet (D5)”). This can firmly demonstrate the generalization of the ASR strategy. Therefore, this strategy is an effective and robust data augmentation method. We will make further analysis in the following section.In summary, the proposed model is effective and robust for image sentiment analysis. On the one hand, the ASR strategy can alleviate the data scarcity problem and augment the original benchmark datasets. On the other hand, we created a set of more discriminant but robust features called CMS by mining the cluster correlation among the heterogeneous SENet features. Meanwhile, our two variants (OursCCA and OursDCA) also get very competitive performance, demonstrating the generalization ability of the proposed idea. Moreover, Ours CCM beats OursCCM (coarse) in a large performance margin, which proves that Dfine really contains more high-quality samples than Dcoarse and this fine-grained data are effective for image sentiment analysis. Based on the ASR and CCM strategies, our model beats the state-of-the-art baseline models.


### 4.2. Effectiveness Evaluation of ASR and CCM

In this section, we simultaneously evaluate the ASR and CCM strategies from another view. To obtain statistical interpretations, five SENet features and ten CMSs are chosen to complete the corresponding experiments on the D5 and “D5 + Dfine” datasets, respectively. Then, the mean accuracies are presented in Figure 4.Figure 4
Effectiveness evaluations of the ASR and CCM strategies. “ORI” represents the original SENet feature. CMS represents the proposed cross-modal semantics. “ORI (ASR)” means that we use the SENet feature on “D5 + Dfine.” “CMS (ASR)” means that we use CMS on “D5 + Dfine.” (a) Twitter I (b) FI.
(a)(b)Figure4 shows that any CMS outperforms the corresponding SENet feature. For example, a 2.59% performance improvement of the AdaBoost classifier can be observed on the FI dataset. This phenomenon is more evident in the boosting-based classifiers, such as AdaBoost and XGBoost. A similar phenomenon can be found on each dataset. All these demonstrate the effectiveness and robustness of the CCM strategy. The implicit valuable discriminative information among the heterogeneous SENet layers is fully mined and used to promote the final performance. Compared with D5, the augmented dataset (“D5 + Dfine”) performs better. For example, a 1.58% improvement of the AdaBoost classifier can be observed on the FI dataset. All these demonstrate the effectiveness and robustness of the ASR strategy. Sufficient high-quality images with definite sentiment semantics “generated” by the ASR strategy build a firm data foundation for model training. Surprisingly, the single ASR strategy combined with SVM (or LR) outperforms CMS without ASR on the FI dataset. More high-quality data play a positive role in image sentiment analysis. More importantly, our model seamlessly combines ASR and CCM to achieve the best performance.Herein, we make an ablation analysis to evaluate the real contribution of each component of our model. We calculate the average improvement of four classifiers on both datasets. Compared to the original SENet feature, the corresponding performance improvement of “ORI (ASR),” “CMS,” and “CMS (ASR)” is 0.24%, 1.20%, and 1.34% on Twitter I whereas 1.41%, 1.52%, and 2.57% on FI. This is a very interesting phenomenon. Meanwhile, to validate the real contribution of ASR and CMS, we calculate the average performance difference between “ORI” and “ORI (ASR)” on the two datasets. A similar difference between “CMS” and “ORI” can also be obtained. The average performance difference is 0.83% and 1.36%, respectively. This indicates that CMS contributes more than the proposed ASR strategy. We can say the feature is more important than the refined samples. Certainly, the combination of the proposed CMS and ASR strategies plays a positive role in image sentiment analysis.To further demonstrate the effectiveness and versatility of the ASR strategy, we performed four traditional data augmentations (TDAs), including rotate, crop, saturate, and scale operations, and the ASR strategy on the D5 data subset of Twitter I. The corresponding results are provided in Table 3. We selected the fine-tuned VGG16 [39], fine-tuned Res101 [42], WILDCAT [19], and WSCNet [17] to complete comparisons. In Table 3, “NDA” represents “no data augmentation.” “TDA + ASR” means that we performed the ASR strategy after “TDA.” Our experimental details are as follows: first, we chose a basic data augmentation operation among all the four operations. Then, we combined this basic operation with the other three random augmentation operations to generate new images. Hence, the “TDA” approach generated 3,528 images based on the D5 data. Furthermore, we implemented the ASR strategy on these images. Thus, 2,121 refined image samples are obtained. Finally, we used the refined image samples to train each model introduced above and made performance comparisons. We also combined the proposed ASR strategy with the popular adaptive data augmentation method called Fast AutoAugment (FAA). We implemented ASR after the application of the FAA. The corresponding model is Ours FAA+ASR.Table 3
Effectiveness and versatility evaluation of ASR. The best value is shown as92.57. Unit: %.
MethodsAccuracyMethodsAccuracyMethodsAccuracyVGG16NDA86.56Res101ASR86.93WSCNetNDA87.62VGG16TDA88.75Res101TDA+ASR90.18WSCNetTDA88.23VGG16ASR82.69WILDCATNDA71.43WSCNetASR89.40VGG16TDA+ASR89.46WILDCATTDA85.00WSCNetTDA+ASR89.64Res101NDA87.96WILDCATASR73.85OursFAA+ASR88.30Res101TDA88.37WILDCATTDA+ASR92.57OursCCM89.90As shown in Table3, apparent performance improvements can be observed when using the “TDA” method. For example, a 2.2% performance improvement is obtained on the VGG16 model. This demonstrates the effectiveness of the “TDA” method. Hence, most CV tasks employ the “TDA” method to augment their own datasets. However, we got further performance improvements by combining the proposed ASR strategy with “TDA.” For example, a 1.81% performance improvement is obtained on the Res101 model by using “TDA + ASR.” We found that owing to more scarce data, only using the ASR strategy on the fine-tuned deep learning models brings performance degradation (similar to that of Table 2). Surprisingly, unlike the fine-tuned deep learning models, more robust performance occurs on the state-of-the-art WILDCAT and WSCNet models. Hence, our sample refinement strategy can be regarded as a useful supplement to the current “TDA” method, which can be seamlessly incorporated into the state-of-the-art baselines to obtain better performance (please see WILDCATTDA+ASR). However, when we combine the ASR strategy with FAA, there is a certain performance degradation. The underlying reason is that there are still a small amount of samples after the combination of FAA and ASR.In conclusion, the ASR strategy is effective and versatile. It can be combined with the “TDA” method to obtain better performance. Hence, it is a useful supplement to current data augmentation methods. More importantly, it can be incorporated into the state-of-the-art baselines, such as WILDCAT and WSCNet, to achieve better performance.


### 4.3. Real-Time Efficiency Comparison

Table4 shows the average time that our model needs to test one image. The testing procedure can be roughly divided into three stages: feature extraction, CCM, and final classification. Since the feature selection procedure has been completed in the training stage, we only need to extract two features here. Meanwhile, we save our classifier models at the training stage. Therefore, we only require to use them for the final classification. Because it is too fast, the time cost of ensemble learning can be negligible here. The entire time required to test one image of our model is 0.17 s, demonstrating that our model is lightweight and easy to deploy.Table 4
Real-time efficiency comparison. The best value is shown as 0.170. Unit:s.
MethodTest timeMethodTest timeVGG160.053ME2M (M)2.170Res1010.098Ours0.170


### 4.4. Parameters Tuning

#### 4.4.1. Feature Selection

In this section, we need to choose effective but robust image features for CCM. We selected SIFT [45], GIST [46], LBP [47], DenseNet, VGG [39], ResNet, SENet [33], and SENetXT as our candidate features. We implemented principal component analysis on each traditional feature. Thus, each traditional feature was reduced to 500. For the deep learning features, we chose the bottleneck features of the corresponding CNN model. For example, we used the 16th and 19th layers of the VGG model in our experiments. Owing to a large amount of valuable distributed information, we did not implement feature reduction on these deep learning features. Detailed experimental results are shown in Table 5. “Dim” means dimension.Table 5
Classification performance of each single image feature. Unit: %.
DatasetsFeatureAccuracyDimFeatureAccuracyDimTwitter ISIFT83.02500VGG1682.264096GIST69.43500VGG1983.774096LBP67.55500SENet10187.802048DenseNet12167.171024SENet15288.502048DenseNet16165.662208SENet5084.322048ResNet5070.192048SENetXT10187.112048ResNet15269.062048SENetXT5089.202048FISIFT55.47500VGG1671.184096GIST50.51500VGG1970.994096LBP46.85500SENet10178.322048DenseNet12129.131024SENet15278.212048DenseNet16142.292208SENet5077.242048ResNet5057.442048SENetXT10179.462048ResNet15258.722048SENetXT5077.982048As shown in Table5, the five SENet features outperform any other feature. In particular, the two SENetXT features obtain the best trade-off between accuracy and efficiency. As we know, the SENet features pay more attention to those local regions with strong sentiment tendencies, including Ferris wheel, smiling face, shabby house, and so on. These sentimental signals can accurately depict the visual content and create a powerful foundation for CCM. In contrast, SIFT mainly describes the key gradient variations in images. However, the gradient variations between background and foreground sometimes are not obvious. GIST and LBP mainly describe the global and local textures in images. Texture only represents a little part of emotion, which is weaker than other features. Summarily, SENet is the best choice for our model. Meanwhile, compared with other features, SENet requires fewer resources and is efficient for extraction. Hence, we chose the five SENet features to complete CCM.


#### 4.4.2. Classifier Tuning for Fine-Grained Sample Refinement

As described above, to avoid selective bias, we employed several heterogeneous classifiers to complete ASR. Therefore, classifier type and classifier number should be appropriately tuned to obtain the best performance. Here, we exhibit the corresponding tuning procedure for fine-grained sample refinement. For the Twitter I dataset, we chose LR and k-nearest neighbor (KNN) algorithms to perform fine-grained sample refinement. For the FI dataset, we used CatBoost, AdaBoost, random forest (RF), and LR to complete the same procedure. The refined samples of each classifier were used to perform the union operation for augmenting the existing benchmark datasets. The corresponding experimental results are shown in Figure5. As analyzed above, the DCA feature has a lower dimension than the proposed CCM. As shown in Table 2, the DCA feature obtains very competitive accuracy. Hence, to save resources, we used the DCA model to fuse the original features, and then, we performed classification using CatBoost.Figure 5
Classifier tuning for fine-grained sample refinement. (a) Twitter I. (b) FI.
(a)(b)As shown in Figure5, using multiple heterogeneous but complementary classifiers can prevent selective bias to a certain degree. More detailed, using multiple classifiers to implement ASR obtains a 1.38% performance improvement compared to using LR on the Twitter I dataset. For the FI dataset, compared with a single classifier, using multiple classifiers obtains the corresponding performance improvements of 1.99%, 1.37%, and 0.64%, respectively. In Figure 5(a), the best accuracy on the testing dataset is obtained when all the classifiers are used. Owing to the lack of (or imbalance of image samples), slight overfitting is observed on the Twitter I dataset. Therefore, training relatively small datasets is still challenging. In the future, we plan to introduce the well-known zero-shot learning method [48] to address this problem. In Figure 5(b), our model performs better when all the classifiers are used. The complementarity among multiple heterogeneous classifiers is fully utilized to boost the final performance. Summarily, we achieved the optimal classifier combination based on the corresponding accuracy and fitting status of the trained model.


#### 4.4.3. Parameters Tuning of Ensemble Learning

Based on the classification results of different classifiers, we employed the ensemble learning strategy to further improve the final performance. Different ensemble learning strategies (hard voting or soft voting) combined with different classifier numbers result in different results. Hence, we should carefully tune the corresponding ensemble learning strategy combined with the top two (or three) classifiers to obtain the best performance. To obtain statistical interpretations, we used ten CMSs with different ensemble learning strategies to complete the tuning procedure. Figure6 exhibits the corresponding experimental results.Figure 6
Parameters tuning of ensemble learning. (a) Twitter I. (b) FI.
(a)(b)As shown in Figure6, the top three classifiers combined with the hard voting strategy perform best on any benchmark dataset. The top three classifiers play a decisive role in the final image sentiment prediction. The implicit complementarity among multiple heterogeneous classifiers is fully mined by the ensemble learning strategy to improve the final performance. In addition, the hard voting strategy is fairer than the weighted soft voting strategy. Hence, we used this setting to complete our experiments.


#### 4.4.4. Visualization Results of Different Features

To intuitively demonstrate the effectiveness of the CCM strategy, in this section, we employ the well-known t-SNE [49] tool to visualize the sample distribution of each kind of CCM. Figure 7 exhibits the corresponding visualization results of different image features. Owing to space limitations, only the results of the Twitter I dataset are provided here. Similar experimental results can be observed on the FI dataset.Figure 7
Visualization results of different features. (a) SENetXT50. (b) SENet 152. (c) SENet101. (d) CCA. (e) DCA. (f) CMS.
(a)(b)(c)(d)(e)(f)In Figures7(a)∼7(c), the corresponding sample distributions of the original SENet features show slight chaos. The aggregation degree of the image samples in the same class is low while that in different classes mixes up each other. Hence, it is difficult for a traditional classifier to fit the sample distribution (please refer to Table 5, the corresponding accuracy of the SENet152 feature is 88.50%. It builds a solid foundation for the subsequent CCM) using the original SENet feature. A relatively complex decision boundary of the traditional classifier is needed to address this problem. However, owing to the lack of high-quality images with definite sentiment semantics, overfitting may be prone to occur. So, a single image feature is insufficient for effective image sentiment analysis. As shown in Figure 7(d), the corresponding sample distribution of the CCA feature is worse than that of any SENet feature. This will bring a challenge to the traditional classifiers (in Table 2, the corresponding accuracy of the CCA feature is 80.08%). Similarly, heavy chaos can also be observed in Figure 7(e). But the result is better than Figure 7(d). As shown in Table 2, the corresponding accuracy of the DCA feature is 87.59%. Unlike other methods (CCA and DCA), as shown in Figure 7(f), the corresponding sample distribution of CMS is better than that of a single SENet feature. The proposed CCM strategy can increase the corresponding distance among different classes. Hence, a relatively simple decision boundary of the traditional classifier is needed to complete classification (in Table 3, the corresponding accuracy of our model is 89.90%). Summarily, the above visualization results intuitively validate the effectiveness of the proposed CCM strategy.


### 4.5. Empirical Analysis

We selected some open images from WeChat and microblog to prove the practicality of our model. We performed our model (the coarse-grained sentiment classification model) on these images. Experimental results are shown in Table6. As shown in Table 6, we classified these images into two categories (positive and negative), and the smiling face usually represents positive sentiment tendency (), while the depressed face usually represents negative sentiment tendency ().Table 6
Evaluation on WeChat and microblog.
Web imageTrue labelOur predictionThe first and second images have a riot of color, whereas the third and fourth images are generally in dark color. Moreover, the first image has a smiling face, but the third image has a sad face. Our model performs well in predicting the sentiment polarity of the first to the third images. This firmly demonstrates the practicality of the proposed model. Certainly, our model makes a prediction bias on the fourth image. Negative images are far fewer than positive images in the Twitter I dataset. Hence, we speculate that the prediction bias is mainly due to the lack of negative images. This further illustrates the importance of the data augmentation operation from another perspective. Summarily, besides the publicly available datasets, our model also performs well on the open images, demonstrating its high practicality.


## 4.1. Comparisons with State-of-the-Arts

In this section, we compare our model with the following state-of-the-art methods:(1)
Classical fine-tuned deep learning models, namely, VGG16 [39], AlexNet [40], CAM-Res101 [41], and Res101 [42].(2)
State-of-the-art feature fusion methods, namely, DCA [15], canonical correlation analysis (CCA) [43], gradKCCA [44], GS-SVM [13], GS-LR [13], and GS-XGB [13].(3)
Recently proposed baseline methods for image sentiment analysis, namely, ResNet-MldrNet [24], SPN [18], WSCNet [17], WILDCAT [19], ME2M (M) [27], SR-w-DCA [27], CycleGAN [25], CycleEmotionGAN [26], MSGAN [27], Sun’s model [20], Rao’s model [21], and SmileyNet [24].We also created two variants of our model by using different correlation mining methods, namely, DCA [15] and CCA [43]. We wanted to demonstrate the generalization of the proposed idea. So, we used “OursCCA” and “OursDCA” to represent the two variants. Meanwhile, in order to demonstrate that Dfine contains more high-quality image samples than Dcoarse, we also perform our model on Dcoarse and compare the performance on Dcoarse with that on Dfine. Hence, “OursCCM (coarse)” means that we perform our model on Dcoarse. The corresponding experimental results are shown in Table 2. Herein, “D5” represents that we only used the D5 data to evaluate the classification model. “D5 + Dfine” means that we executed each model on the augmented dataset. Others only used D3 to evaluate each model. Hence, we trained all the methods under the same setting. Moreover, in the “Type” column, “D” means deep learning-based method. “F” means feature fusion method. “B” means state-of-the-art baseline. “O” means our proposed model.Table 2
Performance comparisons. The best value of each dataset is shown as89.90. Unit: %.
DatasetMethodTypeAccuracyMethodTypeAccuracyTwitter IVGG16D76.75WILDCAT (D5+Dfine)B73.85VGG16 (D5)D86.56WSCNetB84.25VGG16 (D5+Dfine)D82.69WSCNet (D5)B87.62AlexNetD73.24WSCNet (D5+Dfine)B89.40CAM-Res101D82.67MSGANB63.58GS-SVM (D5)F88.72CycleGANB61.59GS-LR (D5)F87.22CycleEmotionGANB62.38GS-XGB (D5)F86.47SmileyNetB89.16GS-SVM (D5+Dfine)F82.04Sun’s modelB88.91GS-LR (D5+Dfine)F84.51ME2M (M)B87.15GS-XGB (D5+Dfine)F83.10SPNB81.67CCA (D5)F80.08OursCCAO85.71gradKCCA (D5)F77.07OursCCA (coarse)O77.81DCA (D5)F87.59OursDCAO82.23WILDCATB79.53OursCCM (coarse)O89.07WILDCAT (D5)B71.43OursCCMO89.90FIRes101D66.16WILDCAT (D5+Dfine)B72.23Res101 (D5)D75.61WSCNetB70.07Res101 (D5+Dfine)D78.11WSCNet (D5)B72.22AlexNetD58.13WSCNet (D5+Dfine)B74.72CAM-Res101D68.54MSGANB70.63GS-SVM (D5)F73.52CycleGANB63.87GS-LR (D5)F73.59CycleEmotionGANB67.78GS-XGB (D5)F72.89MldrNetB67.75GS-SVM (D5+Dfine)F74.79Rao’s modelB75.46GS-LR (D5+Dfine)F75.30SR-w-DCAB75.72GS-XGB (D5+Dfine)F75.98SPNB66.57CCA (D5)F50.29OursCCAO61.89gradKCCA (D5)F61.84OursDCAO75.93DCA (D5)F73.71OursDCA (coarse)O73.86WILDCATB67.03OursCCM (coarse)O79.16WILDCAT (D5)B70.09OursCCMO80.32First, our model outperforms the classical fine-tuned deep learning models. For example, compared to the most powerful VGG16 and Res101 models, our model improves 3.34% on the Twitter I dataset and 2.21% on the FI dataset, respectively. As we know, these classical deep learning models use very complex network structures, which make them more prone to overfitting when high-quality images are very scarce. This phenomenon is more evident on the FI dataset because this fine-grained dataset needs more high-quality samples to train a robust model that can discriminate more sentiment categories. Contrarily, our model handles the above problem well. It uses the ASR strategy to obtain sufficient high-quality images (please refer to Table1), which can augment the original benchmark datasets and build a firm data foundation for the subsequent classification. Meanwhile, the new CMS fits the number of training samples well, which helps reduce the risk of overfitting to a certain degree. More importantly, owing to a simpler structure (please refer to Figure 3), our model is relatively easier to reproduce. Moreover, we found another interesting phenomenon. The ResNet 101 model obtains about 2.50% performance improvement on the FI dataset when the augmented “D5 + Dfine” data are used. Contrarily, the VGG 16 model gets some performance decline when the augmented “D5 + Dfine” data are used. We guess the proposed ASR strategy enriches the original benchmark dataset from the fine-grained sentiment perspective. This offers sufficient training samples for the fine-tuned deep learning models. However, for the coarse-grained Twitter I dataset, the refined samples tend to be consistent with the existing samples in sentimental semantics, which has a little positive influence on performance improvement. Hence, more robust performance improvements can be observed on the fine-grained FI dataset.Second, our model beats the state-of-the-art feature fusion methods. For example, compared to “GS-XGB (D5 + Dfine),” a 4.34% performance improvement is observed on the FI dataset because the ASR strategy can enrich the original dataset from the fine-grained sentiment perspective. Additionally, our model improves 6.61% compared to the DCA model on the FI dataset. This is another evidence of the effectiveness of our ASR strategy. We also found that the performance of CCA is lower than that of DCA on each dataset. Evidently, the corresponding sample distribution of DCA is better than that of CCA (please see Figure 7). The proposed ASR strategy plays an important role in our model. It offers us plentiful high-quality image samples (Table 1). Meanwhile, our model performs CCM, which is a relatively simple feature early-fusion strategy only considering interclass dependence. Contrarily, the GS-based methods require numerous features and need to perform a very complex feature mid-fusion procedure. Thus, our model is easier to reproduce, demonstrating its practicality. Similarly, obvious performance improvements can be observed on the FI dataset when the “D5 + Dfine” data are used. Our ASR strategy offers more valuable samples for the fine-grained FI dataset. So, more robust performance improvements can be observed on this dataset.Last, our model exhibits superior performance over the state-of-the-art baselines. The ME2M (M) and SR-w-DCA models perform static sample refinement. Unlike them, our model performs ASR to adaptively and gradually augment the original datasets, which can bring us more valuable samples. The SPN and WILDCAT methods focus on obtaining the local regions with very strong sentiment tendency, which mostly relies on additional manual annotations. The corresponding performance is unsatisfactory. However, the proposed model only uses the whole image and does not need any additional annotations. This helps alleviate the sentiment annotation burden. The SmileyNet leverages additional private emoji information, whereas our model only uses the publicly available images (D3-5). Compared with the domain adaptation-based methods (MSGAN and CycleGAN), our model exhibits outstanding performance. These domain adaptation-based approaches should face a significant gap between the source and target domains. Hence, it is hard to train them. Contrarily, owing to a simple but clear structure, it is easier to train the proposed model. Moreover, we observe that the CCM strategy is effective and robust, which beats other correlation analysis methods by a large margin. More importantly, larger performance improvements can be observed on the fine-grained FI dataset, demonstrating that our model is highly practical. In our daily life, we usually present more fine-grained emotions rather than “like” or “dislike.” Finally, to our surprise, we found that the refined samples can improve the final performance of the state-of-the-art baselines (please compare “WSCNet (D5 + Dfine)” and “WSCNet (D5)”). This can firmly demonstrate the generalization of the ASR strategy. Therefore, this strategy is an effective and robust data augmentation method. We will make further analysis in the following section.In summary, the proposed model is effective and robust for image sentiment analysis. On the one hand, the ASR strategy can alleviate the data scarcity problem and augment the original benchmark datasets. On the other hand, we created a set of more discriminant but robust features called CMS by mining the cluster correlation among the heterogeneous SENet features. Meanwhile, our two variants (OursCCA and OursDCA) also get very competitive performance, demonstrating the generalization ability of the proposed idea. Moreover, Ours CCM beats OursCCM (coarse) in a large performance margin, which proves that Dfine really contains more high-quality samples than Dcoarse and this fine-grained data are effective for image sentiment analysis. Based on the ASR and CCM strategies, our model beats the state-of-the-art baseline models.


## 4.2. Effectiveness Evaluation of ASR and CCM

In this section, we simultaneously evaluate the ASR and CCM strategies from another view. To obtain statistical interpretations, five SENet features and ten CMSs are chosen to complete the corresponding experiments on the D5 and “D5 + Dfine” datasets, respectively. Then, the mean accuracies are presented in Figure 4.Figure 4
Effectiveness evaluations of the ASR and CCM strategies. “ORI” represents the original SENet feature. CMS represents the proposed cross-modal semantics. “ORI (ASR)” means that we use the SENet feature on “D5 + Dfine.” “CMS (ASR)” means that we use CMS on “D5 + Dfine.” (a) Twitter I (b) FI.
(a)(b)Figure4 shows that any CMS outperforms the corresponding SENet feature. For example, a 2.59% performance improvement of the AdaBoost classifier can be observed on the FI dataset. This phenomenon is more evident in the boosting-based classifiers, such as AdaBoost and XGBoost. A similar phenomenon can be found on each dataset. All these demonstrate the effectiveness and robustness of the CCM strategy. The implicit valuable discriminative information among the heterogeneous SENet layers is fully mined and used to promote the final performance. Compared with D5, the augmented dataset (“D5 + Dfine”) performs better. For example, a 1.58% improvement of the AdaBoost classifier can be observed on the FI dataset. All these demonstrate the effectiveness and robustness of the ASR strategy. Sufficient high-quality images with definite sentiment semantics “generated” by the ASR strategy build a firm data foundation for model training. Surprisingly, the single ASR strategy combined with SVM (or LR) outperforms CMS without ASR on the FI dataset. More high-quality data play a positive role in image sentiment analysis. More importantly, our model seamlessly combines ASR and CCM to achieve the best performance.Herein, we make an ablation analysis to evaluate the real contribution of each component of our model. We calculate the average improvement of four classifiers on both datasets. Compared to the original SENet feature, the corresponding performance improvement of “ORI (ASR),” “CMS,” and “CMS (ASR)” is 0.24%, 1.20%, and 1.34% on Twitter I whereas 1.41%, 1.52%, and 2.57% on FI. This is a very interesting phenomenon. Meanwhile, to validate the real contribution of ASR and CMS, we calculate the average performance difference between “ORI” and “ORI (ASR)” on the two datasets. A similar difference between “CMS” and “ORI” can also be obtained. The average performance difference is 0.83% and 1.36%, respectively. This indicates that CMS contributes more than the proposed ASR strategy. We can say the feature is more important than the refined samples. Certainly, the combination of the proposed CMS and ASR strategies plays a positive role in image sentiment analysis.To further demonstrate the effectiveness and versatility of the ASR strategy, we performed four traditional data augmentations (TDAs), including rotate, crop, saturate, and scale operations, and the ASR strategy on the D5 data subset of Twitter I. The corresponding results are provided in Table 3. We selected the fine-tuned VGG16 [39], fine-tuned Res101 [42], WILDCAT [19], and WSCNet [17] to complete comparisons. In Table 3, “NDA” represents “no data augmentation.” “TDA + ASR” means that we performed the ASR strategy after “TDA.” Our experimental details are as follows: first, we chose a basic data augmentation operation among all the four operations. Then, we combined this basic operation with the other three random augmentation operations to generate new images. Hence, the “TDA” approach generated 3,528 images based on the D5 data. Furthermore, we implemented the ASR strategy on these images. Thus, 2,121 refined image samples are obtained. Finally, we used the refined image samples to train each model introduced above and made performance comparisons. We also combined the proposed ASR strategy with the popular adaptive data augmentation method called Fast AutoAugment (FAA). We implemented ASR after the application of the FAA. The corresponding model is Ours FAA+ASR.Table 3
Effectiveness and versatility evaluation of ASR. The best value is shown as92.57. Unit: %.
MethodsAccuracyMethodsAccuracyMethodsAccuracyVGG16NDA86.56Res101ASR86.93WSCNetNDA87.62VGG16TDA88.75Res101TDA+ASR90.18WSCNetTDA88.23VGG16ASR82.69WILDCATNDA71.43WSCNetASR89.40VGG16TDA+ASR89.46WILDCATTDA85.00WSCNetTDA+ASR89.64Res101NDA87.96WILDCATASR73.85OursFAA+ASR88.30Res101TDA88.37WILDCATTDA+ASR92.57OursCCM89.90As shown in Table3, apparent performance improvements can be observed when using the “TDA” method. For example, a 2.2% performance improvement is obtained on the VGG16 model. This demonstrates the effectiveness of the “TDA” method. Hence, most CV tasks employ the “TDA” method to augment their own datasets. However, we got further performance improvements by combining the proposed ASR strategy with “TDA.” For example, a 1.81% performance improvement is obtained on the Res101 model by using “TDA + ASR.” We found that owing to more scarce data, only using the ASR strategy on the fine-tuned deep learning models brings performance degradation (similar to that of Table 2). Surprisingly, unlike the fine-tuned deep learning models, more robust performance occurs on the state-of-the-art WILDCAT and WSCNet models. Hence, our sample refinement strategy can be regarded as a useful supplement to the current “TDA” method, which can be seamlessly incorporated into the state-of-the-art baselines to obtain better performance (please see WILDCATTDA+ASR). However, when we combine the ASR strategy with FAA, there is a certain performance degradation. The underlying reason is that there are still a small amount of samples after the combination of FAA and ASR.In conclusion, the ASR strategy is effective and versatile. It can be combined with the “TDA” method to obtain better performance. Hence, it is a useful supplement to current data augmentation methods. More importantly, it can be incorporated into the state-of-the-art baselines, such as WILDCAT and WSCNet, to achieve better performance.


## 4.3. Real-Time Efficiency Comparison

Table4 shows the average time that our model needs to test one image. The testing procedure can be roughly divided into three stages: feature extraction, CCM, and final classification. Since the feature selection procedure has been completed in the training stage, we only need to extract two features here. Meanwhile, we save our classifier models at the training stage. Therefore, we only require to use them for the final classification. Because it is too fast, the time cost of ensemble learning can be negligible here. The entire time required to test one image of our model is 0.17 s, demonstrating that our model is lightweight and easy to deploy.Table 4
Real-time efficiency comparison. The best value is shown as 0.170. Unit:s.
MethodTest timeMethodTest timeVGG160.053ME2M (M)2.170Res1010.098Ours0.170


## 4.4. Parameters Tuning

### 4.4.1. Feature Selection

In this section, we need to choose effective but robust image features for CCM. We selected SIFT [45], GIST [46], LBP [47], DenseNet, VGG [39], ResNet, SENet [33], and SENetXT as our candidate features. We implemented principal component analysis on each traditional feature. Thus, each traditional feature was reduced to 500. For the deep learning features, we chose the bottleneck features of the corresponding CNN model. For example, we used the 16th and 19th layers of the VGG model in our experiments. Owing to a large amount of valuable distributed information, we did not implement feature reduction on these deep learning features. Detailed experimental results are shown in Table 5. “Dim” means dimension.Table 5
Classification performance of each single image feature. Unit: %.
DatasetsFeatureAccuracyDimFeatureAccuracyDimTwitter ISIFT83.02500VGG1682.264096GIST69.43500VGG1983.774096LBP67.55500SENet10187.802048DenseNet12167.171024SENet15288.502048DenseNet16165.662208SENet5084.322048ResNet5070.192048SENetXT10187.112048ResNet15269.062048SENetXT5089.202048FISIFT55.47500VGG1671.184096GIST50.51500VGG1970.994096LBP46.85500SENet10178.322048DenseNet12129.131024SENet15278.212048DenseNet16142.292208SENet5077.242048ResNet5057.442048SENetXT10179.462048ResNet15258.722048SENetXT5077.982048As shown in Table5, the five SENet features outperform any other feature. In particular, the two SENetXT features obtain the best trade-off between accuracy and efficiency. As we know, the SENet features pay more attention to those local regions with strong sentiment tendencies, including Ferris wheel, smiling face, shabby house, and so on. These sentimental signals can accurately depict the visual content and create a powerful foundation for CCM. In contrast, SIFT mainly describes the key gradient variations in images. However, the gradient variations between background and foreground sometimes are not obvious. GIST and LBP mainly describe the global and local textures in images. Texture only represents a little part of emotion, which is weaker than other features. Summarily, SENet is the best choice for our model. Meanwhile, compared with other features, SENet requires fewer resources and is efficient for extraction. Hence, we chose the five SENet features to complete CCM.


### 4.4.2. Classifier Tuning for Fine-Grained Sample Refinement

As described above, to avoid selective bias, we employed several heterogeneous classifiers to complete ASR. Therefore, classifier type and classifier number should be appropriately tuned to obtain the best performance. Here, we exhibit the corresponding tuning procedure for fine-grained sample refinement. For the Twitter I dataset, we chose LR and k-nearest neighbor (KNN) algorithms to perform fine-grained sample refinement. For the FI dataset, we used CatBoost, AdaBoost, random forest (RF), and LR to complete the same procedure. The refined samples of each classifier were used to perform the union operation for augmenting the existing benchmark datasets. The corresponding experimental results are shown in Figure5. As analyzed above, the DCA feature has a lower dimension than the proposed CCM. As shown in Table 2, the DCA feature obtains very competitive accuracy. Hence, to save resources, we used the DCA model to fuse the original features, and then, we performed classification using CatBoost.Figure 5
Classifier tuning for fine-grained sample refinement. (a) Twitter I. (b) FI.
(a)(b)As shown in Figure5, using multiple heterogeneous but complementary classifiers can prevent selective bias to a certain degree. More detailed, using multiple classifiers to implement ASR obtains a 1.38% performance improvement compared to using LR on the Twitter I dataset. For the FI dataset, compared with a single classifier, using multiple classifiers obtains the corresponding performance improvements of 1.99%, 1.37%, and 0.64%, respectively. In Figure 5(a), the best accuracy on the testing dataset is obtained when all the classifiers are used. Owing to the lack of (or imbalance of image samples), slight overfitting is observed on the Twitter I dataset. Therefore, training relatively small datasets is still challenging. In the future, we plan to introduce the well-known zero-shot learning method [48] to address this problem. In Figure 5(b), our model performs better when all the classifiers are used. The complementarity among multiple heterogeneous classifiers is fully utilized to boost the final performance. Summarily, we achieved the optimal classifier combination based on the corresponding accuracy and fitting status of the trained model.


### 4.4.3. Parameters Tuning of Ensemble Learning

Based on the classification results of different classifiers, we employed the ensemble learning strategy to further improve the final performance. Different ensemble learning strategies (hard voting or soft voting) combined with different classifier numbers result in different results. Hence, we should carefully tune the corresponding ensemble learning strategy combined with the top two (or three) classifiers to obtain the best performance. To obtain statistical interpretations, we used ten CMSs with different ensemble learning strategies to complete the tuning procedure. Figure6 exhibits the corresponding experimental results.Figure 6
Parameters tuning of ensemble learning. (a) Twitter I. (b) FI.
(a)(b)As shown in Figure6, the top three classifiers combined with the hard voting strategy perform best on any benchmark dataset. The top three classifiers play a decisive role in the final image sentiment prediction. The implicit complementarity among multiple heterogeneous classifiers is fully mined by the ensemble learning strategy to improve the final performance. In addition, the hard voting strategy is fairer than the weighted soft voting strategy. Hence, we used this setting to complete our experiments.


### 4.4.4. Visualization Results of Different Features

To intuitively demonstrate the effectiveness of the CCM strategy, in this section, we employ the well-known t-SNE [49] tool to visualize the sample distribution of each kind of CCM. Figure 7 exhibits the corresponding visualization results of different image features. Owing to space limitations, only the results of the Twitter I dataset are provided here. Similar experimental results can be observed on the FI dataset.Figure 7
Visualization results of different features. (a) SENetXT50. (b) SENet 152. (c) SENet101. (d) CCA. (e) DCA. (f) CMS.
(a)(b)(c)(d)(e)(f)In Figures7(a)∼7(c), the corresponding sample distributions of the original SENet features show slight chaos. The aggregation degree of the image samples in the same class is low while that in different classes mixes up each other. Hence, it is difficult for a traditional classifier to fit the sample distribution (please refer to Table 5, the corresponding accuracy of the SENet152 feature is 88.50%. It builds a solid foundation for the subsequent CCM) using the original SENet feature. A relatively complex decision boundary of the traditional classifier is needed to address this problem. However, owing to the lack of high-quality images with definite sentiment semantics, overfitting may be prone to occur. So, a single image feature is insufficient for effective image sentiment analysis. As shown in Figure 7(d), the corresponding sample distribution of the CCA feature is worse than that of any SENet feature. This will bring a challenge to the traditional classifiers (in Table 2, the corresponding accuracy of the CCA feature is 80.08%). Similarly, heavy chaos can also be observed in Figure 7(e). But the result is better than Figure 7(d). As shown in Table 2, the corresponding accuracy of the DCA feature is 87.59%. Unlike other methods (CCA and DCA), as shown in Figure 7(f), the corresponding sample distribution of CMS is better than that of a single SENet feature. The proposed CCM strategy can increase the corresponding distance among different classes. Hence, a relatively simple decision boundary of the traditional classifier is needed to complete classification (in Table 3, the corresponding accuracy of our model is 89.90%). Summarily, the above visualization results intuitively validate the effectiveness of the proposed CCM strategy.


## 4.4.1. Feature Selection

In this section, we need to choose effective but robust image features for CCM. We selected SIFT [45], GIST [46], LBP [47], DenseNet, VGG [39], ResNet, SENet [33], and SENetXT as our candidate features. We implemented principal component analysis on each traditional feature. Thus, each traditional feature was reduced to 500. For the deep learning features, we chose the bottleneck features of the corresponding CNN model. For example, we used the 16th and 19th layers of the VGG model in our experiments. Owing to a large amount of valuable distributed information, we did not implement feature reduction on these deep learning features. Detailed experimental results are shown in Table 5. “Dim” means dimension.Table 5
Classification performance of each single image feature. Unit: %.
DatasetsFeatureAccuracyDimFeatureAccuracyDimTwitter ISIFT83.02500VGG1682.264096GIST69.43500VGG1983.774096LBP67.55500SENet10187.802048DenseNet12167.171024SENet15288.502048DenseNet16165.662208SENet5084.322048ResNet5070.192048SENetXT10187.112048ResNet15269.062048SENetXT5089.202048FISIFT55.47500VGG1671.184096GIST50.51500VGG1970.994096LBP46.85500SENet10178.322048DenseNet12129.131024SENet15278.212048DenseNet16142.292208SENet5077.242048ResNet5057.442048SENetXT10179.462048ResNet15258.722048SENetXT5077.982048As shown in Table5, the five SENet features outperform any other feature. In particular, the two SENetXT features obtain the best trade-off between accuracy and efficiency. As we know, the SENet features pay more attention to those local regions with strong sentiment tendencies, including Ferris wheel, smiling face, shabby house, and so on. These sentimental signals can accurately depict the visual content and create a powerful foundation for CCM. In contrast, SIFT mainly describes the key gradient variations in images. However, the gradient variations between background and foreground sometimes are not obvious. GIST and LBP mainly describe the global and local textures in images. Texture only represents a little part of emotion, which is weaker than other features. Summarily, SENet is the best choice for our model. Meanwhile, compared with other features, SENet requires fewer resources and is efficient for extraction. Hence, we chose the five SENet features to complete CCM.


## 4.4.2. Classifier Tuning for Fine-Grained Sample Refinement

As described above, to avoid selective bias, we employed several heterogeneous classifiers to complete ASR. Therefore, classifier type and classifier number should be appropriately tuned to obtain the best performance. Here, we exhibit the corresponding tuning procedure for fine-grained sample refinement. For the Twitter I dataset, we chose LR and k-nearest neighbor (KNN) algorithms to perform fine-grained sample refinement. For the FI dataset, we used CatBoost, AdaBoost, random forest (RF), and LR to complete the same procedure. The refined samples of each classifier were used to perform the union operation for augmenting the existing benchmark datasets. The corresponding experimental results are shown in Figure5. As analyzed above, the DCA feature has a lower dimension than the proposed CCM. As shown in Table 2, the DCA feature obtains very competitive accuracy. Hence, to save resources, we used the DCA model to fuse the original features, and then, we performed classification using CatBoost.Figure 5
Classifier tuning for fine-grained sample refinement. (a) Twitter I. (b) FI.
(a)(b)As shown in Figure5, using multiple heterogeneous but complementary classifiers can prevent selective bias to a certain degree. More detailed, using multiple classifiers to implement ASR obtains a 1.38% performance improvement compared to using LR on the Twitter I dataset. For the FI dataset, compared with a single classifier, using multiple classifiers obtains the corresponding performance improvements of 1.99%, 1.37%, and 0.64%, respectively. In Figure 5(a), the best accuracy on the testing dataset is obtained when all the classifiers are used. Owing to the lack of (or imbalance of image samples), slight overfitting is observed on the Twitter I dataset. Therefore, training relatively small datasets is still challenging. In the future, we plan to introduce the well-known zero-shot learning method [48] to address this problem. In Figure 5(b), our model performs better when all the classifiers are used. The complementarity among multiple heterogeneous classifiers is fully utilized to boost the final performance. Summarily, we achieved the optimal classifier combination based on the corresponding accuracy and fitting status of the trained model.


## 4.4.3. Parameters Tuning of Ensemble Learning

Based on the classification results of different classifiers, we employed the ensemble learning strategy to further improve the final performance. Different ensemble learning strategies (hard voting or soft voting) combined with different classifier numbers result in different results. Hence, we should carefully tune the corresponding ensemble learning strategy combined with the top two (or three) classifiers to obtain the best performance. To obtain statistical interpretations, we used ten CMSs with different ensemble learning strategies to complete the tuning procedure. Figure6 exhibits the corresponding experimental results.Figure 6
Parameters tuning of ensemble learning. (a) Twitter I. (b) FI.
(a)(b)As shown in Figure6, the top three classifiers combined with the hard voting strategy perform best on any benchmark dataset. The top three classifiers play a decisive role in the final image sentiment prediction. The implicit complementarity among multiple heterogeneous classifiers is fully mined by the ensemble learning strategy to improve the final performance. In addition, the hard voting strategy is fairer than the weighted soft voting strategy. Hence, we used this setting to complete our experiments.


## 4.4.4. Visualization Results of Different Features

To intuitively demonstrate the effectiveness of the CCM strategy, in this section, we employ the well-known t-SNE [49] tool to visualize the sample distribution of each kind of CCM. Figure 7 exhibits the corresponding visualization results of different image features. Owing to space limitations, only the results of the Twitter I dataset are provided here. Similar experimental results can be observed on the FI dataset.Figure 7
Visualization results of different features. (a) SENetXT50. (b) SENet 152. (c) SENet101. (d) CCA. (e) DCA. (f) CMS.
(a)(b)(c)(d)(e)(f)In Figures7(a)∼7(c), the corresponding sample distributions of the original SENet features show slight chaos. The aggregation degree of the image samples in the same class is low while that in different classes mixes up each other. Hence, it is difficult for a traditional classifier to fit the sample distribution (please refer to Table 5, the corresponding accuracy of the SENet152 feature is 88.50%. It builds a solid foundation for the subsequent CCM) using the original SENet feature. A relatively complex decision boundary of the traditional classifier is needed to address this problem. However, owing to the lack of high-quality images with definite sentiment semantics, overfitting may be prone to occur. So, a single image feature is insufficient for effective image sentiment analysis. As shown in Figure 7(d), the corresponding sample distribution of the CCA feature is worse than that of any SENet feature. This will bring a challenge to the traditional classifiers (in Table 2, the corresponding accuracy of the CCA feature is 80.08%). Similarly, heavy chaos can also be observed in Figure 7(e). But the result is better than Figure 7(d). As shown in Table 2, the corresponding accuracy of the DCA feature is 87.59%. Unlike other methods (CCA and DCA), as shown in Figure 7(f), the corresponding sample distribution of CMS is better than that of a single SENet feature. The proposed CCM strategy can increase the corresponding distance among different classes. Hence, a relatively simple decision boundary of the traditional classifier is needed to complete classification (in Table 3, the corresponding accuracy of our model is 89.90%). Summarily, the above visualization results intuitively validate the effectiveness of the proposed CCM strategy.


## 4.5. Empirical Analysis

We selected some open images from WeChat and microblog to prove the practicality of our model. We performed our model (the coarse-grained sentiment classification model) on these images. Experimental results are shown in Table6. As shown in Table 6, we classified these images into two categories (positive and negative), and the smiling face usually represents positive sentiment tendency (), while the depressed face usually represents negative sentiment tendency ().Table 6
Evaluation on WeChat and microblog.
Web imageTrue labelOur predictionThe first and second images have a riot of color, whereas the third and fourth images are generally in dark color. Moreover, the first image has a smiling face, but the third image has a sad face. Our model performs well in predicting the sentiment polarity of the first to the third images. This firmly demonstrates the practicality of the proposed model. Certainly, our model makes a prediction bias on the fourth image. Negative images are far fewer than positive images in the Twitter I dataset. Hence, we speculate that the prediction bias is mainly due to the lack of negative images. This further illustrates the importance of the data augmentation operation from another perspective. Summarily, besides the publicly available datasets, our model also performs well on the open images, demonstrating its high practicality.


## 5. Conclusions and Future Work

We presented a novel model for image sentiment analysis with the newly designed ASR and CCM strategies. We demonstrated its effectiveness and robustness on two benchmark datasets. Incorporating the ASR strategy into our model helps augment the original benchmark datasets, whereas CCM, which encourages robust and effective feature learning, has the potential to improve the final performance. More importantly, the ASR strategy is a useful supplement to the current data augmentation method. It can be seamlessly incorporated into the state-of-the-art baselines to achieve better performance.However, there are still two limitations in our method. First, our method is based on the traditional machine learning idea. Although it has achieved very competitive performance, it is not end-to-end, which may limit its practicality. Second, the performance on the Twitter I dataset needs to be further improved in our future research.In the future, we plan to use the state-of-the-art attention mechanism [16] or object detection systems [50–52] to retrieve the corresponding affective regions. Then, we will combine the whole image and the affective regions to obtain better performance and more powerful interpretability. Second, we plan to fuse the proposed ASR strategy with other popular data augmentation methods, such as DADA [31] and AutoAugment [32]. We hope to obtain more and more high-quality images with definite sentiment semantics. Finally, we will employ the well-known deep mutual learning [53] method to mine much more sentimental knowledge between heterogeneous neural networks for improving the final accuracy.



---

*Source: 2477605-2022-03-24.xml*