To Err Like Human: Affective Bias-Inspired Measures for Visual Emotion Recognition Evaluation

Authors: chenxi  zhao
jinglei  shi
liqiang  nie
jufeng  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 06:17:16

Abstract: Accuracy is a commonly adopted performance metric in various classification tasks, which measures the proportion of correctly classified samples among all samples. It assumes equal importance for all classes, hence equal severity for misclassifications. However, in the task of emotional classification, due to the psychological similarities between emotions, misclassifying a certain emotion into one class may be more severe than another, e.g., misclassifying 'excitement' as 'anger' apparently is more severe than as 'awe'. Albeit high meaningful for many applications, metrics capable of measuring these cases of misclassifications in visual emotion recognition tasks have yet to be explored. In this paper, based on Mikel's emotion wheel from psychology, we propose a novel approach for evaluating the performance in visual emotion recognition, which takes into account the distance on the emotion wheel between different emotions to mimic the psychological nuances of emotions. Experimental results in semi-supervised learning on emotion recognition and user study have shown that our proposed metrics is more effective than the accuracy to assess the performance and conforms to the cognitive laws of human emotions. The code is available at https://github.com/ZhaoChenxi-nku/ECC.

Main paper: 
 introduction: 
 "The best and most beautiful things in the world cannot be seen or even touched. They must be felt with the heart."-Helen KellerEmotion, as a complex state of feeling that results in physical and psychological changes and influences thoughts and behavior, involves subjective experiences, physiological arousal, cognitive appraisal and behavioral expressions [33] . Emotional similarity is a significant characteristic of emotions, revealing the commonalities among diverse emotional states in their essence, modes of expression, and influencing factors. And the essence of emotion can be explored through understanding the hierarchical nature of human cognitive processes. The Reverse Hierarchy Theory [14] in neuroscience suggests that, instead of exhibiting absolute object classification ability, humans first recognize coarse-grained categories and then proceed to identify finer-grained detailed information [21, 22] . This process involves comparing with prior information to make a comparative classification [10] . Under the same cognitive mechanism, both emotion recognition and object recognition follow a similar pattern of recognition [24] , i.e., progressing from global to local, and from coarse-grained to fine-grained identification processes. And such emotional cognitive process is often characterized by the term Emotion Granularity in psychology [31, 18] , where it involves the degree of nuance with which individuals can perceive. It also implies that there exists relative proximity between different emotions. For instance, belonging to positive emotions, the similarity between excitement and awe is greater than that between excitement and anger. This proximity is often associated with a misclassification-similar phenomenon in psychology referred to as 'affective biased attention', which is the tendency to pay more attention to some emotions than others. In complex psychological conditions, people tend to show negative bias compared to positive information, that is, they tend to use more negative information [36] . Though the emotion similarity and affective bias have been considered when evaluating cognitive ability for humans in psychology, the measurement of such ability for emotion recognition methods [19, 8, 25] in computer vision field still remains within the framework of the ordinary classification task. They primarily focus on employing global representation of objects [19] , language prompts [8] or a multi-stage perception (entity, attribute and emotion) model [25] to improve the number of correctly classified samples, but totally ignore the influence (or severity) of different misclassifications caused by the similarity between emotions. As shown in Fig. 1 (a), the relative distances between emotions are different. And after testing with the recent representative emotion classification methods in terms of both accuracy and severity of misclassification in Fig. 1 (b), we observed that, although the classification accuracy has been improved to a high level, the severity of misclassification stayed at the same level or even worse. Therefore, metrics that aligns more closely with psychological models by taking the emotional similarity and the severity of misclassification into account are necessary for the visual emotion recognition task.Studies on the misclassification for other tasks can be traced back to the early 'cost-sensitive' problem, where researchers employ the cost matrices to assess and enhance classifiers, but they care more about the class imbalance in datasets rather than the costs led by misclassification. In the field of object classification, some researchers began to pay attention to the problem of misclassification and devised methods to measure and reduce mistake severity based on the hierarchical information of labels [1, 12, 4] . For instance, Bertinetto et al. [1] devised the hierarchical distance of a mistake and average hierarchical distance of top-K to quantify the severity of mistakes, and proposed the hierarchical cross-entropy loss to reduce the severity of mistakes. Garg et al. [12] proposed to learn a hierarchy-aware feature space to explicitly learn the hierarchy of labels during the training phase. All these works are based on the cost definition composed of semantic hierarchical information and lowest common ancestor, which can be traced back to WordNet [11] .However, directly applying the experience from hierarchy-based severity learning to the visual emotion recognition task is challenging, because the multi-hierarchy relationships in object categories do not exist in emotional categories, thus we can not utilize hierarchy information to define the severity of misclassification. And we could not adopt additional classifier to capture the structure available in the label space [12, 3] . While this severity is essential for designing a robust measure and effective loss function for the visual recognition task. And prior works have pointed out that making the classifier to learn both ground truth information and label structure will degrade the classification accuracy [12] .To address the above-mentioned issue, we define the concept of emotional distance (or cost) for misclassification based on the Mikel's wheel [23, 57] , and further propose novel metrics for evalu-ating the performance of emotion classification methods using this concept as a foundation. More specifically, we first define the cost matrix for different misclassification cases according to the definition of the distance in [57] and of the affective polarity in Mikel's Wheel. On the basis of the confusion matrix between the cost matrix and the classification results, we then propose a new visual emotion recognition evaluation measure ECC to measure all possible classification results, as well as a measure EMC focusing more on the cases of misclassification. We verify the effectiveness of the measures via the semi-supervised emotion recognition task, demonstrating that our metrics provide a more robust assessment of method performance than accuracy alone. And we prove through user study that the design of our metrics are in line with human cognition. As far as we know, both EMC and ECC are the first metrics that evaluate the performance of visual emotion classifiction algorithms by taking the influence of misclassifiction into consideration.Our main contributions can be summarized as follows: 1). We are the first to introduce the concept of mistake cost into visual emotion recognition, and propose new measures based on Mikel's wheel to better assess the performance of emotion classification methods. 2). In semi-supervised emotion classification tasks, from the perspectives of threshold adjustment and model selection, on one hand, compared to complex confidence threshold adjustment mechanisms, our measures can be more simply yet effectively applied in the pseudo-labeling process, enhancing the model's classification performance. On the other hand, we demonstrate that our metrics can help select the model capable of generating higher-quality pseudo-labels that are beneficial for training. 3). Finally, we discussed the relationship between our metrics and ACC 2 , and we further validate our metrics through user study, confirming that they provide results consistent with human emotional cognition in evaluating the severity of sample misclassification by the model.2 Related Work 
visual emotion recognition: 
 With the advent of deep learning, many methods [60, 30, 53, 38, 49, 29, 46, 42, 56, 54, 55] have begun to use convolutional neural network (CNN) for visual emotion recognition. Many of these researchers have explored the relationship between emotions. Based on Plutchik's [28] theory that different emotions have different similarities. Zhao et al. [57] proposed the emotional distance for the first time based on Mikel's Wheel, which is used to calculate the emotional similarity between two pairs of pictures. In addition, Yang et al. [46] designed affective polarity on Mikel's wheel and proposed a new hierarchical cross-entropy loss to distinguish between difficult and easy cases in a specific emotional way. Inspired by the large-scale pretrained language models, Deng et al. [8] propose a fine-tuning strategy based on prompts. Pan et al. [25] generate pseudo labels through visual language models as auxiliary guidance for multi-stage visual perception. However, due to the ambiguity and subjectivity of emotions, a single image often elicits multiple emotional responses, when dealing with visual emotion recognition tasks, it is more reasonable to use label distribution than single label classification. In [48] , they generate sentiment distributions from a single emotion dataset based on emotional distance to solve this problem. Moveover, inspired by the inherent relationship between emotions in psychology. Yang et al. [45] propose a well-based circular structure representation to use prior knowledge to learn visual emotion distribution. However, these efforts mainly focused on improving accuracy, ignoring the fact that the severity of misclassification is not the same for different classification results, and at the same time, the constraints of fitting the label distribution may be strict, which often have the opposite effect.The importance of studying misclassification has attracted a lot of attention in the era of machine learning, but it has been neglected in the era of deep learning. In the field of machine learning, Wei et al. [40] proposed the measure of confusion entropy to evaluate the performance of classifiers, which utilizes the distribution information of misclassified categories for all categories. This problem is also described as 'cost-sensitive' by introducing the cost imbalance between different misclassifications in real-world applications, and providing solutions that meet practical needs. Classic problems include bank lending issues, disease diagnosis issues [9, 51, 4] , etc. The most common cost-sensitive solution is rescaling, which mainly preprocesses the training set to improve the sensitivity to classification results. Specifically, Turney [35] studies how to choose the correct cost assignment in cost-sensitive classification problems, and further explores the meaning and impact of cost values. Metacost [9] calculates the ideal class for each training sample by estimating the posterior probability density of the training samples and modifies the class of the original training sample to change the new data set. Different from the previous approaches, Zhou et al. [59] pointed out that in addition to the correction of the classification algorithm and dataset, the cost matrix needs to be corrected, which solves the problem that it is often ineffective for multi-class cost-sensitive learning. However, this kind of problem focuses more on the class imbalance of the dataset, and gradually loses interest in the cost of misclassification.In recent years, a small part of the work has begun to focus on the problem of the mistake severity and devised methods to reduce it. Deng et al. [6] pointed out that classification can be improved by using semantic hierarchical information from WordNet [11] , which laid the foundation for future research. Furthermore, Bertinetto et al. [1] summarized and analyzed the issue of mistake severity, and proposed hierarchical cross-entropy loss to reduce the severity of mistakes. They also introduced two measures, hierarchical distance of a mistake and average hierarchical distance of top-K, to quantify the severity of misclassification. However, reducing the severity of misclassification is premised on reducing accuracy. Karthik et al. [17] use Conditional Risk Minimization (CRM) to improve this shortcoming, reducing the severity of mistakes without compromising accuracy. But CRM doesn't change the model, it's about making the best choice of the moment during the testing phase. In order to solve this problem elegantly, Garg et al. [12] propose to learn a Hierarchy Aware Feature(HAF) space to explicitly learn the hierarchy of labels during the training phase. In [15] , they train two separate models for coarse-grained and fine-grained, and make the final prediction by calculating the normalized scores of the two models in the reasoning process. Although some progress have been made in these studies in the direction of hierarchical labeling, there is still a lack of research in visual emotion recognition. Different from they use lowest common ancestor (LCA) measure to assess the severity of mistakes, we define our measures based on Mikel's wheel and confusion matrix, and demonstrate their effectiveness.3 Misclassification-Aware Measure Design 
definition of emotional distance: 
 In almost classification tasks, accuracy serves as an important evaluation metric, measuring the model's capacity of predicting categories. However, this measure is built on a binary philosophy, i.e., only considering whether or not the predicted category matches the true label. In fact, distinguishing between different types of mismatch is meaningful, particularly in the field of emotion classification. For example, due to the similarity between emotions, misclassifying 'excitement' as 'awe' may be acceptable in certain diagnose, while misclassifying it as 'sadness' could lead to severe diagnostic errors in clinic psychology. Based on this reasoning, we therefore define emotional distances to characterize their relative relationships, and further to aid in quantifying the degree of severity of misclassification. Existing psychological models such as CES and DES, either model emotions as completely independent categories or represent them as multidimensional continuous vectors. The former ignores the similarity between emotions, while the latter requires precise measurement of emotions by experts, which can be hard to achieve for visual emotion recognition task. In [57] , authors provide a definition between emotional labels based on Mikel's wheel, where the paired emotional distance is 1+ 'wheel distance'. The 'wheel distance' means the number of steps when moving from one category to another on the wheel like shown in Fig. 2 (a). However, this definition of emotion distance neglects the emotional polarity [47] , e.g., the distance between 'fear' and 'excitement' being the same as the distance between 'fear' and 'sadness'. But the distance between categories with opposite polarity should be greater than those with the same polarity, as shown in Fig. 2 (b ). We thus define the emotional distance as follows:EQUATIONwhere dist (e i , e j ) represents the number of steps on the Mikel's wheel, C is a constant to adapt the importance of polarity, and a larger C means misclassifying one emotion into the opposite polarity is more severe, and we define C as 4 in order to ensure that the emotional distance within the same polarity is always less than the emotional distance between different polarities. C p represents the classes that have the same polarity. We can finally obtain a symmetric cost matrix with each element being the distance between the corresponding emotions. Although our emotional distance is defined based on Mikel's wheel for classifying eight emotions, it is important to note that when calculating the classification of six emotions based on Ekman's model, we can still use a similar method for determining emotional distances. To achieve it, we can initially follow the definition used by Mikel's wheel, calculating the emotional distances for the same emotions present in both models. Subsequently, we can adjust the distances between opposite emotion labels based on emotional polarity. Due to the inherent nature of the distance definition in Mikel's Wheel, our main focus here is on the issue of classifying the eight categories of emotions.Wei et al. [40] used the category distribution information of all categories of misclassification to propose the concept of confusion entropy to measure the standard. Inspired by them, we design our measures based on a confusion matrix. As shown in Fig. 2 (c), the confusion matrix is defined as a N×N matrix, where N represents the number of classes. The rows of the confusion matrix represent the true labels, while the columns represent the predicted ones. Each element in S i,j represents the number of samples classified from the correct category i to the category j. The diagonal elements of the confusion matrix represents the number of samples correctly classified for each class. Hence the sum of diagonal elements of confusion matrix N c divided by the total number of samples N represents ACC. While other elements represent the number of samples belonging to the category i that are wrongly classified into category j. The drawback of accuracy lies in its exclusive consideration of correct classifications along the diagonal of the confusion matrix, while disregarding equally important misclassifications elsewhere in the matrix. This implies that accuracy may be misleadingly high in scenarios of class imbalance. Although directly using the confusion matrix as a metric allows for the consideration of both correct and incorrect classifications (misclassifications), this metric overlooks the distance between emotions in psychology. To take both correct classifications, misclassifications and emotional distance into evaluation, we propose to use emotional distance to modulate the confusion matrix. Based on this design philosophy, the ACC can be re-formulated as the product of the confusion matrix and a modulation factor M i,j as:EQUATIONwhere c denotes the number of all classes, N c represents the count of correct classifications. It implies that one sample is counted as 1 only if it is correctly classified, other misclassified samples are not distinguished and are all recorded as 0. However, as we explained previously, misclassifications can be acceptable to some extent when the emotional classes are similar. Thus we tackle these samples as 'quasi correctly classified samples' and weighted with a value between (0, 1). The smaller the similarity of emotions, the less acceptable the misclassified sample becomes, hence the smaller the corresponding value it will have, and vice versa. To achieve this, we rely on the reciprocal of the emotional distance W i,j defined in Eq. 1 to replace the modulation factor M i,j in Eq. 2 for the misclassified samples to obtain ECC in Eq. 3.EQUATIONIn this way, misclassified samples are no longer simply neglected like in ACC, but have weights in terms of emotional distances, which also means that our measure ECC make use of the entire confusion matrix, including elements both inside and outside the diagonal as shown in Fig. 2(c ).In some practical scenarios, people pay more attention to cases of misclassification, like estimating the severity of misdiagnosis of a certain mental diease, while both ACC and ECC include the cases of correct classification, hence failing to provide information about misclassification. Therefore, in order to only consider the cases of misclassification, one can extract the term (EQUATIONWi,j )/N that excludes ACC from Eq. 3 as an indicator for evaluating misclassification. However, directly using this terms as a misclassification measure is inappropriate for two reasons: 1). The denominator of this term is N, indicating that its value is still influenced by correct classified samples, rather than solely considering the misclassified samples. 2). The maximal value of this term is 0.5, which we hope to be 1 as ACC and ECC. We accordingly modify this term and propose a novel measure for misclassification description in Eq. 4, named Emotional Misclassification Confidence (EMC):EQUATIONThis metric considers only misclassifed samples S i,j , i ̸ = j and their number N -N c , meanwhile modifying the modulation factor from 1 Wi,j to 1 Wi,j -1 to ensure a maximum value of 1. In this way, we have an ECC that measures the overall classification results, as well as EMC that is specially designed to measure the severity of misclassification. The cooperation of two measures can better evaluate the visual emotion recognition task.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: the paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
datasets: 
 We evaluate our metrics on two widely applied datasets EmoSet [44] and FI [50] . EmoSet is based on Mikel's eight-categorical sentiment model, which uses 810 keywords and collects from four different sources, including openverse, pexels, pixabay and rawpixels. It covers different emotional attributes, i.e., low-level (brightness and colorfulness), mid-level (scene type and object class), and high-level (facial expression and human action). Finally, 60 annotators who passed the test annotated a total of 118,102 images. The FI dataset was collected from Flickr and Instagram through eight sentiment keywords, and was built based on Mikel's eight-category sentiment model, which contains about 23,308 images.Because of the ambiguity of emotions [16] , annotating high-quality and large-scale datasets for visual emotion recognition is arduous and challenging. Semi-supervised learning is an effective solution which consists of training the model based on both labeled and unlabeled data, then annotating unlabled examples by this trained model. While for pesudo labeling based semi-supervised learning methods, selecting appropriate models and labeling confidence thresholds play key roles in determining the final performance. In this section, we carry out experiments on two perspectives: the model selection and the adjustment of labeling threshold, to demonstrate how our proposed measures can benefit this task.Table 1 : We conduct experiments on the datasets FI [50] and EmoSet [44] , and evaluate the experimental results using ACC. It is considered fair to compare our method in 4.2.1 with FixMatch and FlexMatch, given that all of them employ threshold adjustment methodologies.Where 'TA' means threshold adjustment.Based on S 2 -VER, we compared our method in 4.2.2 with all the state-of-the-art semi-supervised methods. For pseudo label-based semi-supervised learning methods, a confidence threshold is used to determine whether current labeled samples are filtered will directly affect the number of training samples and the proportion of correct labels in each epoch: a high threshold will excessively filter out high-quality unlabeled data, leading to insufficient training data, while a low threshold will allow more low-quality unlabeled data, making the model converge to a poor local minima. How to select an appropriate threshold of pseudo-labeling in the training process has always been a hotspot for semi-supervised learning [5] . Existing methods either use an intuitive constant threshold [32] that is widely adopted in other semi-supervised learning tasks, or a dynamic threshold that varies in terms of the estimated learning status [52] . Considering that EMC can measure the severity of misclassification, we therefore recommend using EMC to dynamically adjust the confidence threshold before each epoch. This is similar to how, in a clinical scenario, a patient not only refers to the doctor's current confidence in the diagnosis but also takes into account the doctor's historical reputation (frequency of misdiagnoses and medical errors) to ultimately judge the reliability of the current diagnosis. For semi-supervised emotion recognition tasks, since there are only a few labeled samples available for training, the model can easily overfit these samples, resulting in all correct predictions and thus making it impossible to calculate EMC. To address this situation without introducing additional training data or increasing computational payload, we treat the pseudo-labels generated from weakly augmented unlabeled samples as the ground truth, and use the predicted labels from strongly augmented samples as the model's predictions to calculate EMC. Since the labels of the same sample should remain consistent under different data augmentation methods, a large EMC indicates that the model provide similar emotional predictions with different augment methods. This suggests that the model has grasped the underlying visual elements that represent emotions in images, which reflects the reliability and high quality of the pseudo labels. Therefore, we can lower the confidence threshold, allowing more pseudo-labeled samples to participate in the training process. On the contrary, when the EMC is small, we can raise the threshold and filter out the low-quality samples. We can realize the above confidence adjustment mechanism by simply setting:EQUATIONwhere τ is the pre-defined threshold, τ ′ t is the new threshold at time step t, EMC t represents the EMC at time step t, e is a constant for different datasets. We define τ as 0.95. For FI, we define e as 0.5, and for EmoSet, we define e as 0.4. And we set the upper and lower bounds of τ ′ t to 0.98 and 0.7 [16] respectively to ensure stability of the training process.To test the effectiveness of our proposed confidence adjustment mechanism built on EMC, we compared it with two representative pesudo labeling based methods, Fixmatch [32] and Flexmatch [52] , where Fixmatch directly provides a fixed threshold, while Flexmatch assigns dynamic thresholds to each class according to corresponding learning status. Experimental results in Tab. 1 show that, thanks to our EMC-based threshold mechanism, our method is over 1% better than Flexmatch and Fixmatch. Additionally, let us note that unlike the method Flexmatch that adjusts the threshold for every category, we evaluate the entire ublabeled data based on a statistical perspective, and adjust the overall threshold after measuring the EMC, which is computationally low-cost. And the experiment is followed by [32] .Existing methods that typically use ACC to measure the model's capability of discriminating between categories of unlabeled samples, i.e., models with higher ACC can provide more reliable pesudo label. However, such models often suffer from the problem of confirmation bias [34] , where the model will gradually deepen this error during the learning process. Accumulation of these errors eventually leads to the final model being unable to achieve good classification performances. As both ECC and EMC are designed based on the consideration of the cases of misclassification, which means they can better distinguish the ambiguity of labels, and models selected in terms of ECC or EMC will less effected by the cumulative confirmation bias, providing high-quality labels than those based on ACC.To prove that model with higher ECC or EMC is better for pesudo labeling, we train the same network with different loss functions: cross-entropy loss L CE and the combination of cross-entropy loss and order-based loss ListMLE [41] as L c = L CE + αL ListMLE , where α is 1 and L ListMLE aims to constrain the final prediction probability of the samples to follow a preset order, thus favoring higher ECC and EMC (proved in Appendix A). This constraint will reduce the severity of misclassification, at the same time, the cost is to reduce ACC [1] , specific experiment is in Appendix B. More precisely, we first train the model with L CE , once it starts to converge (the model has preliminary ability of recognition), we then keep the same loss function or replace L CE with L c to make it continue to focus on the correct classification or focus more on the cases of misclassification. Therefore, models trained solely using L CE exhibit better ACC but limited capability to distinguish error samples, and the quality of the pseudo label is poor. Although the pseudo labels might still be incorrect compared to the ground truth when using a combined loss function for training the model, the pseudo-labels become closer to the ground truth. In such cases, these pseudo labels can still have a positive impact on the training process and thereby improve the model's accuracy. As we show in Tab. 1, we adopt the state-of-the-art method S 2 -VER [16] as our baseline, as it generates more reliable pseudo labels by Table 2 : We conducted experiments on the FI [50] and EmoSet [44] datasets using three backbones. As mentioned above, the loss function employed the commonly used the cross-entropy loss L CE and the combined loss L c . calculating the similarity between emotional prototypes and samples, but ignores the error of pseudo labels. We follow the experimental setting of [16] and vary the proportions of the labeled samples as 0.5%, 5% and 10% (corresponding to 80, 800, 1600 label number for FI, and 400, 4000, 8000 label number for EmoSet), it can be observed that under the different settings of the two datasets, our method performs favorably against S 2 -VER by 1% in accuracy. Meanwhile, our method also far surpasses multiple state-of-the-art methods in semi-supervised learning. It indicates that choosing a model with better misclassification ability (better ECC and EMC) can produce pseudo labels of better quality and beneficial to the training process, thus achieving better semi-supervised performance.ACC 2 is a very important binary classification metric in the field of emotion recognition [43, 8, 26, 44] , used to measure whether the classification to the same emotional polarity is correct. More specifically, when a sample labeled as 'excitement' is classified as 'awe', it is incorrect to use accuracy for evaluation. However, for metrics like ACC 2 , such a classification is considered correct. In a certain sense, metrics like ACC 2 , which involve coarse-grained classification, take into account the proximity of labels and consider misclassified cases. This approach aligns with the objectives of our measures. To further demonstrate the effectiveness of our metrics, As mentioned above, we conducted experiments using both the cross-entropy loss L CE and the combined loss L c . As shown in Tab. 2, although the accuracy of the combined loss is lower than that of the cross-entropy loss, its ACC 2 is higher. This indicates the shortcomings of using accuracy alone in certain situations, as it fails to measure for misclassifications. Although the ECC also decreased due to the influence of the ACC, since the ECC takes into account the situation of misclassification, the gap between the two models in terms of ECC is not significant. As EMC considers metrics for misclassification alone, the EMC of the combined loss is significantly higher than that of the cross-entropy loss. In this regard, the trend of EMC aligns with that of ACC 2 , which also demonstrates the correlation between the two metrics. In the confusion matrix, ACC 2 actually represents the proportion of correctly classified samples in the top-left and bottom-right sections. This also indicates that our measures are actually more refined measures that lies between Accuracy and ACC 2 .Since our metrics is founded on principles of human cognition, we aim to further demonstrate that our measures align with human judgments in emotion classification results via user study.In order to have models having different levels of ECC, we take ResNet50 as our network backbone and train it respectively with cross-entropy loss and combined loss L c = L CE + αL ListMLE on the FI dataset, where α is 0.2. Then we perform predictions on the test set and select the images that are misclassified by both two models into different classes. Finally we randomly select 50 eligible images, and filter out the images with no obvious emotion or ambiguous emotion, getting 30 carefully selected images as our tested images for user study.Preference Study We invite 30 participants having different social backgrounds to our user preference study, and the test for every participant lasts about 15 minutes. During the test session, each misclassified image will be presented to participants with three options: the incorrect class predicted from the model trained with cross-entropy loss, the incorrect class predicted from the model trained with the combined loss function, and a third 'Indistinct' option for cases where participants are unable to discern the emotion of the image. And participants will choose their preferred options after viewing each image.In all 900 collected votes, 487 votes are cast for the results produced by the model with higher ECC, while 242 votes opt for the results generated by the model with higher ACC. There are 171 votes that opted for unidentifiable choices. Fig. 3 further shows the distribution of the votes for each image, where we can observe that among 30 tested images, users preferred the classification results provided by the model with higher ECC over the model with higher ACC for 24 images, representing 80% of the total tested images.We want to further explore the rationality of the defined emotional distance and determine whether it can help models learn the semantic structure of labels. To answer the above question, we transform emotional distance into ranking, and designed three sets of experiments based on L c and α is 1. The specific experiments are shown in Table 3 . And the specific experimental settings are detailed in Appendix C. Since we are only focusing on the impact of emotional rank (emotional distance) on the model, and changing the rank of ground truth label would prevent the model from training on correct classification categories. So we randomly shuffle and reverse the rank of the other labels while keeping the category with the first position in the emotional rank as the ground truth. Then, according to the changed order, use ListMLE for training. As shown in Table 3 . Our rank achieve advanced performance in three measures. And the results are worse in reverse rank than random rank, and they are significantly worse than our rank. This shows that our rank is better, in line with human cognition of label rank, and our label rank will help the model learn the emotion category structure.Although the ranking of ground truth has not changed, a reasonable label ranking can often reflect the emotional and visual element relationship between images, which will enable the model to mine the visual and semantic correlation between similar categories, so as to learn a better label semantic structure. We have proven the rationality of emotional distance through above experiments. Since the new measures ECC and EMC are designed based on emotional distance, it also validates the rationality of our measures.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We clearly describe the process of constructing the measures and conducting the validation experiments. In addition, a complete set of code is provided in the supplementary material. 
conclusions: 
 In this work, we define the concept of misclassification in the field of visual emotion recognition, and propose new measures to evaluate the mistake severity in visual emotion recognition based on Mikel's Wheel distance. We define our emotional distance using the Mikel wheel and adopt it to build cost matrix, then exert it to confusion matrix to compute emotion confusion confidence (ECC) and emotional mistakes confidence (EMC). And we demonstrate that our measures are more robust in semi-supervised learning. Our measures can not only help to select the model that can produce highquality pseudo labels, but also can be used as a reference standard to adjust the threshold adaptively. Moreover, we verify that our new measures are consistent with human emotional cognition through user study. Finally, we verify the validity of our emotional distance. Figure 4 : In EmoSet, Eq. 14 is compared with the t-SNE graph [37] between alpha equals 0 and 1, where the left two pictures are the results of Resnet18 [13] , and the right two pictures are the results of Resnet50.In this way, we clearly explain the relationship between ACC and cross-entropy, and put forward the expectation of emotional label arrangement on the basis of emotional distance, so as to transform ACC and cross-entropy into ECC and ListMLE, that is, why ListMLE can be used as the backbone of ACC.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We submitted the complete code in the supplemental material and the code will be publicly available. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Justification: The details of all semi-supervised experiments follow S 2 -VER [16] and FixMatch [32] , and other experimental details are explained in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: As shown in Tab. 1, we conducted three sets of experiments each and reported their mean values and standard deviations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: There is no such risk in our paper. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We clearly marked the work of follow in the paper, and these works are all open source according to the rules. Where Fixmatch [32] is in https:// github.com/google-research/fixmatch and S 2 -VER [16] is in https://github. com/exped1230/S2-VER Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have released our new measures ECC and EMC, and provided the code in the supplementary material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: In the paper, we explain the specific way of user study in subsection 4.4. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Here we further explore the performance of ListMLE in visual emotion recognition tasks. The ListMLE loss function is designed to constrain the final prediction probability of the sample to conform to a given rank arrangement. According to the emotional distance in Eq. 1 defined in the measure, we convert it into the label order to cooperate with ListMLE for training. For example, if a image is labeled excitement, the defined labels are sorted from front to back as excitement,awe,contentment,amusement,fear,sadness,disgust and anger. We mix ListMLE with cross-entropy loss, where α is a hyperparameter to adjust the proportion of ListMLE.EQUATIONWe compare ListMLE and cross-entropy loss on three backbone. The details are in Tab. 4. Similar to the conclusion of previous work [1] , when focusing on the results of misclassification, accuracy will be reduced to some extent. When alpha is 1.0, because accuracy decreases more, ECC is worse than only using cross-entropy loss, but EMC performs best. When alpha is equal to 0.2, a better trade-off is reached between ACC and EMC, and ECC reaches the highest. In order to further emphasize the difference between ListMLE and cross-entropy, we present the visual comparison diagram in Fig. 4 . It can be seen that the visualization result of ListMLE has a better clustering effect, and the relationship between categories is easier to distinguish, and it is more consistent with our defined emotional distance, indicating that ListMLE has learned the label structure information of emotion. Furthermore, this indicates a high correlation between our measures and tasks involving clustering, such as emotional image retrieval.should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: After the rebuttal period, the paper received three ratings of borderline accept and one of borderline reject.

The paper proposes to evaluate the performance in visual emotion recognition by considering the distance between different emotions on Mikel’s emotion wheel. Experimental results are conducted in semi-supervised learning on emotion recognition. The reviewers overall agree the approach is novel and innovative. Strengths of the paper include aiding in model selection and threshold adjustment with semi-supervised learning, validation through user studies, improved performance over baselines, and the novelty of new measure is good.

The original concerns, raised by the reviewers, include too few comparisons, confusing description of the evaluation and annotation, applicability of Mikel’s wheel to Ekman’s six basic emotions, multiple emotions occurring at once not being considered, separation of related emotions, and no fully supervised learning experiments.

The authors and reviewers took part in a good discussion during the rebuttal phase, with the authors providing significant clarification and new experimental results resulting in multiple reviewers increasing their scores. After the rebuttal phase, the paper strengths outweigh weaknesses as many of them were addressed. This paper can be accepted for publication.

Reviews: review: paper_summary: paper_summary: This paper proposed new measures to evaluate the severity of misclassifications in visual emotion recognition. It addresses the limitations of traditional accuracy metrics by considering the psychological similarities between emotions. Utilizing Mikel's emotion wheel, the authors define an emotional distance metric and apply it to create a cost matrix for emotion classification. They introduce the Emotion Confusion Confidence (ECC) and Emotional Mistakes Confidence (EMC) metrics and validate their effectiveness through experimental results and user studies, demonstrating their robustness in semi-supervised learning contexts.
strength_weakness: strength_weakness: 1. The proposed Emotion Confusion Confidence (ECC) and Emotional Mistakes Confidence (EMC) metrics offer limited contribution as new evaluation standards to the community. Moreover, the exploration of these new metrics and their application in the manuscript is insufficient, resulting in a lack of depth and richness in the content.

2. The manuscript includes too few comparative methods, and the comparison forms and application richness are insufficient, which limits its credibility. In the User Study, only the analysis of ECC is included, with no analysis of EMC.
questions: questions: 1. The authors frequently mention the example: “misclassifying ‘excitement’ as ‘anger’ apparently is more severe than as ‘awe’.” How many methods currently misclassify this category of emotion as anger? Is there corresponding data to support this claim?

2. To fully demonstrate the effectiveness of the proposed new metrics, should these metrics be used with a more diverse set of methods? Additionally, the mainstream methods should be re-evaluated using the proposed metrics and compared with the accuracy (Acc) metric. Analyzing the strengths and weaknesses of methods from multiple dimensions could illustrate the effectiveness and advantages of the proposed evaluation metrics.
limitations: limitations: The paper acknowledges several limitations, including the use of a simple and direct method for threshold adjustment in semi-supervised learning. While the proposed measures offer a novel perspective, further investigation is needed to refine these methods. Additionally, the application of the new measures should be explored in other fields, such as pre-training models and large language models, to fully validate their effectiveness.
paper_summary: paper_summary: The paper proposes a new measure for emotion recognition performance based on Mikel’s emotion wheel. The measure takes the distance between emotions into account. Experiments in semi-supervised learning on emotion recognition and user study demonstrate the effectiveness and superiority of the proposed metrics over accuracy.
strength_weakness: strength_weakness: 1. Few baselines are compared. Tables 1 and 2 are the main experiments while only one and two baselines are discussed and compared.
2. Section 4.2.1, the setting of pseudo labeling evaluation is confusing to me. How did you get the accuracy for models with different label num? Is there an official test set in FI/EmoSet for evaluation? Or did you use the whole set? If that's the case, what’s the difference between pseudo labeling evaluation and simple model training/testing evaluation?
3. Section 4.3 is also unclear to me. Can you clarify the annotation instruction in the user study? I am not sure how participants vote for models with higher ECC or ACC.
4. Mikel’s emotion distance is not applicable to Ekman six basic emotions which is widely used in most categorical emotion datasets.
questions: questions: 1. Line 304-307, how do you select the values for different hyperparameters?
2. How do you measure the standard deviations in tables 1 and 2?
limitations: limitations: Please refer to weaknesses
paper_summary: paper_summary: This paper proposes a novel evaluation approach based on Mikel’s emotion wheel from psychology, which considers the "emotional distance" between different emotions. It is claimed that the measure design considering the granularity of the emotions can be a better metric to evaluate visual emotion recognition. Experimental results adopting some of the semi-supervised learning methods and user studies show that the proposed metric is more effective than accuracy in evaluating performance and aligns with human emotional cognition.
strength_weakness: strength_weakness: -The use of Mikel's Wheel, a relatively less-studied emotion model in the context of building computational methods for automatic recognition, limits the study's impact. The first work utilizing this model dates back to 2016. Given the extensive body of work based on e.g., Ekman's model, the significance of this study is unclear.
- An important aspect completely overlooked in this paper is the frequent evaluation mechanism involving binary classification for each emotion class. Multiple emotions can be experienced simultaneously, making a multi-class classification task inappropriate. Consequently, accuracy (ACC) should be applied to individual emotion classes, where the positive class indicates the presence of a specific emotion and the negative class indicates its absence. Additionally, several methods have adopted the use of F1-macro and F1-weighted metrics, which were neither discussed nor applied in this study.
-Another issue that caught my attention is the limited range of researchers cited in the related work section regarding the use of Mikel's Wheel. Only the same group of researchers (e.g., Jingyuan Yang, Dongyu She) is cited, indicating that the impact of this work may be limited.
-The context is crucial in defining emotional distances, which seems to be missed in this study.
-References to CES and DES are missing (see Line 163).
-I expected to see more comprehensive comparisons in Tables 1 and 2. For example, comparisons could include works like "Li, J., Xiong, C., Hoi, S.C.: Comatch: Semi-supervised learning with contrastive graph regularization. In: ICCV (2021)." Currently, the comparisons are not as extensive as to confirm the claims.
- I do not find the venue for this submission appropriate. This work should be submitted to conferences focused on affective computing, such as ACII, where the community can discuss the theoretical underpinnings of the metric in conjunction with psychological claims and findings.
questions: questions: - Have you considered using emotional lexicons that include a broader range of emotion classes?
-How were the results of S2Ver on Emoset produced?
limitations: limitations: - Emotion and overall affective computing always have a social impact but it is not discussed in this paper.
- Limitations are discussed in the appendix, while can be improved.
paper_summary: paper_summary: The principal subject of this paper is the definition of the concept of misclassification in the field of visual emotion recognition, and the proposal of a novel evaluation method based on Mikel's Wheel distance to assess the degree of misclassification in methods of visual emotion classification. The paper also discusses the application of these evaluation methods in selecting models and adjusting label thresholds in semi-supervised learning tasks.
strength_weakness: strength_weakness: 1. In the related work section, subsections 2.2 and 2.3 appear to be erroneously treated as separate entities, whereas they should logically be combined into a single subsection, given their interconnected content.

2. Within the Emotion Wheel, adjacent emotions typically reflect interconnected affective experiences. However, the authors' adjustments to the numbering or distances on the Emotion Wheel, based on emotional polarity, have regrettably separated correlated emotion pairs such as excitement and fear, as well as amusement and anger, while paradoxically bringing the unrelated emotions of amusement and fear closer together. This configuration introduces substantial issues.

3. The assertions depicted in Figure 1 raise concerns; trends across various models, with the exception of the final MAPNET, demonstrate substantial consistency. Moreover, given the variations in model architectures, it is illogical to evaluate the differences in ACC and EMC performance based purely on release dates or ACC statistics.

4. In practical contexts, the emphasis on errors does not necessarily correlate with inaccuracies in polarity, suggesting that the EMC's utility may be limited.

5. While semi-supervised learning is utilized due to the difficulties in annotating data for emotion recognition tasks, the discussion of new evaluation metrics necessitates more accurate methodologies to effectively demonstrate these metrics' utility. Therefore, integrating experiments with fully supervised learning is advised.

6. In Section 4.2.1, the authors assert that models trained using CE loss achieve superior ACC performance, whereas those trained with a composite loss function, despite lower ACC, more effectively address misclassification and produce higher-quality pseudo-labels. Nevertheless, it remains ambiguous why subsequent sections claim an accuracy improvement of 1% over $S^2$-VER using our methodology.
7. Table 3's presentation lacks clarity. The meaning of "Our Rank" is unclear, and the justification for comparing RA w/o R1 with RE w/o R1 is not thoroughly elucidated.
questions: questions: Same problems as mentioned above.
limitations: limitations: The authors mention some limitations in semi-supervised learning. However, it is recommended that they include a discussion of related work on supervised learning.

Comments: comment: Thank you for the author's efforts and responses, which address some of my concerns. After considering the other reviewers' comments and the rebuttal, I will raise my score.
Dear Reviewer,

We conducted some additional experiments and provided responses to your concerns. We sincerely hope to continue our discussion with you and are very keen to explore further insights and suggestions you might have.
Dear Reviewer,

We have addressed the concerns you raised last time. We benefit a lot from the discussion with you, and we are looking forward to have more suggestions from you and continue the discussion.
Dear reviewer，

We are very glad to address your other concerns and add the corresponding experiments to the manuscripts later.
Dear authors,

Thanks for the comments and hard work. My major concerns have been addressed. I will raise my ratings and vote to accept the paper.
## Response
Thank you for your response.

We agree with your point that discussing emotion models is problematic, so we will not continue this discussion. 

What we want to address is the difference between object recognition and emotion recognition. In traditional object recognition, misclassification-related research is often based on WordNet, it can be used to define distances between categories. However, there is a fundamental difference between emotion recognition and object recognition. As we discussed earlier, there is no definitive answer as to whether six or eight discrete emotions better represent basic human emotions, whereas object recognition is developed based on ImageNet categories. This highlights a key difference between the fields.

Returning to our paper, We very much hope that our metric can be applied to all emotion datasets, but this inevitably requires modeling an emotion distance calculation method for every emotion model. Mikel and Ekman already encompass most of the mainstream datasets and research in the current emotion recognition field. What we need is a comprehensive emotion category model to address this issue, but that is beyond the scope of this paper. Our primary goal is to reveal that accuracy is not a suitable metric for emotion analysis. Based on this, we propose a new evaluation metric grounded on the majority of existing work and provide corresponding baselines.

Finally, we will follow your valuable review comments and add how the Ekman model uses our metrics to our future manuscripts.
It seems that even when applying other emotion models, you still rely on Mikel's wheel. For instance, in the case of Ekman's model, categorizing emotions like happiness and surprise as strictly positive is overly simplistic and limiting. Moreover, directly comparing different emotion models is problematic (and not something typically done by psychologists) because they do not align perfectly—for example, surprise, which can vary greatly in valence.

If I apply the same logic to using Mikel's wheel, which is associated with five datasets in [5], then it seems more appropriate to explore sentiment analysis, as it is linked to 11 datasets according to [5].

Nevertheless, your paper presents a method validated only for Mikel's wheel, which I still find limited. [5] has shown that other datasets exist, meaning that other emotion theories could have been tested using your distance measure. Such an approach would significantly enhance the impact of the paper.
## Response
Thank you for your response.

- First, we would like to say that the wheel model can also be extracted from Ekman and Plutchik to build our ECC and EMC, but we chose Mikel’s model as it already offers us an off-the-shelf wheel model.

  For example, in Ekman’s model, there are four negative emotions that are the same as those in Mikel’s wheel. These emotion categories can directly use Mikel’s distances. For happiness and surprise, we only need to ensure that they are classified as positive emotions. The emotion distance matrix is defined as follows:

  |Fear      | Sadness | Disgust| Anger|Happiness|Surprise|
  | ------- | ------- |------- | ------- |------- | ------- |
  | 1      | 2       |3      |4       |7       |6       |
  | 2      | 1       |2      |3       |8       |7       |
  | 3      | 2       |1      |2       |7       |8       |
  | 4      | 3       |2      |1       |6       |7       |
  | 7      | 8       |7      |6       |1       |2       |
  | 6      | 7       |8      |7       |2       |1       |
  
  
- Second, Mikel’s wheel is a model for eight categories of emotions, with the earliest paper[1] appearing in 2005, not 2016. Mikels et al.[1] through rigorous psychological research, categorizes emotions into eight types. Additionally, the paper points out that happiness, one of the six categories in Ekman’s model, often manifests as a mixed emotion rather than a single discrete emotion, which is clearly inappropriate.

  Furthermore, a significant portion of research[2,3,4] based on art theory has defined eight different pixel-level features, which have been proven to be related to emotional responses. Therefore, the eight-category emotion model is a more reasonable classification model and has been extensively studied.

  In addition, from the existing image emotion datasets, taking[5] as an example, there are 8 datasets using Mikel’wheel, but only 3 datasets using Ekman, and this does not include EmoSet.
  
  So we think the Mikel's Wheel is widely used.


## References
[1]Mikels, Joseph A., et al. "Emotional category data on images from the International Affective Picture System." Behavior research methods 37 (2005): 626-630.

[2]Itten, Johannes. "The art of color: the subjective experience and objective rationale of color." (1961).

[3]Valdez, Patricia, and Albert Mehrabian. "Effects of color on emotions." Journal of experimental psychology: General 123.4 (1994): 394.

[4]Machajdik, Jana, and Allan Hanbury. "Affective image classification using features inspired by psychology and art theory." Proceedings of the 18th ACM international conference on Multimedia. 2010.

[5]Zhao, Sicheng, et al. "Affective image content analysis: Two decades review and new perspectives." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.10 (2021): 6729-6751.
Dear Authors,

Thank you for your rebuttal. You have made an effort to address the implementation of your method with more widely recognized emotion models, such as those of Ekman and Plutchik. However, despite this concern being raised by another reviewer (Reviewer 133Y), your discussion remains inadequate. This reinforces my concern about the method's limited applicability, especially given its reliance on Mikel’s wheel, an emotion model that is one of the least utilized and referenced in the field.
We are very glad to address your other concerns and add the corresponding experiments to the manuscripts later.
Thank you for the author's response. This addresses some of my concerns, hence I will increase my score.