To Err Like Human: Affective Bias-Inspired Measures for Visual Emotion Recognition Evaluation
Authors: Chenxi Zhao
Jinglei Shi
Liqiang Nie
Jufeng Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: visual emotion recognition
evaluation measure
Primary area: evaluation
TL;DR: New metrics inspired by cognitive nuroscience are proposed to evaluate

Abstract: Accuracy is a commonly adopted performance metric in various classification tasks, which measures the proportion of correctly classified samples among all samples. It assumes equal importance for all classes, hence equal severity for misclassifications. However, in the task of emotional classification, due to the psychological similarities between emotions, misclassifying a certain emotion into one class may be more severe than another, e.g., misclassifying 'excitement' as 'anger' apparently is more severe than as 'awe'. Albeit high meaningful for many applications, metrics capable of measuring these cases of misclassifications in visual emotion recognition tasks have yet to be explored. In this paper, based on Mikel's emotion wheel from psychology, we propose a novel approach for evaluating the performance in visual emotion recognition, which takes into account the distance on the emotion wheel between different emotions to mimic the psychological nuances of emotions. Experimental results in semi-supervised learning on emotion recognition and user study have shown that our proposed metrics is more effective than the accuracy to assess the performance and conforms to the cognitive laws of human emotions. The code is available at https://github.com/ZhaoChenxi-nku/ECC.

To Err Like Human: Affective Bias-Inspired Measures for Visual Emotion Recognition Evaluation
Chenxi Zhao 1.2 Jinglei Shi 11 Liqiang Nie 3 Jufeng Yang 1,2,4 VCIP & TMCC & DISSec, College of Computer Science, Nankai University Nankai International Advanced Research Institute (SHENZHEN- FUTIAN)
3 Harbin Institute of Technology (SHENZHEN) Peng Cheng Laboratory zhaochenxi @mail nankai edu. en, jinglei. shi maanaai.edu.
Abstract
Accuracy is a commonly adopted performance metric in various classification tasks, which measures the proportion of correctly classified samples among all samples. It assumes equal importance for all classes, hence equal severity for misclassifica- tions. However, in the task of emotional classification, due the psychological similarities between emotions, misclassifying a certain emotion into one class may be more severe than another, e.g., misclassifying "excitement" is more severe than as 'awe'. Albeit high meaningful for many applications, metrics capable of measuring these cases of misclassifications in visual emotion recogni- tion tasks have yet to be explored. In this paper, based on Mikel's 's emotion wheel from psychology, we propose a novel approach for evaluating the performance visual emotion recognition, which takes into account the distance on the emotion wheel between different emotions to mimic the psychological nuances of emotions. Experimental results in semi-supervised learning on emotion recognition and user study have shown that our proposed metrics is more effective than the accuracy to assess the performance and conforms the cognitive laws of human emotions. The code is available at lhttt::/ /ttp///gtttt.tt///////e com/ Www.ChexxChhhiiccoaaaaa wwwtieeeeniiinnnnneeeeee -nku/Ek ECC
'anger' apparently
Introduction
"The best and most beautiful things in the world cannot be seen or even touched. -Helen Keller
They must be felt with the heart."
Emotion, as a complex state of feeling that results in physical and psychological changes and in- fluences thoughts and behavior, involves subjective experiences, physiological arousal, cognitive appraisal and behavioral expressions [33 Emotional similarity is significant characteristic of emotions, revealing the commonalities among diverse emotional states their essence, modes of expression, and influencing factors. And the essence of emotion can be explored through under- standing the hierarchical nature of human cognitive processes. The Reverse Hierarchy Theory [[4] in neuroscience suggests that, instead of exhibiting absolute object classification ability, humans first recognize coarse-grained categories and then proceed to identify finer-grained detailed infor- mation [1 22). This process involves comparing with prior information make comparative classification [10]. Under the same cognitive mechanism, both emotion recognition and object recognition follow similar pattern of recognition [24]. i.c., progressing from global local, and from coarse-grain -rasined to fine-gratie mnn-grained identification processes. And such emotional cognitive process is often characterized by the term Emotion Granularity in psychology [1] 18), where it involves the degree of nuance with which individuals can perceive. It also implies that there exists relative
*Coreesponing author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Excilement
Anger
ACC
RMC
0.8036
0.7921
0.7
0.7396
Awe
Disgust
0.65
0.6
0.648
0.655
0.67
0.688
0.668
0.679
0.661
0.5638
0.55
0,5186
0.5)19
0.5234 0.5276
0.5
Contenimenl
Sadness
0,4602
0.45
0,4188
0,4076 0.4098
0.4
Amasement
Fear
Vige16
Vegto
Res50
Resi01
Joint
WSCNer
PDANe
OSANe
SinEmo
MSPNd
Figure 1: (a) Different misclassification situations can not be treated equally. It better misclassify an image labeled as 'excitement" as 'awe' than to misclassify it as 'anger'. (b) Although the ACC of the sentiment classification algorithm has been significantly improved on FI ((0) in recent years, its Emotional Misclassífication CO))) has decreased significantly.
proximity between different emotions. For instance, belonging positive emotions, the similarity between excitement and awe is greater than that between excitement and anger. This proximity is often associated with a misclassification-siiiii biased attention', ' which is the tendency to pay more attention to some emotions than others. In complex psychological conditions, people tend to show negative bias compared to positive informa- tion, that is, they tend to use more negative information [36]. Though the emotion similarity and affective bias have been considered when evaluating cognitive ability for humans in psychology. the measurement of such ability for emotion recognition methods [19 8 25 in computer vision field still remains within the framework of the ordinary classification task. They primarily focus on [ or multi-stage perception (entity, attribute and emotion) model (25 to improve the number of correctly classified samples, but totally ignore the influence (or severity) of different misclassifications caused by the similarity between emotions. As shown in Fig. Ia), the relative distances between emotions are different. And after testing with the recent representative emotion classification methods in terms of both accuracy and severity of misclassification in Fig. [b), we observed that, although the classification accuracy has been improved to high level, the severity of misclassification stayed at the same level worse. Therefore, metrics that aligns more closely with psychological models by taking the emotional similarity and the severity of misclassification into account are necessary for the visual emotion recognition task.
phenomenon in psychology referred to as 'affective
employing global representation of objects [19] language prompts
Studies on the misclassification for other tasks can be traced back to the early 'cost-sensitive' problem, where researchers employ the cost matrices to assess and enhance classifiers, but they care more about the class imbalance in datasets rather than the costs led by misclassification. In the field of object classification, some researchers began to pay attention to the problem of misclassification and devised methods measure and reduce mistake severity based on the hierarchical information of labels 12 44. For instance, Bertinetto et al. [ devised the hierarchical distance of a mistake and average hierarchical distance of top-K to quantify the severity of mistakes, and proposed the hierarchical cross-entropy loss to reduce the severity of mistakes. Garg et al. [12 proposed to learn a hierarchy- feature space to explicitly learn the hierarchy of labels during the training phase. All these works are based on the cost definition composed of semantic hierarchical information and lowest common ancestor, which can be traced back WordNet [11]
However, directly applying the experience from hicrarchy- based severity learning to the visual emotion recognition task is challenging, because the multi multi-hierarchy relationships in object categories do not exist in emotional categories, thus we can not utilize hierarchy information to define the severity of misclassification. And could not adopt additional classifier to capture the structure available in the label space 12 While this severity is essential for designing robust measure and effective loss function for the visual recognition task. And prior works have pointed out that making the classifier to learn both ground truth information and label structure will degrade the classification accuracy [12].
To address the above-mentioned issue, we define the concept of emotional distance (or cost) for misclassification based on the Mikel's 'ss wheel 23 57 and further propose novel metrics for evalu-

---

ating the performance of emotion classification methods using this concept as foundation. More specifically, we first define the cost matrix for different misclassification cases according to the definition of the distance in [57 and of the affective polarity in Mikel's Wheel. On the basis of the confusion matrix between the cost matrix and the classification results, we then propose a new visual emotion recognition evaluation measure ECC to measure all possible classification results, as well as a measure EMC focusing more on the cases of misclassffication. We verify the effectiveness of the measures via the semi-supervised emotion recognition task, demonstrating that metrics provide more robust assessment of method performance than accuracy alone, And we prove through user study that the design of our metries in line with human cognition. As far as we know, both EMC and ECC are the first metrics that evaluate the performance of visual emotion classifiction algorithms by taking the influence of misclassifiction into consideration.
Our main contributions can be summarized as follows: 1). We are the first to introduce the concept of mistake cost into visual emotion recognition, and propose new measures based on Mikel's wheel to better assess the performance of emotion classification methods. 2). In semi-supervised emotion classification tasks, from the perspectives of threshold adjustment and model selection, on one hand, compared to complex confidence threshold adjustment mechanisms, our measures can be more simply yet effectively applied in the pseudo-labeling process, enhancing the model's s classification performance. On the other hand, we demonstrate that our metrics can help select the model capable of generating higher-quality pseudo-labels that are beneficial for training. 3). Finally, we discussed the relationship between our metrics and ACC₂, and we further validate our metrics through user study, confirming that they provide results consistent with human emotional cognition in evaluating the severity of sample misclassification by the model.
2 Related Work
2.1 Visual Emotion Recognition
With the advent of deep learning, many methods [60| 30 53 38
49
29
46 42 56
54 55 have
begun to use convolutional neural network (CNN) for visual emotion recognition. Many of these researchers have explored the relationship between emotions. Based on Plutchik's (28] theory that different emotions have different similarities. Zhao et al. [57 proposed the emotional distance for the first time based on Mikel's Wheel, which is used to calculate the emotional similarity between two pairs of pictures. In addition, Yang et al. [46] designed affective polarity on Mikel's wheel and proposed a new hierarchical cross-entropy loss to distinguish between difficult and easy cases in specific emotional way. Inspired by the large-scale pretrained language models, Deng al. [8] propose a fine-tuning strategy based on prompts. Pan et al. (25) generate pseudo labels through visual language models auxiliary guidance for multi-stage visual perception. However, due the ambiguity and subjectivity of emotions, a single image often elicits multiple emotional responses, when dealing with visual emotion recognition tasks, is more reasonable to use label distribution than single label classification. In [48), they generate sentiment distributions from a single emotion dataset based on emotional distance to solve this problem. Moveover, inspired by the inherent relationship between emotions in psychology. Yang et al. [45] propose a well-based circular structure representation to use prior knowledge to learn visual emotion distribution. However, these efforts mainly focused on improving accuracy, ignoring the fact that the severity of misclassification is not the same for different classification results, and at the same time, the constraints of fitting the label distribution may be strict, which often have the opposite effect.
2.2 Cost-Sensitive Classification
The importance of studying misclassification has attracted a lot of attention in the era of machine learning, but it has been neglected in the era of deep learning. In the field of machine learning, Wei et al. [40] proposed the measure of confusion entropy to evaluate the performance of classifiers, which utilizes the distribution information of misclassified categories for all categories. This problem also described as cesstssensitive by introducing the cost imbalance between different misclassifications in real-world applications, and providing solutions that meet practical needs. Classic problems include bank lending issues, disease diagnosis issues 2 51 etc. The most common cost-sensitive solution rescaling, which mainly preprocesses the training set to improve the sensitivity to classification results. Specifically, Turney [35] studies how to choose the correct cost assignment in cost-sensitive
3

---

classification problems, and further explores the meaning and impact of cost values. Metacost [9] calculates the ideal class for each training sample by estimating the posterior probability density of the training samples and modifies the class of the original training sample to change the new data set. Different from the previous approaches, Zhou al. [59] pointed out that in addition to the correction of the classification algorithm and dataset, the cost matrix needs to be corrected, which solves the problem that it is often ineffective for multi-class cost-sensitive learning. However, this kind of problem focuses more on the class imbalance of the dataset, and gradually loses interest in the cost of misclassification.
2.3 Hierarchy Aware Classification
In recent years, a small part of the work has begun focus on the problem of the mistake severity and devised methods to reduce it. Deng et al. [6] pointed out that classification can be improved by using semantic hierarchical information from WordNet [II], which laid the foundation for future research. Furthermore, Bertinetto et al. [ summarized and analyzed the issue of mistake severity, and proposed hierarchical cross-entropy loss to reduce the severity of mistakes. They also introduced two measures, hierarchical distance of mistake and average hierarchical distance of top-K, K, to quantify the severity of misclassification. However, reducing the severity of misclassification is premised on reducing accuracy. Karthik et al. [17] use Conditional Risk Minimization (CRM) to improve this shortcoming, reducing the severity of mistakes without compromising accuracy. But CRM doesn't I change the model, it's about making the best choice of the moment during the testing phase. In order to solve this problem elegantly, Garg et al. [12] propose to learn a Hierarchy Aware Feature(HAF) space to explicitly learn the hierarchy of labels during the training phase. In (15), they train two separate models for coarse-grained and fine-grained, and make the final prediction by calculating the normalized scores of the two models in the reasoning process. Although some progress have been made in these studies in the direction of hierarchical labeling, there is still a lack of research in visual emotion recognition. Different from they lowest common ancestor (LCA) measure to assess the severity of mistakes, we define our measures based on Mikel's 's wheel and confusion matrix, and demonstrate their effectiveness.
3 Misclassifccation-Aaaal Measure Design
3.1 Definition of Emotional Distance
In almost classification tasks, accuracy serves as an important evaluation metric, measuring the model's ss capacity of predicting categories. However, this measure is built on binary philosophy, i.e., only considering whether or not the predicted category matches the true label. In fact, distinguishing between different types of mismatch is meaningful, particularly in the field of emotion classification. For example, due to the similarity between emotions, misclassifying 'excitement' as 'awe' may be acceptable in certain diagnose, while misclassifying it as 'sadness" could lead to severe diagnostic errors in clinic psychology. Based on this reasoning, we therefore define emotional distances to characterize their relative relationships, and further to aid in quantifying the degree of severity of misclassification. Existing psychological models such as CES and DES, either model emotions as completely independent categories or represent them as multidimensional continuous vectors. The former ignores the similarity between emotions, while the latter requires precise measurement of emotions by experts, which can be hard to achieve for visual emotion recognition task. In [57], authors provide a definition between emotional labels based on Mikel's 's wheel, where the paired emotional distance is 1+ 'wheel distance'. The 'wheel distance' means the number of steps when moving from one category to another on the wheel like shown in Fig. 2 (a). However, this definition of emotion distance neglects the emotional polarity 477, e.g., the distance between 'fear" and 'excitement' being the same the distance between "fear" and 'sadness'. But the distance between categories with opposite polarity should be greater than those with the same polarity, as shown in Fig. (b). We thus define the emotional distance as follows:
1 + dist e5) e, CCp. C + dist (et, Ez) e, $ C2
(1)
where dist. (ei, ej) represents the number of steps on the Mikel's wheel, C is constant to adapt the importance of polarity, and a larger C means misclassifying emotion into the opposite polarity

---

distte; "ear")
Positive
Negative
ACC
EMC
BCC
Predicted Labels
excitement
Fpar
fear
sadness
Thee Labels
confentment
disgust
contentment
disgust
3
nnger
amusement
anger
8
(a) Mikel's emotion distance
(b) Our emotion distance/munk
Confusion matrix
Figure 2: (a) Mikel's emotion distance. (b) Our emotion distance/rank which is labeled with 'excitement'. (c) The correspondence between the three measures ACC, ECC and EMC in the confusion matrix.
is more severe, and we define C as in order to ensure that the emotional distance within the same polarity is always less than the emotional distance hetween different polarities. Cp represents the classes that have the same polarity. We can finally obtain a symmetric cost matrix with each element being the distance between the corresponding emotions. Although our emotional distance is defined based on Mikel's wheel for classifying eight emotions, it is important to note that when calculating the classification of six emotions based Ekman's model, can still use similar method for determining emotional distances. To achieve it, we can initially follow the definition used by Mikel's wheel, calculating the emotional distances for the same emotions present in both models. Subsequently, we can adjust the distances between opposite emotion labels based on emotional polarity. Due the inherent nature of the distance definition in Mikel's Wheel, our main focus here is on the issue of classifying the eight categories of emotions.
3.2 Emotion Confusion Confidence (ECC)
Wei al. 40 used the category distribution information of all categories of misclassification to propose the concept of confusion entropy to measure the standard. Inspired by them, we design our measures based on confusion matrix. As shown Fig. (c). the confusion matrix is defined as a NxN matrix, where N represents the number of classes. The rows of the confusion matrix represent the true labels, while the columns represent the predicted ones. Each element in SJ represents the number of samples classified from the correct category i to the category j. The diagonal elements of the confusion matrix represents the number of samples correctly classified for each class. Hence the sum of diagonal elements of confusion matrix N. divided by the total number of samples N represents ACC. While other elements represent the number of samples belonging to the category i that are wrongly classified into category j. The drawback of accuracy lies in its exclusive consideration of correct classifications along the diagonal of the confusion matrix, while disregarding equally important misclassifications elsewhere in the matrix. This implies that accuracy may be misleadingly high in scenarios of class imbalance. Although directly using the confusion matrix as a metric allows for the consideration of both correct and incorrect classifications (misclassifications) this metric overlooks the distance between emotions in psychology. To take both correct classifications, misclassifications and emotional distance into evaluation, we propose modulate the confusion matrix. Based on this design philosophy, the ACC can be re-formulated as the product of the confusion matrix and a modulation factor Mi,5 as:
emotional distance
N. ΣS
c Sis5 X M Mi.
{ i=j i $
ACC
where denotes the number all classes, N₂ represents the count correct classifications. implies that one sample is counted as I only if it is correctly classified, other misclassified samples are not distinguished and are all recorded as O. However, as we explained previously, misclassification can be acceptable to some extent when the emotional classes similar. Thus tackle these samples
5

---

as "quasi correctly classified samples' and weighted with a value between (0, 1). The smaller the similarity of emotions. the less acceptable the misclassified sample becomes, hence the smaller the corresponding value it will have, and vice versa. To achieve this, we rely on the reciprocal of the emotional distance Wa, defined in Eq. to replace the modulation factor Mij in Eq. for the misclassified samples to obtain ECC in Eq.
Tq
ECC = Σ 1= Sig X W., = ACC+ Ej- Eim Sa. x W.,
(3)
N
N
In this way, misclassified samples are no longer simply neglected like in ACC, but have weights in terms of emotional distances, which also means that our measure ECC make use of the entire confusion matrix, including elements both inside and outside the diagonal as shown in Fig. (c).
3.3 Emotional Misclassiffcation Confidence (EMC)
In some practical scenarios, people pay more attention to cases of misclassification, like estimating the severity of misdiagnosis of certain mental diease, while both ACC and ECC include the cases of correct classification, hence failing to provide information about misclassification. Therefore, in order to only consider the cases of misclassification, one can extract the term (2, D Lim Sag X
Wa. )/N that excludes ACC from Eq. as an indicator for evaluating misclassification. However, directly using this terms as misclassification measure is inappropriate for two reasons: 1). The denominator of this term N, indicating that its value still influenced by correct classified samples, rather than solely considering the misclassified samples. 2). The maximal value of this term is 0.5, which we hope to be I as ACC and ECC. We accordingly modify this term and propose a novel measure for misclassification description in Eq. named Emotional Misclassification Confidence (EMC):
Di 1,17 Sin X W.
FMC
Ne
This metric considers only misclassifed samples Si,j + j and their number N Ne. meanwhile modifying the modulation factor from W.y
W ensure maximum value of
In this way, we have an ECC that measures the overall classification results, as well EMC that is specially designed to measure the severity of misclassification The cooperation of two measures can better evaluate the visual emotion recognition task.
4
Experiments and Results
4.1 Datasets
We evaluate our metrics on two widely applied datasets EmoSet and [50). EmoSet based on Mikel's eight-categorical sentiment model, which uses 810 keywords and collects from four different sources, including openverse, pexels. pixabay and rawpixels. It covers different emotional attributes. i.e., low-level (brightness and colorfulness). mid-level (scene type and object class), and high-level (facial expression and human action). Finally, 60 annotators who passed the test annotated total of 1 118, 102 images. The FI dataset was collected from Flickr and Instagram through eight sentiment keywords, and was built based on Mikel's eight-category sentiment model, which contains about 23,308 images.
4.2 Applications to Semi-Supervised Emotion Recognition
Because of the ambiguity of emotions [6]. annotating high-quality and large-scale datasets for visual emotion recognition is arduous and challenging. Semi-supervised learning is an effective solution which consists of training the model based on both labeled and unlabeled data, then annotating unlabled examples by this trained model. While for pesudo labeling based semi-supervised learning methods, selecting appropriate models and labeling confidence thresholds play key roles determin- ing the final performance. In this section, we carry out experiments on two perspectives: the model selection and the adjustment of labeling threshold, to demonstrate how our proposed measures can benefit this task.

---

Table I: We conduct experiments on the datasets FI [50] and EmoSet 44 and evaluate the exper- imental results using ACC. considered fair to compare our method in 4.2.1 with FixMatch and FlexMatch, given that all of them employ threshold adjustment methodologies. Where "TA' means threshold adjustment. Based on S 3 we compared our method in 4.2.2 with all the state-of-the remi-supervised methods.
F1
EmoSet
label num
80
800
1600
400
4000
8000
Fixmatch
32
28.2+0.78
37.4+0.51
42.2#0.29
31.1+0.41
42.3+0.65
45.8
Flexmatch 62
29.7=0.90
38.2+0049
40.6+0.55
30.4+0.78
42.8+0.34
44.9#1.24
Ours(
4.2.1
31.2+0.12 40.8+0,34
42.7+0.21
31.6+0.56
43.7=0.69
47.6+0.61
Comatch
20
36.7+0.87
43.5+0.39
47.9=0.26
30.3+0.97
44.2+0.41
46.8=0.49
State-of Balle-ofthhe af-the-art
Simmatch
58
31.4+1.26 41.9=0.57
43.7=0.62
36.3+0.22
44.7+0.34
50.2+0.71
Freematch
39
26.0#1.66
37.320.43
39.920.87
31.2+1163
41.5+0.59
46.3+0.61
Softmatch
S
30.7+1.31 37.940.78
10.7=0.19
30.8+0.35
44.0#1.28
45.8=0.25
S 16
39.1+0.66
46.9+0.46
51.8+0.21
44.9=0.35
57.5+0.51
60.2+0.34
Ours( 4.2.2
40.2+1.08 48.9+0.91 52.1+0.33
47.020.18
59.0+0.33 61.5+0.12
4.2.1 Adjustment of Confidence Threshold
For pseudo label-based semi-supervised learning methods, a confidence threshold is used to determine whether current labeled samples are filtered will directly affect the number of training samples and the proportion of correct labels in each epoch: a high threshold will excessively filter out high-qualig, unlabeled data, leading insufficient training data, while a low threshold will allow more -wwquullty unlabeled data, making the model converge to a poor local minima. How to select an appropriate threshold of pseudo-labeling in the training process has always been hotspot for semi-supervised learning [5] Existing methods either use an intuitive constant threshold 32 that is widely adopted in other semi-supervised learning tasks, dynamic threshold that varies in terms of the estimated learning status (52). Considering that EMC can measure the severity of misclassification we therefore recommend using EMC to dynamically adjust the confidence threshold before each epoch. This is similar to how, in a clinical scenario, a patient not only refers to the doctor's s current confidence in the diagnosis but also takes into account the doctor's historical reputation (frequency of misdiagnoses and medical errors) to ultimately judge the reliability of the current diagnosis. For semi cumi-supereieed emotion recognition tasks, since there are only few labeled samples available for training, the model can casily overfit these samples, resulting in all correct predictions and thus making it impossible to caleulate FMC. To address this situation without introducing additional training data or increasing computational payload, we treat the pseudo-labels generated from weakly augmented unlabeled samples as the ground truth, and use the predicted labels from strongly augmented samples as the model's 's predictions to calculate EMC. Since the labels of the same sample should remain consistent under different data augmentation methods, a large EMC indicates that the model provide similar emotional predictions with different augment methods. This suggests that the model has grasped the underlying visual elements that represent emotions in images, which reflects the reliability and lower the confidence threshold, allowing more
high quality of the pseudo labels. Therefore, pscudo-labeled samples to participate in the training process. On the contrary, when the EMC is small, we can raise the threshold and filter out the low-quality samples. We can realize the above confidence adjustment mechanism by simply setting:
EMC,
where 7 is the pre-defined threshold, T/ is the new threshold at time step t, EMC, represents the EMC at time step t, is constant for different datasets. We define 7 as 0.95. For FI, we define as 0.5, and for EmoSet, we define e as 0.4. And we set the upper and lower bounds of T/ to 0.98 and 0.7 [16] respectively to ensure stability of the training process.
To test the effectiveness of our proposed confidence adjustment mechanism built on EMC, we com- pared it with two representative pesudo labeling based methods, Fixmatch (32 and Flexmatch [52).
7

---

The test set misclassification
ACC
Indistinet
ECC
30
Data Preparation
20
User Chosen Num
Model_A A
Model_E E
15
Sadness
Amusement
Survey Questionnaire
10
1. ..Saness
2. Amusement
3. Innniistnntt
"which emotion closer to the picture's emotion?"
3
5
9
11
13
15
17
19
21
23
25
27
29
Image ID
Figure 3: User study about measures. (a) The pipeline of our study. (b) The result of user study, where the horizontal axis is the of the image, the vertical axis is the number of votes, and the results of different options have been distinguished by different colors.
where Fixmatch directly provides fixed threshold, while Flexmatch assigns dynamic thresholds to each class according to corresponding learning status. Experimental results Tab. show that, thanks to our EMC-based threshold mechanism, Fixmatch. Additionally, let us note that unlike the method Flexmatch that adjusts the threshold for every category, evaluate the entire ublabeled data based statistical perspective, and adjust the overall threshold after measuring the EMC, which is computationally low-cost. And the experiment followed by [32
method is over 1% better than Flexmatch and
4.2.2 Selection of Better Pseudo Labeling Models
Existing methods that typically use ACC to measure the model's capability of discriminating between categories of unlabeled samples, i.e., models with higher ACC can provide more reliable pesudo label. However, such models often suffer from the problem of confirmation bias [4], where the model will gradually deepen this error during the learning process. Accumulation of these errors eventually leads to the final model being unable to achieve good classification performances. As both ECC and EMC are designed based on the consideration of the cases of misclassification. which means they can better distinguish the ambiguity of labels, and models selected in terms of ECC or EMC will less effected by the cumulative confirmation bias, providing high-quality labels than those based on ACC.
To prove that model with higher ECC or EMC is better for pesudo labeling. we train the same network with different loss functions: cross-entropy loss LoE and the combination of cross-entropy loss and order-based loss ListMLE 41 as L. Lcr + a where & is and LLiMMLE aims to constrain the final prediction probability of the samples to follow preset order, thus favoring higher ECC and EMC (proved in Appendix AA.. This constraint will reduce the severity of misclassification. at the same time, the we first train the model with LOE. once it starts to converge (the model has preliminary ability of recognition), we then keep the same loss function or replace LeE with L. make continue to focus on the correct classification or focus more on the cases of misclassification. Therefore, models trained solely using LeE exhibit better ACC but limited capability distinguish error samples, and the quality of the pseudo label is poor. Although the pseudo labels might still be incorrect compared to the ground truth when using a combined loss function for training the model, the pseudo- labels become closer to the ground truth. In such cases, these pseudo labels can still have positive impact on the training process and thereby improve the model's 's accuracy. As we show Tab. [ we adopt the state-of-te of-the-art method S2- VERVER [16] our baseline, it generates more reliable pseudo labels by
to reduce ACC [1], specific experiment is in Appendix B More precisely.

---

Table 2: We conducted experiments on the Fl [50] and EmoSet (44]
datasets using three backbones.
As mentioned above, the loss function employed the commonly used the cross- cros-ennroppy loss LCE and the combined loss f..
Dataset
Fl
EmoSet
Backbone
Resnet18
Resnet50
Resnet101
Resnet18
Resnet50
Resnet101
Loss function
LeE
L.
LCE
L.
Lcx
L.
LCR
C.
Les
Le
LeE
Le
ACC
65.8 64.4
67.6
66.2
68.1
65.6
73.9
72.4
76.2 74.3
76.7
74.5
ACC2
79.0 86.2
83.7 86.0
84.7 86.0
85.0 85.6
85.3 85.8
85.7 85.8
ECC
76.1
75.8
77.2
76.8
77.9
76.7
82.6
81.9
84.2
83.2
84.5
83.3
EMC
50.1 54.8
49.0
53.2
$1.9
54.8
57.0
59.3
57.0
59.9
56.9
60.5
calculating the similarity between emotional prototypes and samples, but ignores the error of pseudo labels. We follow the experimental setting of [T6) and vary the proportions of the labeled samples as 0.5%, 5% and 10% (corresponding to 80, 800, 1600 label number for FI, and 400, 4000, 8000 label number for EmoSet), it can be observed that under the different settings of the two datasets, our method performs favorably against S VER by 1% in accuracy. Meanwhile, our method also far surpasses multiple state-of-ther the-art methods in semi-supervised learning. It indicates that choosing a model with better misclassification ability (better ECC and EMC) can produce pseudo labels of better quality and beneficial to the training process, thus achieving better cemi-supervised performance.
4.3 Compare with Other Measure
ACC₂ is very important binary classification metric in the field of emotion recognition 43 used to measure whether the classification to the same emotional polarity is correct. More specifically, when a sample labeled as 'excitement" is classified as 'awe', is incorrect to use accuracy for evaluation. However, for metrics like ACC₂, such classification is considered correct. In certain sense, metrics like ACC2, which involve coarse-grained classification, take into account the proximity of labels and consider misclassified cases. This approach aligns with the objectives of our measures. To further demonstrate the effectiveness of our metries, As mentioned above, we conducted experiments using both the cross-entropy loss LeE and the combined loss L. As shown in Tab. 2, although the accuracy of the combined loss is lower than that of the cross-entropy loss, its ACC₂ is higher. This indicates the shortcomings of using accuracy alone in certain situations, it fails to measure for misclassifications. Although the ECC also decreased due to the influence of the ACC, since the ECC takes into account the situation of misclassification, the gap between the two models in terms of ECC is not significant. As EMC considers metrics for misclassification alone, the EMC of the combined loss is significantly higher than that of the cross-entropy loss. In this regard, the trend of EMC aligns with that of ACC2, which also demonstrates the correlation between the two metrics. In the confusion matrix, ACC₂ actually represents the proportion of correctly classified samples in the top-left and bottom-right sections. This also indicates that our measures are actually more refined measures that lies between Accuracy and ACC₂-
4.4 User Study
Since our metrics is founded on principles of human cognition, aim to further demonstrate that our measures align with human judgments in emotion classification results via user study.
Data preparation In order to have models having different levels of ECC, take ResNet50 as our network backbone and train it respectively with cross-entropy loss and combined loss L. = the FI dataset, where a is 0.2. Then we perform predictions the test set
LeE + aLLisaml and select the images that are misclassified by both two models into different classes. Finally we randomly select 50 eligible images, and filter out the images with no obvious emotion ambiguous emotion, getting 30 carefully selected images as our tested images for user study.
Preference Study We invite 30 participants having different social backgrounds to our user preference study, and the test for every participant lasts about 15 minutes. During the test session, each misclassified image will be presented to participants with three options: the incorrect class predicted

---

Table 3: The table of comparative analysis of the impact of label ranking on single visual emotion classification task. We used ListMLE loss to do experiments on FI. 'Our Rank' stands for the Mikel's emotion rank we defined based on Mikel's s wheel. 'RA' means random scrambled labels, and 'RE' means scrambled labels in reverse rank. 'w/o R1' means keeping the ground truth rank first when scrambling the Label rank. Among them, the red font represents the best.
Resnet18
Resnet50
Resnet101
Label Rank
ACC
ECC
EMC
ACC
ECC
EMC
ACC
ECC
EMC
RE w/o R1
60.6
70.2
40.4
63.4
73.1
43.3
63.8
73.6
44.4
RA w/o R1
61.2
71.5
48.6
63.8
74.1
48.6
64.4
74.6
49.5
Our Rank
63.9
75.1
58.4
65.7
77.0
51.5
67.9
77.6
52.2
from the model trained with cross-entropy loss, the incorrect class predicted from the model trained with the combined loss function, and third 'Indistinct' option for cases where participants unable to discern the emotion of the image. And participants will choose their preferred options after viewing each image.
Results In all 900 collected votes, 487 votes are cast for the results produced by the model with higher ECC, while 242 votes opt for the results generated by the model with higher ACC. There are 171 votes that opted for unidentifiable choices. Fig. 3ig-33iin further shows the distribution of the votes for each image, where results provided by the model with higher ECC over the model with higher ACC for 24 images, representing 80% of the total tested images.
observe that among 30 tested images, users preferred the classification
4.5 Validity of Emotional Distance Definitions
We want to further explore the rationality of the defined emotional distance and determine whether it can help models learn the semantic structure of labels. To answer the above question, we transform emotional distance into ranking, and designed three sets of experiments based on Le and a is 1. The specific experiments are shown in Table 3 And the specific experimental settings are detailed in Appendix
C Since we are only focusing on the impact of emotional rank (emotional distance) on the model, and changing the rank of ground truth label would prevent the model from training on correct classification categories. So we randomly shuffle and reverse the rank of the other labels while keeping the category with the first position in the emotional rank as the ground truth. Then, according to the changed order, use ListMLE for training. As shown in Table 3] Our rank achieve advanced performance in three measures. And the results are worse reverse rank than random rank, and they are significantly worse than our rank. This shows that our rank is better, in line with human cognition of label rank, and our label rank will help the model learn the emotion category structure. Although the ranking of ground truth has not changed, reasonable label ranking the emotional and visual element relationship between images, which will enable the model to mine the visual and semantic correlation between similar categories, structure. We have proven the rationality of emotional distance through above experiments. Since the new measures ECC and EMC are designed based emotional distance, it also validates the rationality of our measures.
often reflect
to learn better label semantic
Conclusions
In this work, we define the concept of misclassification in the field of visual emotion recognition, and propose new measures evaluate the mistake severity in visual emotion recognition based on Mikel's Wheel distance. We define our emotional distance using the Mikel wheel and adopt it to build cost matrix, then exert it to confusion matrix to compute emotion confusion confidence (ECC) and emotional mistakes confidence (EMC). And we demonstrate that our measures are more robust in semi-supervised learning. Our measures can not only help to select the model that can produce high- quality pseudo labels, but also can be used as a reference standard to adjust the threshold adaptively. Moreover, we verify that our new measures are consistent with human emotional cognition through user study. Finally, we verify the validity of our emotional distance.
10

---

Acknowledge
This work was supported by the Natural Science Foundation of Tianjin, China(NO.200C1QJC00002 NO.22JCQNICO116G) the National Natural Science Foundation of China mental Research Funds for the Central Universities, Supercomputing Center of Nankai University (NKSC).
Funda-
References
[1] Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 12506-12515, 2020.
[2] Zhe Cao, Tao Qin, Tie Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. Proceedings of the 24th international conference on Machine learning, pages 129-136, 2007.
[3] Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Your" flamingo" is my" bird": fine- ane-graine, or not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11476-1 1485, 2021.
[4] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classifi- cation with rejection based on cost-sensitive classification. In International Conference Machine Learning, pages 1507-1517. PMLR, 2021.
[5] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi- supervised learning. arXiv preprint arXiv:2301. 2023.
[6] Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In European Conference on Computer Vision, pages 71-84. Springer, 2010.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[8] Sinuo Deng, Lifang Wu, Ge Shi, Lehao Xing, Wenjin Hu, Heng Zhang, and Ye Xiang. Simple but powerful, language-supervised method for image emotion classification. IEEE Transac- tions on Affective Computing, 2022.
[9] Pedro Domingos. Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 155-164, 1999.
[10] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representa- tions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588 9597, 2021.
[11] Christiane Fellbaum. WordNet: An electronic lexical database. MIT press, 1998.
[12] Ashima Garg, Depanshu Sani, and Saket Anand. Learning hierarchy aware features for reducing mistake severity. In European Conference on Computer Vision, pages 252-267. Springer, 2022.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[14] Shaul Hochstein and Merav Ahissar. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36(5):791-804, 2002.

---

[15] Kanishk Jain, Shyamgopal Karthik, and Vineet Gandhi. Test-time amendment with coarse classifier for fine-grained classification. Advances in Neural Information Processing Systems, 36, 2023.
[16] Guoli Jia and Jufeng Yang. S2-ver: Semi-supervised visual emotion recognition. In European Conference Computer Vision, pages 493-509, 99riigr, 2022.
[17) Shyamgopal Karthik, Ameya Prabhu, Puneet K Dokania, and Vineet Gandhi. No cost likelihood manipulation at test time for making better mistakes in deep networks. International Conference on Learning Representations, 2020.
[18] Todd B Kashdan, Lisa Feldman Barrett, and Patrick E McKnight. Unpacking emotion differen- tiation: Transforming unpleasant experience by perceiving distinctions in negativity. Current Directions in Psychological Science, 24(1):10-16 2015.
[19] SangEun Lee, Chaeeun Ryu, and Eunil Park. Osanet: Object semantic attention network for visual sentiment analysis. IEEE Trunsuctions on Multimedia, 2022.
[20] Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi- Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9475-9445 9484, 2021.
[21] Xiao Liu, Xiaolong Zou, Zilong Ji, Gengshuo Tian, Yuanyuan Mi, Tiejun Huang, KY Wong, and Si Wu. Push-pull feedback implements hierarchical information retrieval efficiently, Advances Neural Information Processing Systems, 32, 2019.
[22] Xiao Liu, Xiaolong Zou, Zilong Ji, Gengshuo Tian, Yuanyuan Mi, Tiejun Huang, KY Michael Wong, and Si Wu. Neural feedback facilitates rough-to-fine information retrieval. Neural Networks, 151:349-364, 2022.
[23] Joseph A Mikels, Barbara L Fredrickson, Gregory R Larkin, Casey M Lindberg, Sam J Maglio, and Patricia A Reuter-Lorenz. Emotional category data on images from the international affective picture system. Behavior Research Methods, 37:626-630, 2005.
[24] Albert Newen, Anna Welpinghus, and Georg Juckel. Emotion recognition as pattern recognition: the relevance of perception. Mind & Language, 30(2):187-208, 2015.
[25] Jicai Pan, Jinqiao Lu, and Shangfei Wang. A multi-stage visual perception approach for image emotion analysis. IEEE Transactions on Affective Computing, 2024.
[26] Jicai Pan and Shangfei Wang. Progressive visual content understanding network for image emotion classification. In Proceedings of the 31 ACM International Conference on Multimedia, pages 6034-60444 2023.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeffrmance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[28] Robert Plutchik. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist, 894):344-350, 2001.
[29] Tianrong Rao, Xiaoxu Li, and Min Xu. Learning multi-level deep representations for image emotion classification. Neural Processing Letters, 51:2043-2061, 2020.
[30] Tianrong Rao, Xiaoxu Li, Haimin Zhang, and Min Xu. Multi-level region- convolutional neural network for image emotion classification. Neurocomputing. 333:429-439, 2019.
[31] Katharine E Smidt and Michael Suvak. A brief, but nuanced, review of emotional granularity and emotion differentiation research. Current Opinion in Psychology, 3:48-51, 2015.
12

---

[32] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608 2020.
[33] Robb Stanley and Graham D Burrows. Varieties and functions of human emotion. Emotions at work: Theory, research and applications in management, pages 3-19, 2001.
[34] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30, 2017.
[35] Peter D Turney. Types of cost in inductive concept learning. arXiv preprint cx/02 12034, 2002.
[36] Amrisha Vaish, Tobias Grossmann, and Amanda Woodward. Not all emotions are created equal: the negativity bias in social-emotional development. Psychological bulletin, 134(3):333, 2008.
[37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[38] Xiaohui Wang, Jia Jia, Jiaming Yin, and Lianhong Cai. Interpretable aesthetic features for affective image classification. In 2013 IEEE International Conference on Image Processing, pages 22304. IEEE, 2013.
[39] Yidong Wang, Hao Chen, Qiang Heng. Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint urXiv:2205.07246. 2022.
[40] Jin-Mao Wei, Xiao-Jie Yuan, Qing-Hua Hu, and Shu-Qin Wang. A novel measure for evaluating classifiers. Expert Systems with Applications, (7(5):3799-3800) 2010.
[41] Fen Xia, Tie- Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning rank: theory and algorithm. In Proceedings of the 25th International Conference on Machine Learning, pages 1192-1199, 2008.
[42] Wuyou Xia, Shengzhe Liu, Qin Rong, Guoli Jia, Eunil Park, and Jufeng Yang. Perceive before respond: Improving sticker response selection by emotion distillation and hard mining. In ACM Multimedia, 2024.
[43] Liwen Xu, Zhengtao Wang, Bin Wu, and Simon Lui. Mdan: Multi-level dependent attention network for visual emotion analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9479-9488, 2022.
[44] Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Emoset: A large-scale visual emotion dataset with rich attributes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20383-20394, 2023.
[45] Jingyuan Yang, Jie Li, Leida Li, Xiumei Wang, and Xinbo Gao. A circular-structuree represen- tation for visual emotion distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4237-4246, 2021.
[46] Jingyuan Yang, Jie Li, Xiumei Wang, Yuxuan Ding, and Xinbo Gao. Stimuli-aware visual emotion analysis. IEEE Transactions on Image Processing, 00:7432-7445. 2021.
[47] Jufeng Yang, Dongyu She, Yu-Kun Lai, and Ming-Hsuan Yang. Retrieving and classifying affective images via deep metric learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[48] Jufeng Yang, Dongyu She, and Ming Sun. Joint image emotion classification and distribu- tion learning via deep convolutional neural network. In In Proceedings of the Internal Joint Conference Artificial Intelligence, pages 3266-3272, 2017.
[49] Xingxu Yao, Dongyu She, Sicheng Zhao, Jie Liang, Yu-Kun Lai, and Jufeng Yang. Attention- aware polarity sensitive embedding for affective image retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1140-1150, 2019.
13

---

[50] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
[51] Bianca Zadrozny and Charles Elkan. Learning and making decisions when costs and probabili- ties are both unknown. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 204-213, 2001.
[52] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 1e:1888011149) 2021.
[53] Wei Zhang, Xuanyu He, and Weizhi Lu. Exploring discriminative representations for image emotion recognition with enns. IEEE Transactions on Multimedia, 22(2):515 523,, 2019.
[54) Zhicheng Zhang, Lijuan Wang, and Jufeng Yang. Weakly supervised video emotion detection and prediction via cross-modal temporal erasing network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18888-18897, 2023.
[55] Zhicheng Zhang and Jufeng Yang. Temporal sentiment localization: Listen and look in untrimmed videos. In Proceedings of the 30th ACM International Conference on Multimedia, pages 199-208, 2022.
[56] Zhicheng Zhang, Pancheng Zhao, Eunil Park, and Jufeng Yang. Mart: Masked affective representation learning via masked temporal distribution distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12830-12840, 2024.
[57] Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, and Tat- Fat-Seng Chua. Predicting personalized emotion perceptions of social images. In Proceedings of the 24th ACM International Conference on Multimedia, pages 1385-1394, 2016.
[58] Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14471-14481, 2022.
[59] Zhi-Hua Zhou and Xu- Ying Liu. On multi-class cost-sensitive learning. Computational Intelligence, 26(3):232-257, 2010.
[60] Xinge Zhu, Liang Li, Weigang Zhang, Tianrong Rao, Min Xu, Qingming Huang, and Dong
Xu. Dependency exploitation: A unified cnn-rnn approach for visual emotion recognition. In In Proceedings of the Internal Joint Conference on Artificial Intelligence, pages 3595-3601, 2017.
14

---

Appendix
A Proof of the Relationship between New Measures and ListMLE
In this chapter, we mainly analyze the relationship between ACC and cross-entropy. ListMLE and ECC. To prove that the relationship between ListML.E and ECC is equal to the relationship between cross-entropy and ACC. so as to explain why ListMLE can be used as backbone between ECC. First of all, let's review the formula of cross-entropy loss:
1 TE
Lee
Mig log (z,;
(6)
where n represents the number of samples and represents the number of categories. Yij represents the jth element of one-hot encoded label of the sample z;. A the parameter set of the classifier. (z () represents the probability that the prediction of the ith sample is of category j.
For single sample, the formula becomes:
Lee =
Σ My log (z(
However, the formula of ACC is:
ACC Li= Si x M.j
(8)
N
In fact, the effect of My is the same as that of Mi.- optimization goal of cross-entropy to maximize the prediction probability of real categories, while ACC only calculates the number of samples predicted to the correct category.
In fact, the cross-entropy loss is also sensitive to the order. According to the paper 2 the cross- entropy loss can be written in the form of likelihood loss. Suppose that æ a permutation of objects, and @ is strictly increasing positive function, then the probability of permutation of given score list s defined as
Φ ($R())
(T( II
(9)
Σn Φ (sT(K)
In addition, Top One Probability is defined as:
PA())
Σ
PA(T)
(10)
If the predicted ranking score for given category is given, then the cross-entropy is equal to the row that wants to put ground truth first the ranking:
Loe
Σ yJ log (,; log P.(),, J=1
(11)
If we want to consider the label correlation in the following sorting function, we only need to change the permutation probability of Packers (j) to the sorting expectation for all categories. we want to consider the label correlation in the later sorting function, we only need to change the permutation probability of (j) the sorting expectation for all categories.
LLsst - log (m)
(12)
Here, we get the expectation permutation म, which is the emotional distance that we define. The transformation of the likelihood function form of cross-entropy into ListMLE form is actually the probability arrangement of prediction, from what is expected to be the first element to expecting all elements to satisfy our defined element arrangement. So in terms of formula, the difference between ECC and ACC is the difference in weight M.,j and w So terms of formula, ACC is transformed into ECC, that is, M, is replaced by emotional distance.
Σj=1 Ei St X W. ECC
(13)
15

---

Table 4: The results of experiments on three single-label classification datasets, FI, EmoSet and UnbiasedEmo, in which experiments carried out on multiple classical baselines based proposed loss function method, and the results on three measures ACC, ECC and EMC are reported.
Backbone
Resnet18
Resnet50
Resnet101
Dataset
Alpha
ACC
ECC
EMC
ACC
ECC
FMC
ACC
ECC
EMC
0
66.2
76.2
51.5
67.3
77.0
51.5
67.9
77.6
52.2
FI
0.2
65.2
76.3
55.8
67.3
77.5
54.9
67.7
77.8
55.2
1.0
63.9
75.1
58.4
65.7
76.2
57.3
65.9
76.6
61.8
0
73.8
82.4
57.7
76.3
84.0
58.5
76.9
84.5
58,9
EmoSet
0.2
73.1
82.5
60.3
75.5
84.1
60.1
75.9
84.3
59.8
1.0
72.2
81.8
65.5
74.5
83.3
61.7
74.5
83.3
61.3
Amusement
Contentment
Awe
Excitement
Fear
Sadness
Disgust
Anger
a-0
Resnett8
Restet50 a-0
Resect50 a-1
Figure 4: In EmoSet, Eq. 14 is compared with the t-SNE graph [37 between alpha equals 0 and 1, where the left two pictures are the results of Resnet18 [3, and the right two pictures are the results of Resnet50.
In this way, we clearly explain the relationship between ACC and cross-entropy, and put forward the expectation of emotional label arrangement on the basis of emotional distance, as transform ACC and cross-entropy into ECC and ListMLE, that is, why ListMLE can be used as the backbone of ACC.
B Experiment Analysis
Here we further explore the performance of ListMLE in visual emotion recognition tasks. The ListMLE loss function is designed to constrain the final prediction probability of the sample to conform to given rank arrangement. According in the measure, we convert it into the label order to cooperate with ListMLE for training. For example, if a image is labeled excitement, the defined labels are sorted from front to back as woweeeconnettettameeeee
the emotional distance in Eq. [ defined
and anger. We mix ListMLE with cross-entropy loss, where a is hyperparameter to adjust the proportion of ListMLE.
L. = LCE + &LLisMMLS
(14)
We compare ListML. and cross-entropy loss on three backbone. The details are in Tab. 4 Similar to the conelusion of previous work ([]. when focusing on the results of misclassification, accuracy will be reduced to some extent. When alpha is 1.0, because accuracy decreases more, ECC is worse than only using cross-entropy loss, but EMC performs best. When alpha equal 0.2, better trade-off is reached between ACC and EMC, and ECC reaches the highest. In order to further emphasize the difference between ListMLE and cross-entropy. we present the visual comparison diagram in Fig. 4 It can be seen that the visualization result of ListMLE has better clustering effect, and the relationship between categories is easier to distinguish, and is more consistent with our defined emotional distance, indicating that ListMLE has learned the label structure information of emotion. Furthermore, this indicates a high correlation between our measures and tasks involving clustering, such emotional image retrieval.
16

---

C Implementation Details
The above experiments Tab. 4 and Tab. are carried out on three backbone, including ResNet18, ResNet50 and ResNet101 [3] All models used pretrained weights in ImageNet [7] before training. For all train images were resized to 256 * 256. To reduce overfitting, we randomly crop the image to 224 224 and flip it horizontally randomly. For test images, we also resize the image to 256 * 256, then make 224*224 crop in the center of it. For EmoSet, follow the experimental setup of the author 44 for the transform of the dataset. We use SGD with a momentum of 0.9 to optimize the network and we use learning rate 0.001. After epochs, we decay the learning rate to 0.0002. Specifically, We warm up in the first epoch, which means the learning rate gradually increases to 0.001 in each iteration. The batch size of training data is 64. We are based on Pytorch for experiments [27
All experiments 1 are performed on two RTX 3090 GPUs. Each of these GPU has 24 GB of memory. For each set of experiments, it takes one to two days to use an RTX 3090.
D Limitation
Finally, we would like discuss the limitations of this work. For the threshold adjustment method in semi-supervised learning, we only introduce simple and direct method. However, using our measures to guide the adjustment of thresholds semi-supervised learning provides novel perspective that deserves further investigation. Moreover, we should explore the application of our new measures in more fields, such as pre-training models and large language models, to fully validate effectiveness of our measures.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose new measures in visual emotion recognition, and verify the effectiveness of our measures through semi cemi-supervised experiments and user study,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitation of our work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to
violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors
17

---

should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We clearly describe the process of constructing the measures and conduct- ing the validation experiments. In addition, a complete set of code is provided in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
18

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submitted the complete code in the supplemental material and the code will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swi.iiossiippcccc.oo for more details. //ipss.c/pubbll wttp:/cccccggguuueee///e/////e// wwcpi///gidssse/eoo Caaeeee/oooSSSmmiieeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
19

---

Answer: [Yes]
Justification: The details of all semi-supervised experiments follow S2-VER 16 and FixMatch [32], and other experimental details are explained in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: As shown Tab. we conducted three sets of experiments each and reported their mean values and standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We explained in Appendix C that all the experiments were carried out on two RTX 3090 GPUs and each set of experiments need use one GPUs for one to two days.And research doesn't 't need more computing resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
20

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, our research conduct in our paper conforms to the NeurIPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not involve privacy disclosure or any other contents that may cause social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk in our paper.
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We clearly marked the work of follow in the paper, and these works are all open source according to rules. Where Fixmatch [32 is in https://
gi gitb.b.r com wewooooeeeeerrrrffiiiaeeeeeeeeee and S2- S2-VER [16 is in https://gith:: com/ expped230/s/2----ee VER
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have released our new measures ECC and EMC, and provided the code in the supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer: [Yes]
Justification: In the paper, we explain the specific way of user study in subsection 4.4
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
There are no potential risks in our research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "oZu3Xrknwr", "reviewer": "Reviewer_Ro51", "summary": "This paper proposed new measures to evaluate the severity of misclassifications in visual emotion recognition. It addresses the limitations of traditional accuracy metrics by considering the psychological similarities between emotions. Utilizing Mikel's emotion wheel, the authors define an emotional distance metric and apply it to create a cost matrix for emotion classification. They introduce the Emotion Confusion Confidence (ECC) and Emotional Mistakes Confidence (EMC) metrics and validate their effectiveness through experimental results and user studies, demonstrating their robustness in semi-supervised learning contexts.", "questions": "1. The authors frequently mention the example: “misclassifying ‘excitement’ as ‘anger’ apparently is more severe than as ‘awe’.” How many methods currently misclassify this category of emotion as anger? Is there corresponding data to support this claim?\n\n2. To fully demonstrate the effectiveness of the proposed new metrics, should these metrics be used with a more diverse set of methods? Additionally, the mainstream methods should be re-evaluated using the proposed metrics and compared with the accuracy (Acc) metric. Analyzing the strengths and weaknesses of methods from multiple dimensions could illustrate the effectiveness and advantages of the proposed evaluation metrics.", "limitations": "The paper acknowledges several limitations, including the use of a simple and direct method for threshold adjustment in semi-supervised learning. While the proposed measures offer a novel perspective, further investigation is needed to refine these methods. Additionally, the application of the new measures should be explored in other fields, such as pre-training models and large language models, to fully validate their effectiveness.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work introduced the concept of mistake cost into visual emotion recognition, enhancing the evaluation of emotion classification methods by considering the psychological impact of misclassifications. By proposing new measures based on Mikel’s emotion wheel, it aids in model selection and threshold adjustment within semi-supervised learning, leading to improved classification performance. The study's validation through user studies confirms that the proposed metrics are consistent with human emotional cognition, offering a robust framework for future research.", "weaknesses": "1. The proposed Emotion Confusion Confidence (ECC) and Emotional Mistakes Confidence (EMC) metrics offer limited contribution as new evaluation standards to the community. Moreover, the exploration of these new metrics and their application in the manuscript is insufficient, resulting in a lack of depth and richness in the content.\n\n2. The manuscript includes too few comparative methods, and the comparison forms and application richness are insufficient, which limits its credibility. In the User Study, only the analysis of ECC is included, with no analysis of EMC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1 Weakness1: Contribution to visual emotion recognition\n- Emotional ambiguity and emotional relevance have always been important issues in the field of emotions, and previous work has been dedicated to solving these problems.Previous works[1] extended single labels into label distributions, or the work of Visual Sentiment Distributions[2][3], Whether it is based on the voted label distribution or the label distribution according to Gaussian fitting, only a rough and inaccurate distribution can be given, and there is a gap between ground truth label distribution and real-world label distribution. It is obviously unreasonable to give the same label distribution to all images of the same category according to Gaussian fitting, while ignoring the differences between the emotions of different images. Conversely, our measure using label rank information can make the ground truth label rank closer to the real-world label rank, or exactly equal, to better guide the model to learn the emotion categories structure.\n- From the perspective of misclassification, emotions have a natural adaptability compared to previous work. There is a clear correlation between different emotions, and there is no need for complex category distance definitions, which is of great significance for the field of visual emotion recognition.\n\n\n## 2 Weakness2: \nAs shown in Tab.1 and Tab.2 of our rebuttal material. We conducted supplementary experiments in semi-supervised settings and also tested our metrics on image retrieval and binary classification tasks. In the user study, we primarily analyzed ECC, which is equivalent to analyzing ACC+EMC. Previous work on misclassification has always aimed to reduce the severity of misclassification while ensuring the accuracy of ACC. Indeed, both ACC and EMC are generally used together to evaluate the model.\n\n## 3 Question1：\nFrom the confusion matrix in Fig.1 of our rebuttal material, we can see that a significant portion of emotions are misclassified into categories with opposite polarity. For example, using OSANet, when the emotional polarity is positive, an average of 6.75% of the samples are misclassified as negative emotions. When the emotional polarity is negative, an average of 19.25% of the samples are misclassified as positive emotions.\n\n## 4 Question2:\nIn fact, we reassessed the current mainstream methods in Figure 1 of our manuscripts, where the relationship between ACC and EMC is shown. \n\n## References\n[1].Yang, Jufeng, et al. \"Weakly supervised coupled networks for visual sentiment analysis.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[2].Yang, Jufeng, Dongyu She, and Ming Sun. \"Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network.\" IJCAI. 2017.\n\n[3].Yang, Jufeng, Ming Sun, and Xiaoxiao Sun. \"Learning visual sentiment distributions via augmented conditional probability neural network.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bPCtMSVLrT", "reviewer": "Reviewer_133Y", "summary": "The paper proposes a new measure for emotion recognition performance based on Mikel’s emotion wheel. The measure takes the distance between emotions into account. Experiments in semi-supervised learning on emotion recognition and user study demonstrate the effectiveness and superiority of the proposed metrics over accuracy.", "questions": "1. Line 304-307, how do you select the values for different hyperparameters?\n2. How do you measure the standard deviations in tables 1 and 2?", "limitations": "Please refer to weaknesses", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Good performance compared with baselines in tables 1 and 2. Besides, standard deviations are reported which demonstrates the improvement is significant.\n2. The idea of designing new measures for ER is novel and easy to follow. Though, it may not be applicable to the basic emotion space.", "weaknesses": "1. Few baselines are compared. Tables 1 and 2 are the main experiments while only one and two baselines are discussed and compared.\n2. Section 4.2.1, the setting of pseudo labeling evaluation is confusing to me. How did you get the accuracy for models with different label num? Is there an official test set in FI/EmoSet for evaluation? Or did you use the whole set? If that's the case, what’s the difference between pseudo labeling evaluation and simple model training/testing evaluation?\n3. Section 4.3 is also unclear to me. Can you clarify the annotation instruction in the user study? I am not sure how participants vote for models with higher ECC or ACC.\n4. Mikel’s emotion distance is not applicable to Ekman six basic emotions which is widely used in most categorical emotion datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1 Weakness1:\nWe have added comparative methods for the semi-supervised experiments in Tab.1 of our rebuttal material.\n\n## 2 Weakness2: \nIn fact, it is a semi-supervised learning experiment setting. We divided the training set according to a predetermined number of labeled samples into labeled training samples and samples we consider unlabeled. Then, we trained on the partially labeled training samples and a large number of unlabeled training samples and tested on the official test sets of FI/Emoset. During training, we can assess the model’s ability to label pseudo-labels by obtaining the true labels of the pseudo-labels. In the early stages of using different functions (CE and mixed functions), the model trained with CE loss has higher ACC for labeled pseudo-labels but lower EMC. In contrast, the model trained with mixed loss has lower ACC but higher EMC, which means the model can learn useful knowledge from incorrectly labeled tags that are similar to the true labels, thereby improving the final semi-supervised classification accuracy.\n\n\n## 3 Weakness3: \nWe added a description of the user study in Fig.2 of our rebuttal material. First, we selected samples that both models misclassified. Then, we presented the classification results of the two models, along with ‘Indistinct’ as three options, and asked users to choose which emotion was closer to the true emotion of the image\n\n\n## 4 Weakness4: \nOur metrics primarily emphasize the similarity between emotional labels, as there are differences in the quality of misclassifications in emotions. However, Mikel’s wheel is off the shelf. Objectively speaking, Ekman also has emotional label similarities. It can be used as long as an appropriate model is defined just like Mikel’s wheel.\n\n\n## 5 Question1：\nBased on existing semi-supervised work such as S2VER, FixMatch, and FlexMatch, we chose 0.98 and 0.7 as the upper and lower bounds of the threshold, and $\\tau$ is 0.95. For  e , we derived empirical values for each dataset. For example, in the FI dataset, EMC is usually around 0.5, and in Emoset, EMC is usually around 0.6, so we set them to 0.5 and 0.4, respectively.\n\n## 6 Question2:\nWe followed the previous papers FlexMatch and S2VER, conducting the experiments three times under the same experimental settings with different random seeds, and then we calculated the standard deviation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TUrLsKOZXF", "reviewer": "Reviewer_MhCQ", "summary": "This paper proposes a novel evaluation approach based on Mikel’s emotion wheel from psychology, which considers the \"emotional distance\" between different emotions. It is claimed that the measure design considering the granularity of the emotions can be a better metric to evaluate visual emotion recognition. Experimental results adopting some of the semi-supervised learning methods and user studies show that the proposed metric is more effective than accuracy in evaluating performance and aligns with human emotional cognition.", "questions": "- Have you considered using emotional lexicons that include a broader range of emotion classes?\n-How were the results of S2Ver on Emoset produced?", "limitations": "- Emotion and overall affective computing always have a social impact but it is not discussed in this paper.\n- Limitations are discussed in the appendix, while can be improved.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The first attempt to consider Emotion Granularity in the big datasets and propose a new evaluation procedure. The idea is innovative and can contribute but also see the weaknesses, especially in the emotion model considered as well as the comparisons.\n- User study with 30 participants confirming the results are inline with human cognition.\n-", "weaknesses": "-The use of Mikel's Wheel, a relatively less-studied emotion model in the context of building computational methods for automatic recognition, limits the study's impact. The first work utilizing this model dates back to 2016. Given the extensive body of work based on e.g., Ekman's model, the significance of this study is unclear.\n- An important aspect completely overlooked in this paper is the frequent evaluation mechanism involving binary classification for each emotion class. Multiple emotions can be experienced simultaneously, making a multi-class classification task inappropriate. Consequently, accuracy (ACC) should be applied to individual emotion classes, where the positive class indicates the presence of a specific emotion and the negative class indicates its absence. Additionally, several methods have adopted the use of F1-macro and F1-weighted metrics, which were neither discussed nor applied in this study.\n-Another issue that caught my attention is the limited range of researchers cited in the related work section regarding the use of Mikel's Wheel. Only the same group of researchers (e.g., Jingyuan Yang, Dongyu She) is cited, indicating that the impact of this work may be limited.\n-The context is crucial in defining emotional distances, which seems to be missed in this study.\n-References to CES and DES are missing (see Line 163).\n-I expected to see more comprehensive comparisons in Tables 1 and 2. For example, comparisons could include works like \"Li, J., Xiong, C., Hoi, S.C.: Comatch: Semi-supervised learning with contrastive graph regularization. In: ICCV (2021).\" Currently, the comparisons are not as extensive as to confirm the claims.\n- I do not find the venue for this submission appropriate. This work should be submitted to conferences focused on affective computing, such as ACII, where the community can discuss the theoretical underpinnings of the metric in conjunction with psychological claims and findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness:\n- First, we have added more extensive semi-supervised comparative experiments in Tab.1 of our rebuttal material, as well as experiments in Tab.2 of our rebuttal material on image retrieval tasks and emotion binary classification tasks.\n\n- Secondly, we strongly agree with your view that “human emotions are not singular,people experience a range of emotions.” Emotions have similarities, so accuracy in traditional classification tasks is not entirely applicable in emotion classification. Our metrics and the loss functions we provided incorporate this design. However, the task you mentioned is essentially a multi-label classification task, similar to emotion distribution learning. Both tasks face challenges in data annotation, and label quality may not always be reliable. It is difficult to determine which specific emotions are present in an image and how the emotional distribution of an image looks.In response to the first point of Reviewer 1, we also addressed this issue. Previous emotion distribution learning has often provided rough labels, and directly using KL loss to force the model’s final distribution to match the ground truth distribution is unreasonable. Therefore, we believe that constraining emotions through emotional ranking at the loss level is more appropriate. This approach helps the model learn which emotions are predominant in an image and which emotions are more prevalent relative to others.\n\n- Discussion of F1-macro and F1-weighted metrics: In fact, these are common metrics in machine learning for misclassification, but they are fundamentally different from our metrics. F1-macro is the average of the F1 scores for each category, while F1-weighted is the weighted sum of the F1 scores for each category. However, the precision and recall in F1 scores are calculated based on correct classifications, which differs from the research significance of our metrics. We aim to highlight the correlation of labels, where each category has a proximity relationship with other categories, but F1 scores cannot reflect this relationship.\n\n- However, we observed many works on facial expression recognition and error severity classification at NeurIPS. Our work could inspire thinking in these two fields and even broader areas. Therefore, we believe that submitting to NeurIPS is still meaningful.\n\n- Finally, we want to discuss the Mikel’s wheel and Ekman models. Any emotion model has varying distances between related emotional labels. Ekman does not have a mature distance definition like Mikel’s wheel, so we did not conduct experiments on Ekman. However, if a reasonable six-category distance for Ekman is defined, or if we define emotional distances based on Mikel’s wheel for the emotions in Ekman, the same metrics can be used. Additionally, Jingyuan Yang and Dongyu She are not a research group. Mikel’s wheel is the foundational model for the eight-category visual emotion model. Almost all visual emotion tasks are now conducted on FI, and further works like EmoGen and EmoEdit, which are based on the eight-category model, are built on this. Therefore, there is no need to worry about the limited scope of researchers using Mikel’s wheel.\n\n## Question:\n- Yes, we have considered classification based on Plutchik’s model, which is a more fine-grained 24-category model built on the eight primary categories. If using such a model, similar to Ekman’s, it would only require defining a reasonable emotional distance model, such as assigning a weight to each layer of emotions in Plutchik’s model.\n\n- We processed EmoSet in the same way as FI because both are eight-category models based on Mikel’s wheel. Aside from the difference in dataset size, there is no significant difference between the two datasets during training. All our experiments on EmoSet used the same parameter settings as FI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZQGhnSjaIx", "reviewer": "Reviewer_jkV6", "summary": "The principal subject of this paper is the definition of the concept of misclassification in the field of visual emotion recognition, and the proposal of a novel evaluation method based on Mikel's Wheel distance to assess the degree of misclassification in methods of visual emotion classification. The paper also discusses the application of these evaluation methods in selecting models and adjusting label thresholds in semi-supervised learning tasks.", "questions": "Same problems as mentioned above.", "limitations": "The authors mention some limitations in semi-supervised learning. However, it is recommended that they include a discussion of related work on supervised learning.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The primary contribution of this paper lies in the introduction of a novel emotional recognition evaluation framework that accounts for emotional similarity and the severity of misclassification. Furthermore, the paper applies this framework within the context of semi-supervised learning tasks.", "weaknesses": "1. In the related work section, subsections 2.2 and 2.3 appear to be erroneously treated as separate entities, whereas they should logically be combined into a single subsection, given their interconnected content.\n\n2. Within the Emotion Wheel, adjacent emotions typically reflect interconnected affective experiences. However, the authors' adjustments to the numbering or distances on the Emotion Wheel, based on emotional polarity, have regrettably separated correlated emotion pairs such as excitement and fear, as well as amusement and anger, while paradoxically bringing the unrelated emotions of amusement and fear closer together. This configuration introduces substantial issues.\n\n3. The assertions depicted in Figure 1 raise concerns; trends across various models, with the exception of the final MAPNET, demonstrate substantial consistency. Moreover, given the variations in model architectures, it is illogical to evaluate the differences in ACC and EMC performance based purely on release dates or ACC statistics.\n\n4. In practical contexts, the emphasis on errors does not necessarily correlate with inaccuracies in polarity, suggesting that the EMC's utility may be limited.\n\n5. While semi-supervised learning is utilized due to the difficulties in annotating data for emotion recognition tasks, the discussion of new evaluation metrics necessitates more accurate methodologies to effectively demonstrate these metrics' utility. Therefore, integrating experiments with fully supervised learning is advised.\n\n6. In Section 4.2.1, the authors assert that models trained using CE loss achieve superior ACC performance, whereas those trained with a composite loss function, despite lower ACC, more effectively address misclassification and produce higher-quality pseudo-labels. Nevertheless, it remains ambiguous why subsequent sections claim an accuracy improvement of 1% over $S^2$-VER using our methodology.\n7. Table 3's presentation lacks clarity. The meaning of \"Our Rank\" is unclear, and the justification for comparing RA w/o R1 with RE w/o R1 is not thoroughly elucidated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1 Weakness 2，7:\nAs shown in Sec.3 of our rebuttal material，Emotional polarity is extremely important for emotion classification tasks. Therefore, we separate emotions of different polarities based on emotional polarity, but we do not shorten the distance between unrelated emotions. For example, before modification, the distance between amusement and fear was 4, and after increasing the distance, it became 8. To prove the effectiveness of our proposed changes, we followed previous work on misclassification[1] to verify the effectiveness of our proposed emotional distances. We converted these distances into ranking and trained with the mixed loss function based on different ranking.\n\n## 2 Weakness3: \nAlthough the methods in Fig.1 of our manuscript are listed chronologically, the later methods are not necessarily developed based on the earlier ones. For example, when comparing OSANet and ResNet101, although OSANet’s accuracy is much higher than that of ResNet101 and ResNet50, its error severity is also higher. Similarly, even though WSCNet’s accuracy is comparable to ResNet50 and ResNet101, its error severity is very high. With this figure, we want to demonstrate that the current mainstream emotion classification methods do not incorporate the important factor of emotional error severity into the method design and evaluation system.\n\n## 3 Weakness4: \nFrom the confusion matrix in Fig.1 of our rebuttal material, it can be seen that, in general, misclassification into categories with the same polarity occurs more often than into categories with opposite polarity. Taking OSANet as an example, although there is one more misclassification to emotion categories with opposite polarities (4 categories) than to those with the same polarities (3 categories), the proportion of misclassifications to categories with the same emotion polarity is 22.625%, while it is 13% for opposite polarities.This indicates that images with the same emotional polarity often express similar patterns and have relatively close emotions. This shows that there is a close relationship between emotional polarity and emotional misclassification. We also demonstrated through the validation experiment in Section 4.3 that the emotion rank, derived from the emotion distance we defined, has good properties.\n\n\n## 4 Weakness5: \nWe combined the experiments with fully supervised learning to analyze the emotion retrieval and binary classification tasks. As shown in the table, under the same model architecture, when the ACC is not significantly different, models with higher EMC, due to the incorporation of the concept of label distance, tend to perform better on eight retrieval metrics than models with higher ACC. In the binary classification task, models with higher EMC also show better ACC2 performance.\n\n\n## 5 Weakness6:\nIn semi-supervised training, the quality of pseudo-labels determines the final performance. Although models trained with CE loss have higher ACC, the pseudo-labels may be misclassified into emotion-irrelevant labels. For example, an “amusement” sample might be labeled with a pseudo-label of “fear,” which is disastrous for model training. On the other hand, while models trained with a combined function may have lower ACC, they have higher EMC. This means that although the pseudo-labels may be incorrect, they are likely to be mislabeled into categories close to the true category. Under this positive reinforcement, models with higher EMC are trained with continuously generated high-quality pseudo-labels, ultimately achieving higher ACC than the former.\n\n## References\n[1].Bertinetto, Luca, et al. \"Making better mistakes: Leveraging class hierarchies with deep networks.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## 1 Semi-supervised learning supplementary experiment
We added comparison methods for semi-supervised experiments, including CoMatch[1], SimMatch[2], SoftMatch[3] and FreeMatch[4]. Additionally, we combined the two tables into a new one. Please note that comparing other methods with our method in section 4.2.2 is unfair, as our method only involves adjusting the threshold. Nevertheless, our threshold-adjusting method is still better than some methods. Furthermore, the method in section 4.2.1 performs the best among all methods.


## 2 Full supervised learning supplementary experiment
We have demonstrated the various good properties included in our metric through fully supervised experiments. After training on base models (ResNet18, ResNet50, ResNet101) using CE loss and a mixed loss based on the metric, we evaluated the models on image retrieval and binary classification tasks. The experimental results show that models with higher EMC, meaning models with good misclassification properties, generally perform better in image retrieval and binary classification tasks. 
- Image retrieval is essentially a type of metric learning. Boudiaf et al.[5] demonstrated that the final prediction score for each image category in metric learning actually reflects the mutual information between image features and labels. Yao et al.[6] experimentally verified that class-ordered pair-wise loss is effective for emotion retrieval tasks. Our metric aims to emphasize the spatial semantic structure of emotion labels, which essentially involves constraining the relationships between the prediction scores of each category, thus constraining the mutual information between image features and labels. This is why our metric can better distinguish the distances between different image features in image retrieval tasks.
- Furthermore, ACC2 is an important metric in the field of emotions, usually reflecting the model's ability in emotion binary classification (cite papers using ACC2). ACC2 actually corresponds to the classification results of the upper left ([0,3],[0,3]) matrix and the lower right ([4,8],[4,8]) matrix of the confusion matrix in our paper. Our metric makes more detailed considerations based on ACC2 and ACC. 

In conclusion, usually, when the ACC is similar, a higher EMC indicates better model performance.

## 3 Explanation of the Importance of Emotional Polarity
Additionally, we illustrate the importance of emotional polarity by analyzing the classification confusion matrices of three mainstream methods (OSANet, SimEmo, MSPNet). The confusion matrices show that, in most cases, misclassifications tend to occur within categories with the same emotional polarity. However, some categories are also misclassified into those with opposite polarity. A considerable number of studies, such as S2VER, EASE[7] and TSL[8], address issues in the emotional domain through emotional polarity. This indicates that emotional polarity is a crucial feature of emotions, both from the perspective of human cognition and experimental results.

## 4 User Study
We provided a schematic diagram of the User study process. We selected images that were misclassified by both models, then created a questionnaire with the two misclassification results and "Indistinct", asking users to choose “which emotion is closer to the picture’s emotion?”

## References
[1].Li, Junnan, Caiming Xiong, and Steven CH Hoi. "Comatch: Semi-supervised learning with contrastive graph regularization." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[2].Zheng, Mingkai, et al. "Simmatch: Semi-supervised learning with similarity matching." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3].Chen, Hao, et al. "SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning." The Eleventh International Conference on Learning Representations.

[4].Wang, Yidong, et al. "FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning." The Eleventh International Conference on Learning Representations. 2022.

[5].Bertinetto, Luca, et al. "Making better mistakes: Leveraging class hierarchies with deep networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[6].Yao, Xingxu, et al. "Adaptive deep metric learning for affective image retrieval and classification." IEEE Transactions on Multimedia 23 (2020): 1640-1653.

[7].Wang, Lijuan, et al. "Ease: Robust facial expression recognition via emotion ambiguity-sensitive cooperative networks." Proceedings of the 30th ACM international conference on multimedia. 2022.

[8].Zhang, Zhicheng, and Jufeng Yang. "Temporal sentiment localization: Listen and look in untrimmed videos." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

Meta-review: After the rebuttal period, the paper received three ratings of borderline accept and one of borderline reject.

The paper proposes to evaluate the performance in visual emotion recognition by considering the distance between different emotions on Mikel’s emotion wheel. Experimental results are conducted in semi-supervised learning on emotion recognition. The reviewers overall agree the approach is novel and innovative. Strengths of the paper include aiding in model selection and threshold adjustment with semi-supervised learning, validation through user studies, improved performance over baselines, and the novelty of new measure is good.

The original concerns, raised by the reviewers, include too few comparisons, confusing description of the evaluation and annotation, applicability of Mikel’s wheel to Ekman’s six basic emotions, multiple emotions occurring at once not being considered, separation of related emotions, and no fully supervised learning experiments.

The authors and reviewers took part in a good discussion during the rebuttal phase, with the authors providing significant clarification and new experimental results resulting in multiple reviewers increasing their scores. After the rebuttal phase, the paper strengths outweigh weaknesses as many of them were addressed. This paper can be accepted for publication.
Decision: Accept (poster)