Relative Uncertainty Learning for Facial Expression Recognition
Authors: Yuhang Zhang
Chengrui Wang
Weihong Deng
Venue: NeurIPS 2021 Poster (2021)
Keywords: uncertainty learning
facial expression recognition
label noise

Abstract: In facial expression recognition (FER), the uncertainties introduced by inherent noises like ambiguous facial expressions and inconsistent labels raise concerns about the credibility of recognition results. To quantify these uncertainties and achieve good performance under noisy data, we regard uncertainty as a relative concept and propose an innovative uncertainty learning method called Relative Uncertainty Learning (RUL). Rather than assuming Gaussian uncertainty distributions for all datasets, RUL builds an extra branch to learn uncertainty from the relative difficulty of samples by feature mixup. Specifically, we use uncertainties as weights to mix facial features and design an add-up loss to encourage uncertainty learning. It is easy to implement and adds little or no extra computation overhead. Extensive experiments show that RUL outperforms state-of-the-art FER uncertainty learning methods in both real-world and synthetic noisy FER datasets. Besides, RUL also works well on other datasets such as CIFAR and Tiny ImageNet. The code is available at https://github.com/zyh-uaiaaaa/Relative-Uncertainty-Learning.

Relative Uncertainty Learning for Facial Expression Recognition
Yuhang Zhang, Chengrui Wang, Weihong Deng Beijing University of Posts and Telecommunications zyhzyh@bupt. edu. cn, crwangebupt. edu. edu. en, whdengobupt. edu, en
Abstract
facial expression recognition (FER), the uncertainties introduced by inher- ent noises like ambiguous facial expressions and inconsistent labels raise con- cerns about the credibility of recognition results. To quantify these uncertain- ties and achieve good performance under noisy data, we regard uncertainty as a relative concept and propose an innovative uncertainty learning method called Relative Uncertainty Learning (RUL). Rather than assuming Gaussian uncertainty distributions for all datasets, RUL builds an extra branch to learn uncertainty from the relative difficulty of samples by feature mixup. Specif- ically, we use uncertainties as weights to mix facial features and design an add-up loss encourage uncertainty learning. is casy to implement and adds little or no extra computation overhead. Extensive experiments show that RUL outperforms state-of-the-art FER uncertainty learning methods in both real- world and synthetic noisy FER datasets. Besides, RUL also works well on other datasets such CIFAR and Tiny ImageNet. The code is available Learning.
www.aauaaaaaaa www.aaaaaeeeeeeeeeaeaaaaaaaeeeee https://githin ottp:////oooaaaaaaaaaaaaaaaoaooa
Introduction
Although well-trained models can give high-confidence inferences directly, the inference results may actually be wrong and cause serious consequences 22 Therefore, the concept of uncertainty is widely used in measuring how well the model trusts its inference results, which has become research hotspot in trustworthy machine learning model assigns high level uncertainty to its erroneous predictions so that humans can intervene to avoid lots of disasters. As described by Kiureghian er al. (25), the uncertainties in machine learning can be split into aleatoric uncertainty and epistemic uncertainty, which are also called data uncertainty and model uncertainty. In this paper, we focus on quantifying data uncertainty as model uncertainty can be eliminated by introducing more training data. Concretely, data uncertainty in facial expression recognition (FER) is mainly caused by ambiguous facial expressions and the subjectiveness of annotators, which blocks the improvement of recognition performance [47].
Il
26 16]
is expected that reliable
Recent studies propose variety of methods to solve the uncertainty learning problem in the face recognition field. PFE 37 introduces a new branch to learn the uncertainty of face recognition and recognizes faces by measuring the similarity between two Gaussian distributions. Chang al. further propose DUL to learn feature and uncertainty simultaneously in order to improve facial feature learning. When comes to FER, SCN [47 uses a fully-connected layer to learn an importance weight for each image and suppresses uncertainties according to the learned weights. However, the strong learning ability of neural networks will deteriorate the uncertainty learning branch. One solution is adding regularization otherwise, it would lead to underfitting. For example, SCN uses margin loss to keep the gap between certain and uncertain images, but choosing suitable gap value is non-trivial. DUL assumes features
force the model to learn uncertainty, which needs to be tuned carefully,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

to follow Gaussian distribution, but the real distribution of the dataset is not always following such an assumption. Therefore, is still a challenge learn uncertainty distributions of real datasets without deteriorating feature learning.
To address the problem, we perceive data uncertainty from different point of view. We realize that data uncertainty - the difficulty to classify a sample correctly is actually a relative concept. More specifically, the difficulty of classification is a person's subjective feeling that comes from comparison. One can only know whether a sample is easy or not according to a reference. Inspired by this idea, we propose an innovative method called Relative Uncertainty Learning (RUL) to help deep learning models to learn uncertainty for each sample. Concretely, we build a new branch to model the uncertainties of input images and utilize the uncertainties as weights to mix two features of different labels. Through an add-up loss function, the model is encouraged to recognize two expressions simultaneously from the mixed features, which enables the model to learn uncertainty comparatively while minimizing the total loss. As discussed in Section (33] the uncertainty learning branch will assign large uncertainty values to uncertain facial expression images while small uncertainty values to certain images.
The main contributions of this work are follows:
We perceive uncertainty from new perspective and propose innovative uncertainty learning method to learn uncertainty from the relative difficulty of two samples.
We get state-of-the-ar performance in both real-world and synthetic noisy FER datasets.
RUL does not need prior knowledge of the dataset uncertainty distribution and can be easily applied to different classification tasks with low computation cost.
2
Related Work
Uncertainty learning in the face field Uncertainties can be mainly categorized into model uncertainty which accounts for uncertainty in the model parameters and data uncertainty which is the noise inherent in the dataset [22). In the face recognition field, uncertainty learning is important for learning discriminative feature embeddings of noisy face images. Because noisy face images are usually out of the cluster and have large variances in the latent embedding space, which might cause wrong recognition 337. There have been several proposals to solve the uncertainty learning problem in
23 have tried to analyze and learn face representations using model uncertainty. Gong el al. 13 estimate the capacity of a given face representation by explicitly accounting for the manifold structure, model uncertainty and data variability. Khan el al. (23 present a framework for class imbalance learning based on model uncertainty. Zafar et al. [5]] cope with false positives through employing model uncertainty to improve the efficacy of face recognition systems. Thereinto the methods considering data uncertainty learning. Shi et al. (37) use PFE learn a variance for each fixed feature and then measure the likelihood of each positive face pair of (I;, 2z,) sharing the same distribution of latent embedding p(z, 2,). Chang et al. (4) further propose to learn feature and uncertainty simultaneously by encoding the latent feature embedding as Gaussian distribution. In the facial expression recognition task, Wang et al. [47] try learn an attention weight for each image, then utilize to weigh cross-entropy loss in order to suppress the influence of noisy samples. She et al. (36] propose to use several branches to model the latent label distribution of facial expression images and use cosine similarity to capture the uncertain images.
face recognition tasks B 51 23
37 4
Some of them [13, 51
Noisy dataset training With the enhancement of deep learning network capabilities, the datasets for facial expression recognition (FER) are getting larger and larger. For the large-scale FER datasets 43 54 D 9 27, due the ambiguity of expressions and the subjectiveness of human annotators, it is extremely hard to get high-quality labels. Training with label noise has been studied for long time. The basic idea to enable the model to access the quality of the labels or estimate the
noise distribution using a small set of clean dataset [5] 28 39 45). framework to use information from both small clean dataset and label relations in knowledge graphs to prevent the model learning from noisy labels. Veit el al. 45] use small amount of clean annotations to reduce the noisy samples before fine-tuning the network using both the clean set and
Li et al. (28) design a distillation
the full set. Other methods do not use small clean dataset, but usually assume extra constraints

---

labela
j0
pairs
loss
CNN Backbone
labelj
Figure 1: Overview of our Relative Uncertainty Learning method. We use two branches to learn facial features μ and uncertainty vectors simultaneously. Then normalize uncertainty vectors with their counterparts sampled from shuffled indexes. We compute the mean of the uncertainty vectors the uncertainty values. Next, mix the facial features according the mean uncertainty values. Note that the easy sample will get a small uncertainty value because its facial feature is very obvious for recognition. In contrast, the ambiguous sample lends get a large uncertainty value in order to make il easy for the model to recognize its corresponding label from the mixed features. Finally, we use the add-up loss to encourage the model recognize the two expressions from the mixed feature, which implicitly forces the model to learn reliable uncertainty values.
the noisy samples. Natarajan et al. [34] design specific loss function for randomly flipped labels. Goldberger et [12] explicitly model the noises in datasets by softmax layer which connects the correct labels to the noisy ones. Without collecting clean set of dataset or using prior knowledge of noise distribution, our method learns uncertainty through the relativity of the given dataset and suppresses the most uncertain samples to get good performance under label noise.
3 Proposed Method
In order to reduce the negative impact of the strong learning ability of neural networks and learn uncertainty values closer the real dataset uncertainty distribution, we propose simple and effective method named Relative Uncertainty Learning (RUL). ). In this section, we first provide the motivation behind our method. Then demonstrate the details of RUL. Finally, explain why RUL is effective uncertainty learning.
3.1 Motivation
The uncertainty learning methods mentioned before suffer from the strong learning ability of deep neural networks more less. For example, the introduced variances in DUL are equivalent some noises being added to feature points, so the model will learn very small variances for all feature points making the estimation of uncertainty less reliable. SCN also learns large importance weights for noisy samples to obtain small cross-entropy loss values as these noisy images will be memorized by the networks after training for several epochs.
In order to prevent the strong learning ability of neural networks from degrading the uncertainty learning branch, we perceive uncertainty from relative point of view. We find that learning uncertainty through comparison forms natural regularization because if the model assigns a very small uncertainty value for one image of the comparison pair, then it means the counterpart image gets large uncertainty value which prevents the model learning small uncertainty values for all images. Learning uncertainty through relativity also conforms to human cognition it is very difficult for person define which sample certain and which sample is uncertain without using one image reference. In the following section, we thoroughly illustrate the details of our proposed RUL.

---

3.2 Overview of Relative Uncertainty Learning
In this section, we illustrate the structure of our proposed RUL. The pipeline of RUL shown in Figure
Two Branches Model Given batch of face images X, assuming the conventional feature extraction module (X) can be decomposed as
fx - fe" (x(-
(1)
we first extract general facial features Fg from the penultimate feature extraction layer denoted as Fg f 1 x [50]. Then, in order to learn facial features and uncertainty values simultaneously,
we drop the layer indexed by I, i.e., f(() and build two separate branches fe, (500011))/ (x)) and
fo, (18 ((((0(((()) (x)) produce facial features F [11, 142 [4n] E RDN and uncertainty vectors U 1 2 N R D, N denote output dimension and batch size separately.
Relative Uncertainty Learning Inspired by the relativity of the uncertainty concept and the mixup we mix two different facial features according to their uncertainty values which enables the FER model to learn uncertainty through the relativity of different samples. First, we mix mini-batch with their shuffled ones which results in the facial images of the whole dataset being compared with each other during the training process because of the randomness introduced by the dataloader and the shuffle of mini-batches. Before mixing features, we first normalize the uncertainty vectors element-wiss to compare two features with each other following Equation
method [53 42 46
0,
o, oo o: oo
o, and oy are the uncertainty vectors for image i and its counterpart image j sampled from the shuffled indexes, note that they have different labels. This operation not only forms natural regularization as the model can not learn small uncertainty values for both features, but also benefits the training process as extreme uncertainty values might make the training process unstable or even the loss can not converge. As mentioned by DUL E, the mean of the predicted uncertainty vectors à; E R can be viewed as an approximated measurement of the estimated uncertainty for image i, we also use the mean of à; as learned uncertainty value for image i denoted as o &imann E RR. Then we get uncertainty value for each input image denoted as U = |ō Imean 0 2monn> 1 .... Ô Nmeanl E RIXN
We then mix features according to their uncertainty values following Equation
j T= Ōimcan p4 + ō Jmccn H4j
(3)
#4 and μj represent embedding features of the two images, oxmeury and δ Juean represent their corresponding uncertainty values. We introduce negligible computation overhead as we only mix facial features according to their learned uncertainty values in the training phase. Furthermore, the mix feature part will be removed for deployment. Therefore, our network is end-to-end and adds no extra cost on inference.
Classification Loss We design an add-up loss to encourage the uncertainty learning branch to learn different uncertainty values for different facial images. Specifically, add-up loss requires the FER model recognize two expressions of input images simultaneously from mixed features. Since çú the mixed feature, we feed ğu to the classifier:
W 1J De e.W
W. Le w.
Listal = -
Enoo
+ log
(4)
The loss function adds up the loss of recognizing ğ as class and as class j. Vi. Yt means label,, labely, W. is the c-th classifier and C' means the total number of expression classes. The loss function forces the model to recognize two expressions equally from the mixed features. When mixing two facial features, there will be a relatively easy facial feature for expression recognition, and the other will be relatively hard. For simplicity, we note μ4 as the easy facial feature. Our model will learn small uncertainty value for μæ to mix with μıj as μ4; is easy to recognize, the model can already get small classification loss with its label y, after training for several epochs, even μı; only takes up a small amount of the mixed feature ft. While for the hard image j. it needs to take up a large amount of the mixed feature f to enable the model find useful feature to get a small classification loss with label yj.

---

anger
Lange
anger
Langer -0,49c-2
2.73e-5
Leotal -2.17
Lutal -7.17
sad
sad
0.5
0.8
sad =12.17
Isad =7.17
Figure 2: An experiment show why RUL is effective in uncertainty learning. We mix different features with different weights and print the losses of recognizing the mixed feature to different labels. Visually, the face with anger expression is the certain one, and the face with sad expression is the uncertain one. In the process of feature mixup, RUL will assign a larger weight to the uncertain one minimize the total loss.
3.3 Why RUL is Effective
In this section, illustrate why RUL is effective in uncertainty learning. For visualization simplicity, we plot the mixed images instead of the mixed features in Figure At the start of the training, RUL might learn similar uncertainty values for all images, which shown in the left part of Figure 2 As the certain facial expression image with anger label in Figure can can be easily recognized from the mixed feature, the model can already get a small loss value with anger label while gets large loss value with sad label. In order to get the minimum total loss, the model needs to recognize the sad expression from the mixed feature more effectively. As training progress. the mix weight (uncertainty value) of the uncertain sad image will increase to enable the model to recognize the sad expression from the mixed feature. Thus, RUL will learn larger uncertainty values for uncertain images.
4 Experiments
In this section, we first describe three in-the-wild FER datasets and our implementation details. We then evaluate our proposed method on these datasets. We compare our method with sfate-of-the-art uncertainty learning methods the accuracy, noisy label training, and accuracy with rejection [41]. Furthermore, we provide qualitative and guantitative analysis to show what the meaning of our learned uncertainty values. Last, compare also demonstrate that RUL is applicable to datasets in different fields. We also carry out an ablation study find the effectiveness of the mix part in RUL.
method to other kate-of-the-aat FER methods and
4.1 Datasets
To evaluate the effectiveness of our model, we select three widely used in-the-wild FER datasets. Compared with FER datasets collected in the laboratory, they have more noises like noisy labels and ambiguous expressions. So they are more complex, variable, and challenging for expression recognition tasks. The details are as follows.
RAF-DB 27 is crowdsourced facial expression dataset that contains 29672 facial images annotated with basic or compound expressions by 40 trained annotators. this work, we use facial images with basic expressions
and neutral expression, including 12271 images training data and 3068
images as test data.
FER2013 [44] consists of 35,887 grayscale 48x48 pixel images in total, with 38,709 training samples, 3,589 public test samples, and 3,589 private test samples. These facial images are classified into emotion classes, same as RAF-DB.
currently the largest FER dataset, including 440,000 images. The images are collected from the Internet by querying the major search engines with 1,250 emotion-related keywords. There are around 280,000 training images and 4000 testing images annotated by human. They are annotated to 8 emotion classes (7 emotion classes same as RAF-DB and Contempt class).
AffectNet

---

4.2 Implementation Details
Given a batch of images from RAF-DB, we first resize them to 224x224 pixels and use ResNet- 18 [8]. which pretrained on Ms-Celeb- 1M 17 as backbone. The two branches have the same structure as BatchNorm2d [20] Dropout 38 Fatten-FFCBatchNoomml We set dropout rate as 0.4, output dimension as 64. Note that the second branch outputs log o instead of in order stabilize the training following [4]. The model is trained in an end-to-end manner with single GTX 1080ti GPU for 70 epochs with batch size of 64. We also utilize an Adam optimizer [24] with weight decay of 0.0001. The learning rate is initialized as 0,0002 except the last fully connected layer for classification, which is 0.002. We use ExponentialL. [30] learning rate scheduler with gamma 0.9 to decrease the learning rate after each epoch. The setting under AffectNet dataset is the same as RAF-DB. As for FER2013, we use the same setting as [49 as the image in FER2013 is 48x48 pixel grayscale image.
4.3 Evaluation on Label Noises
Both ambiguous facial expressions and inconsistent annotations will cause label noises in FER datasets. We carry out extensive experiments to demonstrate the improvement of RUL compared with ofate-of-thhear noise-tolerant FER methods. Following SCN [47]. we randomly choose 10%, 20%, 30% of training data and flip their labels to other expression categories randomly. To make fair comparison with SCN, we also consider the most uncertain samples in the training process as samples with label noises and if the maximum prediction probability is higher than the one of the given label with threshold (set to 0.2 in all cases), we choose to rely on the FER model and change the label to the index of the maximum prediction probability.
We all experiments three times and compute the mean and standard variance of the results. As shown in Table 1 RUL outperforms SCN and DUL under all circumstances. RUL improves test accuracy by 4.30%, 4.32%, 4.60% with noise ratio of 10%, 20% and 30% RAF-DB, 1.01 %, 1.56%, 2.01% % on FER2013 and 1. 1.94%, 2.01%. 4.05% on AffectNet compared with SCN. Experiment results show that the benefit from RUL becomes more obvious as the noise ratio increases up which means RUL is more robust to label noises. Note that we do not use pretrained model when training in FER2013 to find how the pretrained model affects the experiment results. From the results we can draw the conclusion that using pretrained model can better deal with label noise as RUL improves the accuracy most when using pretrained models which aligns the finding in [19. The results also demonstrate that RUL can learn meaningful uncertainty values as we use uncertainty values to guide model to find the most uncertain training samples and suppress label noises.
Table 1: Test accuracy (%) on RAF-DB, FER2013 and AffectNet with synthetic noisy labels.
Method
Noisy (%)
RAF-DB
FER2013
AffectNet
Baseline
10
80.43+0.72
69.25+0.18
56.85+0.14
SCN 47
10
81.9240.69
69.28+0.05
58.7240.20
DUL 4
10
85.08+0.21
69.43+0.27
58.2640.10
RUL
10
86.22+0.29
70.29+0.28
60.66+0.13
Baseline
20
78.01
64.8740.32
54.7410.62
SCN
20
80.02+0.32
66.3010.49
56.35+0.61
DUL 4
20
81.95+0.32
65.5540.31
56.25+0.09
RUL
20
84.34+0.29
67.86+0.35
58.3640.28
Baseline
30
75. 124078
62.52+0.56
51.46+0.52
SCN 47
30
77.4610.64
62.6140.82
52.60+0.86
DUL
4
30
78.9040.80
60.98+0.34
55.09
RUL
30
82.06+0.44
64.62+0.39
56.65+0.13

---

Table 2: Test accuracy (%) versus rejection rate results of different uncertainty learning methods on RAF-DB and FER2013. The best performance among each column is shown in bold form.
RAF-DB
FER2013 Private Tossset
Method
rejection
10% rejection
20% rejection
30% rejection
D rejection
10% rejection
20% rejection
30% rejection
SCN 47
87.35
$6.85
86.63
87.28
72.67
72.82
74.29
75.80
CONF 6
88.30
91.78
93.97
95.90
72.69
73.00
73.11
72.97
DUL
88.04
90.
92.58
94.50
73.67
76.01
79.10
82.25
RUL
88.98
92.72
95.40
97.35
73.75
77.24
80,98
84.75
4.4 Test with Rejection.
In order to illustrate the quality of our learned uncertainty values more intuitively, we utilize an evaluation metric similar to [4] 2 15 40) named accuracy versus rejection rate. The metric shows a test accuracy over the fraction of unconsidered facial expression images. Specifically, based on the predicted uncertainty values, these unconsidered images are the most uncertain images recognize, and the test accuracy is calculated on the remaining images. The metric indicates good uncertainty estimation when the test accuracy increases consistently when increasing the ratio of unconsidered images. We also compare with the out-of-distribun uses labels as hints to help indicate when the network tends to give wrong predictions. We show the test accuracy in Table 2 We also plot the accuracy versus rejection rate curve in the supplementary material. RUL gets the best performance in all cases, which illustrates that the uncertainty values learned by RUL are more related to the recognition confidence than other methods. Another interesting finding is that CONF performs pretty well on RAF-DB while degrades on FER2013. It may be explained by the following reason that we use pretrained model on RAF-DB while training from scratch on FER2013, CONF needs strong backbone to make good use of the information from labels while our method can perform well in both ways.
detection method [6] noted as CONF. CONF
4.5 Understand learned uncertainty values.
In this section, we visualize the learned features of different uncertainty learning methods and provide qualitative and quantitative analysis to show the meaning of our learned uncertainty values.
to visualize the learned feature distributions of different uncertainty learning methods to show the effectiveness of RUL. The results are shown in Figure 3 It is shown that the comparison of different facial expressions encourages intra-class compactness and inter-class seperability of the learned features. We believe that this is because RUL needs to recognize both expressions from the mixed feature, and it will be forced to learn the most discriminative feature that can tell an expression image apart from all the other expression images compared with it. We plot the uncertainty distribution of images with high uncertainty and images with low uncertainty seperately in the supplementary material. The results show that the samples with high uncertainty learned by RUL congregate the center of the figure. They tend to contain ambiguous expressions like neutral (class 6), which can be easily confused with other expressions. However, the samples with small uncertainty are far from each other which are very easy to be recognized. The feature visualization demonstrates that RUL can learn feature distribution which reflects the uncertainty distribution of the dataset.
Visualization of learned features. We use t-SNE
44
12-
-22
Figure 3: The learned feature distribution of SCN, DUL and RUL on RAF-DB dataset. (O:nsetr 1:Fear, 2:Disuust, 3:Happy, 4:Sad, 5:Angry, 6:Neutrll)
Visualization of uncertainty values. We display some expression images in testset with their
estimated uncertainty values in Figure A The first row shows that images with large uncertainty
7

---

values are more likely to contain ambiguous expressions. They are easy to be misclassified (The wrong predicted labels are marked at the bottom of the first row images in red color). In contrast, the second row shows that images with small uncertainty values tend to contain obvious facial features, and they are easy to rightly classified. This illustrates that our learned uncertainty values can represent the difficulty of expression images as well as the model's confidence of its predictions. That property brings us benefits that are important for the development of safe Al systems. the model knows when it is prone to be wrong then it can turn to human for help, which might avoid disasters in fields with high accuracy requirements.
3,28
3.15
2.77
2.74
2.64
2:61
2.53
disgust Label happy
disgust Label : sad
happy
Label disgust
Label foar
Label happy
Label disgust
Label happy
Label neutral
0.62
0.66
0.68
0.72
0.76
0.76
0.83
0.86
anpty
anery
disgust
BITRTT
Tear
Sad
neutral
Label angry
Label angry
Label: disgust
Label angry
Label sad
Labell: fear
Label sad
Labell: neutral
Figure 4: The learned uncertainty values for different images. Images with large and small uncertainty values shown in the first and second rows respectively. Uncertainty values marked in the upper left corner, and predicted labels are marked at the bottom. Note that RUL learns large uncertainty values for images with ambiguous expressions while small uncertainty values for images with obvious expressions.
Distribution of uncertainty values. RUL learn different uncertainty distributions based different datasets instead of constraining uncertainty distributions of all datasets to the diagonal multivariate normal Gaussian distribution like DUL. We show the learned uncertainty distributions of different datasets Figure and Figure In order to make a comparison, we normalize RUL learned uncertainty values to [0,1]. Note that larger uncertainty values uncertain. We cut off at 400 on the y-axis in Figure SCN learns too many uncertainty values close 0, which indicates bad performance. The results demonstrate that RUL learns large uncertainty values for more samples while the other two methods learn relative small uncertainty values (smaller than 0.4 RAF-DB and smaller than 0.6 on FER2013) for all samples, which means RUL can reduce the negative influence from the strong learning ability of networks mentioned in Section 3.1 Furthermore, we also randomly sample several images which are hard for human to recognize and find that only RUL learns large uncertainty values for them.
the images more
400
Number of samples
300
250
8.1251
ILESSS
EL5545
100
80
Number
150
186I
(L9933
60
100
08337
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Uncertainty value
Uncertainty value
Figure 5: The learned uncertainty distribution of RAF-DB. Uncertainty values marked at the top of the image, different colors represent different methods.
Figure 6: The learned uncertainty distribution of FER2013. Uncertainty values marked at the top of the image, different colors represent different methods.

---

Table 3: Comparison on RAF-DB
Table 4: Comparison on FER2013
Table 5: Comparison of different structures.
Method
Acc.(%)
Method
Acc.(%)
Method
Acc.(%)
DLP-CNN 27
84.22
Conv-Inccetion
66.40
mix output 88.36
loss attention
88.75
gaCNN 29
85.07
Bag of Words 21
67.40
RUL
88.98
IPA2LT
52
86.77
31
70.02
RAN 48
86.90
VGG 35
72.70
Method
10% rejection
SCN 47
87.03
SCN 47
72.67
mix output
17.17
loss attention
89.32 92.72
RUL
88.98
RUL
73.75
RUL
Table 6: The test accuracy versus rejection rate of different uncertainty learning methods on CIFAR-10 and ImageNet-100.
CIFAR-II -10
ImtgeNet- 100
Method
rejection
10% rejection
20% rejoction
30% rejection
rejeccton
10%
rejection
20% rejection
30% rejection
6
96.84
76.36
76.21
74.61
74.49
DUL
95.24
99.22
76.03
75.80
79.64
82.86
RUL
95.58
98.60
99.34
99.44
77.74
77.65
82.48
86.74
4.6 Comparison with state-of-the-art FER methods.
Although we aim to deal with uncertainty learning problems in FER, our RUL method still gets better test accuracy comparing to series of state-of-the-art FER methods according to the results in Table 3
and T
Note that though RUL uses no extra data for co-training, it still achieves better performance. We speculate that this because the comparison of different FER images in the training process implicitly introduces the data augmentation effect. Meanwhile, as uncertain images will get larger weights to mix with certain images, the FER model will pay more attention to learning the uncertain images after it fits the certain images, which improves the model's 's ability to recogmize difficult facial expressions. We provide the confusion matrix of different methods on RAF-DB in the supplementary material. As mentioned by [23] the FER model tends to make more mistakes in classes with fewer samples, such as disgust and fear. However, the confusion matrices show that RUL increases test accuracy mainly in these two classes, which means RUL. can better recognize difficult facial expressions.
We also replace the Gaussian variances which are the learned uncertainty values in DUL with RUL learned uncertainty values to guide the DUL feature learning branch. Though RUL does not constrain the uncertainty distribution Gaussian distribution, still improves the performance of DUL from 88.17% to 88. 78%, which means the uncertainty learned by RUL can better reflect the uncertainty distribution the dataset.
4.7 Other Experiments.
Experiments CIFAR-10 and ImageNet-10M 100. To show RUL is effective not only on FER tasks, we also carry out experiments on CIAFR-10 and ImageNet- 100. Table that RUL could still improve the performance of the deep learning model on other classification tasks. The test accuracy increases along with the rejection the most uncertain test samples, which illustrates that RUL can still learn meaningful uncertainty values datasets besides FER tasks.
Ablation study. In order to show that the mixed feature part is useful for uncertainty learning, implement mix output method and loss attention method to make a comparison. Specifically, we do mix facial features in the mix output method. Instead, we classify two images separately and mix the logits according to uncertainty values. Results are shown at Table RUL performs better than mix output both the test accuracy and accuracy with 10% rejection, reflecting that mixed features may contain more relativity information than mixed logits. Loss attention method means we skip the mix part and use the uncertainty values to directly weight the loss values. T shows that RUL outperforms the loss attention method, which illustrates that RUL is effective not only because

---

uncertainty values affect loss values but also due to the interaction of different facial features during the training process.
Conclusion
In this paper, we propose novel and effective uncertainty learning method called Relative Uncertainty Learning for uncertainty quantification and noisy label training. Unlike traditional uncertainty learning methods, which uncertainty values to weigh the loss and then add regularization to force the network to learn uncertainty, we view uncertainty as relative concept and learn uncertainty through the relativity of different samples. Extensive experiments have shown that RUL can learn meaningful uncertainty values, which reflect the difficulty of samples to be rightly classified. RUL can also help the FER model achieve state-of of-the-art performance in both real- world and synthetic noisy FER datasets. We further demonstrate that RUL is also useful in other classification tasks.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grants No. 61871052
References
[1] F. Barsoum, C. Zhang, C. C. Ferrer, and 7. Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 279-283, 2016.
[2] L. Best- Rewsen and Jain. Learning face image quality from human assessments, JEEE Transactions on Information Forensics and Security, 13(12)3304-3307, 2018
C. Blundell, J. Cornebise, Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622, PMLR, 2015.
[4] Chang, Lan, C. Cheng. and Y. Wei. Data uncertainty learning face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5710-5711 2020.
[5] Dehghami, Severyn, S. Rothe, and Kamps. Avoiding your teacher's mistakes: Training neural networks with controlled weak supervision. arXiv preprint arXiv:1711.0031/ 2017.
[6] T. DeVeies and G. W. Taylor. Learning confidence -idt---distiibtiinn detection neural networks. arXiv preprint 1rXiv:1802.04488 2018.
[7] A. Dhall, R. Goecke, Lucey, and T. Gedeon. Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 2106 2112. IEEE, 2011.
[8] P. Ekman. An argument for basic emotions. Cognition & emotion, 6(3-4):169-200, 1992.
[9] C. Fabian Benitez-Quiroz, R. Srinivasan, and A. Martinez. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions the wild, In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5562-5570, 2016.
[10] Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
[11] Y. Gal and 7. Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, PMIR, 2016.
[12] Goldberger and E. Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
[13] S. Gong, V. N. Boddeti, and A. K. Jain. On the capacity of face representation. arXiv preprint arXiv: 10433, 2017.
[14] L. J. Goodfellow, D. Erhan, L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang. D. Thaler, H. Lee, et al. Challenges in representation learning: A report on three machine learning contests. In International conference neural information processing, pages 117-124, Springer, 2013.
10

---

[15] P. Grother, M. Ngan. and K. Hanaoka. Face recognition vendor test-face recognition quality assessment concept and goals. In NIST 2019.
[16] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMIR, 2017.
[17] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-Im: A dataset and benchmark for large-scale face recognition. In European conference on computer vision, pages 87-102. Springer, 2016.
[18] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] D. Hendrycks, Lee, and M. Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pages 2712-2721. PMLR, 2019.
[20] S. loffe C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 45684.66..0 PMLR, 2015.
[21] R. T. Jonescu, M. Popescu, and Grozea. Local learning to improve bag of visual words model for facial expression recognition. In Workshop on challenges in representation learning. ICML, 2013.
[22] A. Kendall and Y. Gal. What uncertainties do we need in hayesian deep learning for computer vision? In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17. page 5580-5590, Red Hook, NY, USA, 2017. Curran Associates Inc.
[23] S. Khan, M. Hayat, W. Zamir, Shen, and Shao. Striking right balance with uncertainty. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 103-112, 2019.
[24] D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
[25] A. D. Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Structural Safety, 1012))005-111 2009. Risk Acceptance and Risk Communication.
[26] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, page 6405-6416, 2017.
[27] S. Li. W. Deng, and J. Du. Reliable crowdsoureing and deep locality preserving learning for expression recognition the wild. Proceedings of IEEE conference on computer vision and pattern recognition, pages 2852 2861, 2017.
[28] Y. Li, Yang, Y. Song, Cao, J. Luo, and L.-J. Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1910-1918, 2017.
[29] Y. Li, J. Zeng, S. Shan, and X. Chen. Occlusion aware facial expression recognition using enn with attention mechanism. IEEE Transactions on Image Processing, 28(5):2439-22 24500 2018
[30] Z. Li and S. Arora. An exponential learning rate schedule for deep learning. arXiv preprint arXiv: 1OXiv:9100004544 2019.
[31] S. Minaee, M. Minaei, and A. Abdolrashidi. Deep-emotion: Facial expression recognition using attentional convolutional network. Semsors, 21(9):3046, 2021.
[32] A. Mollahosseini, D. Chan, and M. H. Mahoor. Going deeper in facial expression recognition using deep neural networks. In 2016 IEEE Winter conference on applications of computer vision (WACV), pages 1-10. IEEE 2016.
[33] A. Mollahosseini, B. Hasani, and M. H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions Affective Computing, 10(1):18-33 31, 2017.
[34] N. Natarajan, I. S. Dhillon, P. Ravikumar, and A. Tewari. Learning with noisy labels. In NIPS, volume 26, pages 1196 1204, 2013.
[35] C. Pramerdorfer and M. Kampel. Facial expression recognition using convolutional neural networks: State of the art. CoRR, abs/ Abs/1612,.0293, 2016.
[36] J. She, Y. Hu, H. Shi, J. Wang. Q. Shen, and T. Mei. Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
11

---

[37] Y. Shi and A. K. Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6902-6911, 2019.
[38] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
[39] S. Sukhbaatar and R. Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv: 2080. 2(3) 2014.
[40] E. Tabassi and P. Grother. Biometric sample quality.. 2015.
[41] P. Terhorst, J. N. Kolf, N. Damer, F. Kirchbuchner, and A. Kuijper. Ser-fiq: Unsupervised estimation of face image quality based on stochastic embedding robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5651-5660, 2020.
[42] S. Thulasidasan, G. Chennupati, A. Bilmes, T. Bhattacharya, and S. Michalak. On mixup training: Improved calibration and predictive uncertainly for deep neural networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnelt, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[43] M. Valstar and M. Pantic. Induced disgust, happiness and surprise: An addition to the mmi facial expression database. Proc. Int'l Conf. Language Resources and Evaluation, Workshop EMOTION, pages 65-70, 01 2010.
[44] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research. 9(11), 2008.
[45] A. Veit, N. Alldrin, G. Chechik, I. Krasin, A. Gupta, and S. Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 839 847, 2017.
[46] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio. Manifold mixup: Better representations by interpolating hidden states. International Conference on Machine Learning. pages 6438-6447 PMLR, 2019.
[47] K. Wang, X. Peng. J. Yang. S. Lu, and Y. Qiao. Suppressing uncertainties for large- scale facial expression recognition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6897-6906, 2020.
[48] K. Wang, X. Peng. J. Yang, D. Meng, and Y. Qiao. Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing, 29:4057-4000 4069, 2020.
[49] J. Wu. Facial
Facial- Faiill-rppessionncccoatttooonpp pytpro/eereccoiiiiooooytoochchch github. lattps://github com com/NuJie1000000
Accessed: 2020-08-07.
[50] T. Yu, D. Li, Y. Yang. T. M. Hospedales, and T. Xiang. Robust person re-identification by modelling feature uncertainty. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2019.
[51] U. Zafar, M. Ghafoor, T. Zia, G. Ahmed, A. Latif, K. R. Malik, and A. M. Sharif. Face recognition with bayesian convolutional networks for robust surveillance systems. EURASIP Journal Image and Video Processing, 2019(1):1 10, 2019.
[52] J. Zeng. S. Shan, and X. Chen. Facial expression recognition with inconsistently annotated datasets. In Proceedings of the European conference on computer vision (ECCV), pages 222-237, 2018.
[53] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[54] G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. Pietikälnen. Facial expression recognition from near-infrared vidcos. Image and Vision Computing, 29(9):607-61 6619, 2011.
12


Reviews:
[{"review_id": "dxMeRWVPAIE", "reviewer": "Reviewer_3dyN", "summary": "This paper focuses on the uncertainty problem in the facial expression recognition. To quantify the uncertainties and improve the FER performance, this paper proposes the relative uncertainty learning (RUL), which could learn the uncertainty from the relative difficulty. Experiments on two public FER datasets show the effectiveness of the proposed method.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n\n+Considering the relative idea and human cognition, the RUL learns the uncertainty though comparisons. \n\n+Experiments on the basic RAF-DB are encouraging.\n\nCons:\n\n-The novelty of this work is not enough for NeurIPS acceptance standards. And the add-up loss could be regarded as the cross-entropy loss, which needs to be calculated twice.\n\n-The experiments of this paper are insufficient. Especially, the RUL is just evaluated on RAF-DB and FER-2013. The latter is a disputed FER dataset, and is not widely used at present.\n\n-English needs a good revision in some parts of the text, which also makes it difficult to grasp some details.\n\n\n### Comment\n\n1) The defination of the facial expression pairs is not clear. How to ensure that the image i and its counterpart image j belong to different expression categories?\n\n2) Are the weights of branches model for image i and j sharing? Besides, the output demension D is not given, which is an important parameter in this work. \n\n3) The designed normalized uncertainty vectors are averaged following DUL [1]. What happens if this branch maps faical features to a uncertainty probalibility via the softmax function? It would be better to further discuss more ways. \n\n4) The add-up loss can be viewed as the two-time cross-entropy loss. Why the sum is divided by N, not 2N? In addition, the explanation below the Equ.4 is wrong. It should be 'y_{i}, y_{j} means label_{i}, label_{j}'.\n\n5) The generalization performance of RUL is not convincing. It would be better to evaluate the proposed RUL on more FER datasets, especially AffectNet [2] and FERPlus [3]. The authors only compare with some common methods on FER-2013, not with uncertainty learning methods, for example the SCN [4].\n\n\n[1] Data uncertainty learning in face recognition. CVPR2020.\n[2] Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. on Affective Computing 2017.\n[3] Training deep networks for facial expression recognition with crowd-sourced label distribution. ACM ICMI2016.\n[4] Suppressing Uncertainties for Large-Scale Facial Expression Recognition. CVPR2020.\n\n\nAfter the rebuttal, I decided to raise my score to weakly accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Ed0x_k4dwNk", "reviewer": "Reviewer_RFnf", "summary": "The paper presents a novel method for uncertainty learning in the context of facial expression recognition. As in many existing methods for uncertainty quantification, the method comprises adding an extra branch in the output layer of a network solely used for that purpose. Contrary to other methods in uncertainty estimation, the learning of the network is done in a pairwise manner, where the uncertainty for a given image is estimated relative to the peer image. The output features before the emotion classification layer are combined using the estimated uncertainty and are sent to the classifier to be simultaneously classified as both the two emotion categories of each image. This way, for the classifier to “pay more attention” to the image that is harder to classify, the relative uncertainty must be larger w.r.t. that of the easy image. Under this motivation, the weights assigned to the feature fusion during the learning stage are representative of the uncertainty the network assigns to a particular image. The method can therefore be directly used in a single-image way at test time, adding no complexity at inference. The method is validated in several datasets for facial expression recognition, showing superiority in the process of uncertainty estimation w.r.t. other methods based on that extra branch being used to parameterize the standard deviation of a Gaussian distribution, which directly relates to the uncertainty estimation. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is simple yet interesting and elegant, and it appears to work well in practice. It is also a well-grounded and motivated idea. The uncertainty-driven mixup approach for uncertainty estimation is novel and might generate impact in the domain of representation learning as well. \n\n\nWhile the idea seems to work, the procedure sounds a bit counterintuitive to me. I understand that the pipeline is inspired from the well-established mix-up framework for visual representation learning. However, from my understanding, the feature fusion described in Section 3 is indeed bringing the features closer (in terms of distance), making them hard to be distinguished. Indeed, I would like to ask the authors to include in the rebuttal a t-SNE visualisation of the “mu” features from both their method and that of DUL. If the method is forced to classify both emotions correctly from the same feature combination, then the features might tend to become closer. This goes in the opposite direction of making the features more distinctive of the underlying emotion, assuming these to be mutually exclusive, or even from the research direction that advocates for label distribution learning. If this is the case, it looks to me that the method is expected to put the learning emphasis on the uncertainty value sigma, totally dismissing the capacity of learning meaningful features. Also, in the mixup framework, the target label (one-hot encodings) are also weighted by the same factor that mixes the feature representation. However, in this paper the classifier is expected to classify both emotions equally, which adds more confusion to how this works. I would like to ask authors to discuss about that in the rebuttal. \n\n\nWhy is sigma set to a scalar in Equation 3? I understand that this value (the mean value of sigma) will represent the total uncertainty, but why the weighting of the features is the same for all the dimensions of the feature space? If sigma is of the same dimension as mu, then why not trying a weighting scheme for each dimension? Such an experiment would also help identify if the features are indeed becoming closer or not as I mentioned above: if using a per-dimension weight makes the method work worse then this could mean that the use of a scalar is limiting to which extend the features can become closer (i.e. their relative angle cannot be changed with just a scalar). \n\n\nI believe that the paper should also include an out-of-distribution experiment, akin to those of mixup papers. In particular, a cross-dataset experiment should be included, in order to measure how well the uncertainty estimation generalizes to other data domains. I also believe that authors should consider newer databases such as AffectNet. Finally, and as a mere suggestion, I believe the paper would greatly benefit from experimenting in more difficult tasks such as valence and arousal, where the uncertainty is more obvious. \n\n\nThe method seems quite generic and could have impact in other domains. While there are some results in MNIST and CIFAR, these datasets are quite outdated and saturated (as the results show). I believe that an experiment in ImageNet (or ImageNet100 should the former demand too many resources). \n\nThe writing needs to be polished and proof-reading is necessary. For example:\n\nl. 41 “the difficulty of classification is a person’s subjective feeling (that??) comes from comparison”\nl. 61 “In (the) face recognition field”\nl. 62 “Because noisy face images are usually out of the cluster and have larger variances in the latent embedding space which might cause wrong recognition [36].” Incomplete sentence?\nl. 77 “the datasets for facial expression recognition is (are?) getting larger and larger”.\nl. 99 “The uncertainty learning methods mentioned before suffer from the strong learning ability of deep neural networks more or less.” More or less??\nl. 135 “ned by DUL [4], (the??) mean of the”\nl. 151 “When mixing two facial features, there will be a relatively easy facial feature for expression recognition and the other is relatively hard.” Different use of tenses (one feature will be and the other already is?)\nl. 174 “We also carry out (an??) ablation study”\n\nI would recommend the authors to undergo a thorough writing review to bring the quality of the reading to what is expected for a NeurIPS paper. \n\n\n==== Score updated from 6 to 7 after rebuttal\n\n\nAfter balancing the pros and the cons, I am borderline, towards acceptance, with the paper. It contains some novelty and I believe that, if amended properly, the paper can have a larger impact beyond the field emotion recognition. I would like to encourage the authors to address the main concerns in the rebuttal, to open a fruitful discussion with the fellow peers and AC. In any case, it is an interesting paper, and should it be deemed enough to be accepted to the main venue, I would really encourage authors to improve presentation and readability through a thorough proof-reading, that also uses either British or American English, and not both. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not contain a limitations and societal impact section. I think authors could improve the paper in this regard. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "EAW9gyqyfB", "reviewer": "Reviewer_4ZJJ", "summary": "The authors incorporated a concept of relativity into data uncertainty learning. The main method was basically a feature mixup, but the authors specially designed the mixing weights using the uncertainty values of two images.\n\n- I feel leveraging a mixup framework for uncertainty learning is quite novel.\n- The proposed method is well-motivated and sufficiently simple for people to build on.\n- The proposed method works well particularly under a high level of noisy labels.\n- I appreciate many visual examples.\n- The paper is fairly clearly written and easy to follow.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Concerns & Comments]\nThough I think the main idea is quite good, the experimental evaluations concern me a lot.\n- Although the proposed method looks quite general and can be applied to broad types of data, the experiments focused merely on FER. I would recommend that the authors show the wide applicability of their work by conducting experiments on face recognition datasets as in the DUL paper (Chang, CVPR'20), challenging object recognition datasets (MNIST and CIFAR10 are too easy for recent deep neural models), or others.\n- The number of images in RAF-DB and FER-2013 is not large. To verify the scalability of the proposed method, some experiments on AffectNet or large-scale datasets are highly recommended.\n- Relying on pretrained networks can limit the applicability of the proposed method. Could the authors train the networks from scratch and compare their results with those using pretrained networks? I think such a comparison as presented in Table 2 of the SCN paper (Wang, CVPR'20) is necessary.\n- I am not sure whether the comparison in Table 4 is fair. Were the baseline models pretrained on the Ms-Celeb-1M dataset?\n- It would be better if the references of baselines could be added in the cells in Tables 1, 2, and 6. \n\n====\nAfter the rebuttal, I raised my rating from 5 to 6 (please see the detailed comment below)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of their work were not described.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion, the reviewers are all for accepting the work. It's well-motivated, the idea is simple, and empirically it appears to work well in practice across a solid set of experiments. The specific novelty is fairly low, but I find this a highly overrated criteria for acceptance.  I highly recommend the authors address the rebuttal concerns as they further polish the paper.
Decision: Accept (Poster)