Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors incorporated a concept of relativity into data uncertainty learning. The main method was basically a feature mixup, but the authors specially designed the mixing weights using the uncertainty values of two images.

- I feel leveraging a mixup framework for uncertainty learning is quite novel.
- The proposed method is well-motivated and sufficiently simple for people to build on.
- The proposed method works well particularly under a high level of noisy labels.
- I appreciate many visual examples.
- The paper is fairly clearly written and easy to follow.
main_review: [Concerns & Comments]
Though I think the main idea is quite good, the experimental evaluations concern me a lot.
- Although the proposed method looks quite general and can be applied to broad types of data, the experiments focused merely on FER. I would recommend that the authors show the wide applicability of their work by conducting experiments on face recognition datasets as in the DUL paper (Chang, CVPR'20), challenging object recognition datasets (MNIST and CIFAR10 are too easy for recent deep neural models), or others.
- The number of images in RAF-DB and FER-2013 is not large. To verify the scalability of the proposed method, some experiments on AffectNet or large-scale datasets are highly recommended.
- Relying on pretrained networks can limit the applicability of the proposed method. Could the authors train the networks from scratch and compare their results with those using pretrained networks? I think such a comparison as presented in Table 2 of the SCN paper (Wang, CVPR'20) is necessary.
- I am not sure whether the comparison in Table 4 is fair. Were the baseline models pretrained on the Ms-Celeb-1M dataset?
- It would be better if the references of baselines could be added in the cells in Tables 1, 2, and 6. 

====
After the rebuttal, I raised my rating from 5 to 6 (please see the detailed comment below)
limitations_and_societal_impact: The limitations of their work were not described.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a novel method for uncertainty learning in the context of facial expression recognition. As in many existing methods for uncertainty quantification, the method comprises adding an extra branch in the output layer of a network solely used for that purpose. Contrary to other methods in uncertainty estimation, the learning of the network is done in a pairwise manner, where the uncertainty for a given image is estimated relative to the peer image. The output features before the emotion classification layer are combined using the estimated uncertainty and are sent to the classifier to be simultaneously classified as both the two emotion categories of each image. This way, for the classifier to “pay more attention” to the image that is harder to classify, the relative uncertainty must be larger w.r.t. that of the easy image. Under this motivation, the weights assigned to the feature fusion during the learning stage are representative of the uncertainty the network assigns to a particular image. The method can therefore be directly used in a single-image way at test time, adding no complexity at inference. The method is validated in several datasets for facial expression recognition, showing superiority in the process of uncertainty estimation w.r.t. other methods based on that extra branch being used to parameterize the standard deviation of a Gaussian distribution, which directly relates to the uncertainty estimation. 
main_review: The idea is simple yet interesting and elegant, and it appears to work well in practice. It is also a well-grounded and motivated idea. The uncertainty-driven mixup approach for uncertainty estimation is novel and might generate impact in the domain of representation learning as well. 


While the idea seems to work, the procedure sounds a bit counterintuitive to me. I understand that the pipeline is inspired from the well-established mix-up framework for visual representation learning. However, from my understanding, the feature fusion described in Section 3 is indeed bringing the features closer (in terms of distance), making them hard to be distinguished. Indeed, I would like to ask the authors to include in the rebuttal a t-SNE visualisation of the “mu” features from both their method and that of DUL. If the method is forced to classify both emotions correctly from the same feature combination, then the features might tend to become closer. This goes in the opposite direction of making the features more distinctive of the underlying emotion, assuming these to be mutually exclusive, or even from the research direction that advocates for label distribution learning. If this is the case, it looks to me that the method is expected to put the learning emphasis on the uncertainty value sigma, totally dismissing the capacity of learning meaningful features. Also, in the mixup framework, the target label (one-hot encodings) are also weighted by the same factor that mixes the feature representation. However, in this paper the classifier is expected to classify both emotions equally, which adds more confusion to how this works. I would like to ask authors to discuss about that in the rebuttal. 


Why is sigma set to a scalar in Equation 3? I understand that this value (the mean value of sigma) will represent the total uncertainty, but why the weighting of the features is the same for all the dimensions of the feature space? If sigma is of the same dimension as mu, then why not trying a weighting scheme for each dimension? Such an experiment would also help identify if the features are indeed becoming closer or not as I mentioned above: if using a per-dimension weight makes the method work worse then this could mean that the use of a scalar is limiting to which extend the features can become closer (i.e. their relative angle cannot be changed with just a scalar). 


I believe that the paper should also include an out-of-distribution experiment, akin to those of mixup papers. In particular, a cross-dataset experiment should be included, in order to measure how well the uncertainty estimation generalizes to other data domains. I also believe that authors should consider newer databases such as AffectNet. Finally, and as a mere suggestion, I believe the paper would greatly benefit from experimenting in more difficult tasks such as valence and arousal, where the uncertainty is more obvious. 


The method seems quite generic and could have impact in other domains. While there are some results in MNIST and CIFAR, these datasets are quite outdated and saturated (as the results show). I believe that an experiment in ImageNet (or ImageNet100 should the former demand too many resources). 

The writing needs to be polished and proof-reading is necessary. For example:

l. 41 “the difficulty of classification is a person’s subjective feeling (that??) comes from comparison”
l. 61 “In (the) face recognition field”
l. 62 “Because noisy face images are usually out of the cluster and have larger variances in the latent embedding space which might cause wrong recognition [36].” Incomplete sentence?
l. 77 “the datasets for facial expression recognition is (are?) getting larger and larger”.
l. 99 “The uncertainty learning methods mentioned before suffer from the strong learning ability of deep neural networks more or less.” More or less??
l. 135 “ned by DUL [4], (the??) mean of the”
l. 151 “When mixing two facial features, there will be a relatively easy facial feature for expression recognition and the other is relatively hard.” Different use of tenses (one feature will be and the other already is?)
l. 174 “We also carry out (an??) ablation study”

I would recommend the authors to undergo a thorough writing review to bring the quality of the reading to what is expected for a NeurIPS paper. 


==== Score updated from 6 to 7 after rebuttal


After balancing the pros and the cons, I am borderline, towards acceptance, with the paper. It contains some novelty and I believe that, if amended properly, the paper can have a larger impact beyond the field emotion recognition. I would like to encourage the authors to address the main concerns in the rebuttal, to open a fruitful discussion with the fellow peers and AC. In any case, it is an interesting paper, and should it be deemed enough to be accepted to the main venue, I would really encourage authors to improve presentation and readability through a thorough proof-reading, that also uses either British or American English, and not both. 

limitations_and_societal_impact: The paper does not contain a limitations and societal impact section. I think authors could improve the paper in this regard.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the uncertainty problem in the facial expression recognition. To quantify the uncertainties and improve the FER performance, this paper proposes the relative uncertainty learning (RUL), which could learn the uncertainty from the relative difficulty. Experiments on two public FER datasets show the effectiveness of the proposed method.

main_review: Pro:

+Considering the relative idea and human cognition, the RUL learns the uncertainty though comparisons. 

+Experiments on the basic RAF-DB are encouraging.

Cons:

-The novelty of this work is not enough for NeurIPS acceptance standards. And the add-up loss could be regarded as the cross-entropy loss, which needs to be calculated twice.

-The experiments of this paper are insufficient. Especially, the RUL is just evaluated on RAF-DB and FER-2013. The latter is a disputed FER dataset, and is not widely used at present.

-English needs a good revision in some parts of the text, which also makes it difficult to grasp some details.


### Comment

1) The defination of the facial expression pairs is not clear. How to ensure that the image i and its counterpart image j belong to different expression categories?

2) Are the weights of branches model for image i and j sharing? Besides, the output demension D is not given, which is an important parameter in this work. 

3) The designed normalized uncertainty vectors are averaged following DUL [1]. What happens if this branch maps faical features to a uncertainty probalibility via the softmax function? It would be better to further discuss more ways. 

4) The add-up loss can be viewed as the two-time cross-entropy loss. Why the sum is divided by N, not 2N? In addition, the explanation below the Equ.4 is wrong. It should be 'y_{i}, y_{j} means label_{i}, label_{j}'.

5) The generalization performance of RUL is not convincing. It would be better to evaluate the proposed RUL on more FER datasets, especially AffectNet [2] and FERPlus [3]. The authors only compare with some common methods on FER-2013, not with uncertainty learning methods, for example the SCN [4].


[1] Data uncertainty learning in face recognition. CVPR2020.
[2] Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. on Affective Computing 2017.
[3] Training deep networks for facial expression recognition with crowd-sourced label distribution. ACM ICMI2016.
[4] Suppressing Uncertainties for Large-Scale Facial Expression Recognition. CVPR2020.


After the rebuttal, I decided to raise my score to weakly accept.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: After discussion, the reviewers are all for accepting the work. It's well-motivated, the idea is simple, and empirically it appears to work well in practice across a solid set of experiments. The specific novelty is fairly low, but I find this a highly overrated criteria for acceptance.  I highly recommend the authors address the rebuttal concerns as they further polish the paper.
Decision: Accept (Poster)