Relative Uncertainty Learning for Facial Expression Recognition

1 Introduction . Although well-trained models can give high-confidence inferences directly , the inference results may actually be wrong and cause serious consequences [ 22 ] . Therefore , the concept of uncertainty is widely used in measuring how well the model trusts its inference results , which has become a research hotspot in trustworthy machine learning [ 3 , 11 , 10 , 22 , 26 , 16 ] . It is expected that a reliable model assigns high level uncertainty to its erroneous predictions so that humans can intervene to avoid lots of disasters . As described by Kiureghian et al . [ 25 ] , the uncertainties in machine learning can be split into aleatoric uncertainty and epistemic uncertainty , which are also called data uncertainty and model uncertainty . In this paper , we focus on quantifying data uncertainty as model uncertainty can be eliminated by introducing more training data . Concretely , data uncertainty in facial expression recognition ( FER ) is mainly caused by ambiguous facial expressions and the subjectiveness of annotators , which blocks the improvement of recognition performance [ 47 ] . Recent studies propose a variety of methods to solve the uncertainty learning problem in the face recognition field . PFE [ 37 ] introduces a new branch to learn the uncertainty of face recognition and recognizes faces by measuring the similarity between two Gaussian distributions . Chang et al . [ 4 ] further propose DUL to learn feature and uncertainty simultaneously in order to improve facial feature learning . When it comes to FER , SCN [ 47 ] uses a fully-connected layer to learn an importance weight for each image and suppresses uncertainties according to the learned weights . However , the strong learning ability of neural networks will deteriorate the uncertainty learning branch . One solution is adding regularization to force the model to learn uncertainty , which needs to be tuned carefully , otherwise , it would lead to underfitting . For example , SCN uses margin loss to keep the gap between certain and uncertain images , but choosing a suitable gap value is non-trivial . DUL assumes features 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to follow a Gaussian distribution , but the real distribution of the dataset is not always following such an assumption . Therefore , it is still a challenge to learn uncertainty distributions of real datasets without deteriorating feature learning . To address the problem , we perceive data uncertainty from a different point of view . We realize that data uncertainty – the difficulty to classify a sample correctly is actually a relative concept . More specifically , the difficulty of classification is a person ’ s subjective feeling that comes from comparison . One can only know whether a sample is easy or not according to a reference . Inspired by this idea , we propose an innovative method called Relative Uncertainty Learning ( RUL ) to help deep learning models to learn uncertainty for each sample . Concretely , we build a new branch to model the uncertainties of input images and utilize the uncertainties as weights to mix two features of different labels . Through an add-up loss function , the model is encouraged to recognize two expressions simultaneously from the mixed features , which enables the model to learn uncertainty comparatively while minimizing the total loss . As discussed in Section 3.3 , the uncertainty learning branch will assign large uncertainty values to uncertain facial expression images while small uncertainty values to certain images . The main contributions of this work are as follows : • We perceive uncertainty from a new perspective and propose an innovative uncertainty learning method to learn uncertainty from the relative difficulty of two samples . • We get state-of-the-art performance in both real-world and synthetic noisy FER datasets . • RUL does not need prior knowledge of the dataset uncertainty distribution and can be easily applied to different classification tasks with low computation cost . 2 Related Work . Uncertainty learning in the face field Uncertainties can be mainly categorized into model uncertainty which accounts for uncertainty in the model parameters and data uncertainty which is the noise inherent in the dataset [ 22 ] . In the face recognition field , uncertainty learning is important for learning discriminative feature embeddings of noisy face images . Because noisy face images are usually out of the cluster and have large variances in the latent embedding space , which might cause wrong recognition [ 37 ] . There have been several proposals to solve the uncertainty learning problem in face recognition tasks [ 13 , 51 , 23 , 37 , 4 ] . Some of them [ 13 , 51 , 23 ] have tried to analyze and learn face representations using model uncertainty . Gong et al . [ 13 ] estimate the capacity of a given face representation by explicitly accounting for the manifold structure , model uncertainty and data variability . Khan et al . [ 23 ] present a framework for class imbalance learning based on model uncertainty . Zafar et al . [ 51 ] cope with false positives through employing model uncertainty to improve the efficacy of face recognition systems . Thereinto the methods considering data uncertainty learning . Shi et al . [ 37 ] use PFE to learn a variance for each fixed feature and then measure the likelihood of each positive face pair of ( xi , xj ) sharing the same distribution of latent embedding p ( zi , zj ) . Chang et al . [ 4 ] further propose to learn feature and uncertainty simultaneously by encoding the latent feature embedding as a Gaussian distribution . In the facial expression recognition task , Wang et al . [ 47 ] try to learn an attention weight for each image , then utilize it to weigh cross-entropy loss in order to suppress the influence of noisy samples . She et al . [ 36 ] propose to use several branches to model the latent label distribution of facial expression images and use cosine similarity to capture the uncertain images . Noisy dataset training With the enhancement of deep learning network capabilities , the datasets for facial expression recognition ( FER ) are getting larger and larger . For the large-scale FER datasets [ 43 , 54 , 7 , 1 , 33 , 9 , 27 ] , due to the ambiguity of expressions and the subjectiveness of human annotators , it is extremely hard to get high-quality labels . Training with label noise has been studied for a long time . The basic idea is to enable the model to access the quality of the labels or estimate the noise distribution using a small set of clean dataset [ 5 , 28 , 39 , 45 ] . Li et al . [ 28 ] design a distillation framework to use information from both a small clean dataset and label relations in knowledge graphs to prevent the model learning from noisy labels . Veit et al . [ 45 ] use a small amount of clean annotations to reduce the noisy samples before fine-tuning the network using both the clean set and the full set . Other methods do not use a small clean dataset , but usually assume extra constraints on the noisy samples . Natarajan et al . [ 34 ] design a specific loss function for randomly flipped labels . Goldberger et al . [ 12 ] explicitly model the noises in datasets by a softmax layer which connects the correct labels to the noisy ones . Without collecting a clean set of dataset or using prior knowledge of noise distribution , our method learns uncertainty through the relativity of the given dataset and suppresses the most uncertain samples to get a good performance under label noise . 3 Proposed Method . In order to reduce the negative impact of the strong learning ability of neural networks and learn uncertainty values closer to the real dataset uncertainty distribution , we propose a simple and effective method named Relative Uncertainty Learning ( RUL ) . In this section , we first provide the motivation behind our method . Then we demonstrate the details of RUL . Finally , we explain why RUL is effective in uncertainty learning . 3.1 Motivation . The uncertainty learning methods mentioned before suffer from the strong learning ability of deep neural networks more or less . For example , the introduced variances in DUL are equivalent to some noises being added to feature points , so the model will learn very small variances for all feature points making the estimation of uncertainty less reliable . SCN also learns large importance weights for noisy samples to obtain small cross-entropy loss values as these noisy images will be memorized by the networks after training for several epochs . In order to prevent the strong learning ability of neural networks from degrading the uncertainty learning branch , we perceive uncertainty from a relative point of view . We find that learning uncertainty through comparison forms a natural regularization because if the model assigns a very small uncertainty value for one image of the comparison pair , then it means the counterpart image gets a large uncertainty value which prevents the model learning small uncertainty values for all images . Learning uncertainty through relativity also conforms to human cognition – it is very difficult for a person to define which sample is certain and which sample is uncertain without using one image as a reference . In the following section , we thoroughly illustrate the details of our proposed RUL . 3.2 Overview of Relative Uncertainty Learning . In this section , we illustrate the structure of our proposed RUL . The pipeline of RUL is shown in Figure 1 . Two Branches Model Given a batch of face images X , assuming the conventional feature extraction module fθ ( X ) can be decomposed as fθ ( X ) = f ( l ) θ ( f ( l−1 ) θ ( X ) ) , ( 1 ) we first extract general facial features Fg from the penultimate feature extraction layer denoted as Fg = f ( l−1 ) θ ( X ) [ 50 ] . Then , in order to learn facial features and uncertainty values simultaneously , we drop the layer indexed by l , i.e. , f ( l ) θ and build two separate branches fθµ ( f ( l−1 ) θ ( X ) ) and fθσ ( f ( l−1 ) θ ( X ) ) to produce facial features F = [ µ1 , µ2 , ... , µN ] ∈ RD×N and uncertainty vectors U = [ σ1 , σ2 , ... , σN ] ∈ RD×N . D , N denote output dimension and batch size separately . Relative Uncertainty Learning Inspired by the relativity of the uncertainty concept and the mixup method [ 53 , 42 , 46 ] , we mix two different facial features according to their uncertainty values which enables the FER model to learn uncertainty through the relativity of different samples . First , we mix a mini-batch with their shuffled ones which results in the facial images of the whole dataset being compared with each other during the training process because of the randomness introduced by the dataloader and the shuffle of mini-batches . Before mixing features , we first normalize the uncertainty vectors element-wise to compare two features with each other following Equation ( 2 ) σ̂i , σ̂j = σi σi + σj , σj σi + σj , ( 2 ) σi and σj are the uncertainty vectors for image i and its counterpart image j sampled from the shuffled indexes , note that they have different labels . This operation not only forms a natural regularization as the model can not learn small uncertainty values for both features , but also benefits the training process as extreme uncertainty values might make the training process unstable or even the loss can not converge . As mentioned by DUL [ 4 ] , the mean of the predicted uncertainty vectors σ̂i ∈ RD can be viewed as an approximated measurement of the estimated uncertainty for image i , we also use the mean of σ̂i as learned uncertainty value for image i denoted as σ̂imean ∈ R. Then we get an uncertainty value for each input image denoted as U = [ σ̂1mean , σ̂2mean , ... , σ̂Nmean ] ∈ R1×N . We then mix features according to their uncertainty values following Equation ( 3 ) µ̃ = σ̂imeanµi + σ̂jmeanµj , ( 3 ) µi and µj represent embedding features of the two images , σ̂imean and σ̂jmean represent their corresponding uncertainty values . We introduce negligible computation overhead as we only mix facial features according to their learned uncertainty values in the training phase . Furthermore , the mix feature part will be removed for deployment . Therefore , our network is end-to-end and adds no extra cost on inference . Classification Loss We design an add-up loss to encourage the uncertainty learning branch to learn different uncertainty values for different facial images . Specifically , add-up loss requires the FER model to recognize two expressions of input images simultaneously from mixed features . Since µ̃ is the mixed feature , we feed µ̃ to the classifier : Ltotal = − 1 N N∑ i , j ( log eWyi µ̃∑C c e Wcµ̃ + log eWyj µ̃∑C c e Wcµ̃ ) . ( 4 ) The loss function adds up the loss of recognizing µ̃ as class i and as class j. yi , yi means labeli , labelj , Wc is the c-th classifier and C means the total number of expression classes . The loss function forces the model to recognize two expressions equally from the mixed features . When mixing two facial features , there will be a relatively easy facial feature for expression recognition , and the other will be relatively hard . For simplicity , we note µi as the easy facial feature . Our model will learn a small uncertainty value for µi to mix with µj as µi is easy to recognize , the model can already get a small classification loss with its label yi after training for several epochs , even µi only takes up a small amount of the mixed feature µ̃ . While for the hard image j , it needs to take up a large amount of the mixed feature µ̃ to enable the model to find useful feature to get a small classification loss with label yj .

Summary: The paper presents a novel method for uncertainty learning in the context of facial expression recognition. As in many existing methods for uncertainty quantification, the method comprises adding an extra branch in the output layer of a network solely used for that purpose. Contrary to other methods in uncertainty estimation, the learning of the network is done in a pairwise manner, where the uncertainty for a given image is estimated relative to the peer image. The output features before the emotion classification layer are combined using the estimated uncertainty and are sent to the classifier to be simultaneously classified as both the two emotion categories of each image. This way, for the classifier to “pay more attention” to the image that is harder to classify, the relative uncertainty must be larger w.r.t. that of the easy image. Under this motivation, the weights assigned to the feature fusion during the learning stage are representative of the uncertainty the network assigns to a particular image. The method can therefore be directly used in a single-image way at test time, adding no complexity at inference. The method is validated in several datasets for facial expression recognition, showing superiority in the process of uncertainty estimation w.r.t. other methods based on that extra branch being used to parameterize the standard deviation of a Gaussian distribution, which directly relates to the uncertainty estimation.