Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation (2024)

Abstract: The subjective perception of emotion leads to inconsistent labels from human annotators. Typically, utterances lacking majority-agreed labels are excluded when training an emotion classifier, which cause problems when encountering ambiguous emotional expressions during testing. This paper investigates three methods to handle ambiguous emotion. First, we show that incorporating utterances without majorityagreed labels as an additional class in the classifier reduces the classification performance of the other emotion classes. Then, we propose detecting utterances with ambiguous emotions as out-of-domain samples by quantifying the uncertainty in emotion classification using evidential deep learning. This approach retains the classification accuracy while effectively detects ambiguous emotion expressions. Furthermore, to obtain fine-grained distinctions among ambiguous emotions, we propose representing emotion as a distribution instead of a single class label. The task is thus re-framed from classification to distribution estimation where every individual annotation is taken into account, not just the majority opinion. The evidential uncertainty measure is extended to quantify the uncertainty in emotion distribution estimation. Experimental results on the IEMOCAP and CREMA-D datasets demonstrate the superior capability of the proposed method in terms of majority class prediction, emotion distribution estimation, and uncertainty estimation.

heading: 1 Introduction
text: The inherent subjectivity of human emotion perception introduces complexity in annotating emotion datasets. Multiple annotators are often involved in labelling each utterance and the majorityagreed (MA) class is usually used as the ground truth (Busso et al., 2008; Cao et al., 2014). Utterances that have no majority-agreed (NMA) labels (i.e., with tied votes) are typically excluded during
emotion classifier training (Kim et al., 2013; Poria et al., 2017; Zou et al., 2022; Wu et al., 2021), which may cause issues when the system encounters such utterances in practical applications.
This paper investigates three approaches to handling ambiguous emotion data. First, a naive method is tested which aggregates NMA utterances into an additional class when training an emotion classifier. This approach proves problematic as NMA utterances contain a blend of emotions, thereby confusing the classifier and undermining the classification performance.
Then we explore if an emotion classifier can appropriately respond with “I don’t know” for ambiguous emotion data that does not fit into any predefined emotion class. This is realised by quantifying the uncertainty in emotion classification using evidential deep learning (EDL) (Sensoy et al., 2018). When a classifier trained on MA data encounters an NMA utterance during the test, the model should identify it as an out-of-domain (OOD) sample by providing a high uncertainty score, indicating its uncertainty regarding the specific emotion class to which the NMA utterance belongs.
Moreover, to obtain fine-grained distinctions between ambiguous emotional data, we re-frame the task from classification to distribution estimation. Consider the example shown in Figure 1 with the annotations assigned to three utterances. Since the majority emotion classes are “angry” for both utterances (a) and (b), they will be assigned the same ground-truth label “angry” in the aforementioned classification system, which implies that they convey the same emotion content and is evidently unsuitable. On the contrary, utterance (c), though being an NMA utterance, is more likely to share similar emotional content with utterance (b). Therefore, in order to obtain more comprehensive representations of emotion content, we further propose representing emotion as a distribution rather
2078
than a single class label and re-framing emotion recognition as a distribution estimation problem rather than a classification problem. A novel algorithm is proposed which extends EDL to estimate the underlying emotion distribution given observed human annotations and quantify the uncertainty in emotion distribution estimation. The proposed approach considers all human annotations rather than relying solely on the majority vote class. Multiple evaluation metrics are adopted to evaluate the performance in terms of majority class prediction, uncertainty measure, and distribution estimation. Rather than simply saying “I don’t know”, the proposed system demonstrates the ability to estimate the emotion distributions of the NMA utterances and also offer a reliable uncertainty measure for the distribution estimation.
Our contributions are summarised as follows. (i) To the best of our knowledge, this paper is the first work that treats ambiguous emotion as OOD and detects it by uncertainty estimation; (ii) This is the first work that applies EDL to quantify uncertainty in emotion classification; (iii) Imposing a single ground truth through majority voting leads to under-representation of minority views. We instead estimate the distribution over emotion classes which provides a more comprehensive representation of emotion content as well as a more inclusive representation of human opinions; (iv) A novel algorithm is proposed that extends EDL to quantify uncertainty in emotion distribution estimation.

heading: 2 Related work
text: Human annotators often interpret the emotion of the same utterance differently due to their personal experiences and cultural backgrounds (Busso et al., 2008; Cowen and Keltner, 2017; Sethu et al., 2019). Instead of using the MA annotation as the ground truth label, some research suggests treating emo-
tion classification as a multi-label task (Mower et al., 2010; Zadeh et al., 2018; Chochlakis et al., 2023) where all emotion classes assigned by any annotator are considered as correct classes and the ground truth label is presented as a multi-hot vector. The model is trained to predict the presence of each emotion class for each utterance. An issue with this approach is that it ignores the differences in strengths of different emotion classes.
An alternative approach uses “soft labels” as the proxy of ground truth, which is defined as the relative frequency of occurrence of each emotion class (Fayek et al., 2016; Han et al., 2017; Kim and Kim, 2018). The Kullback–Leibler (KL) divergence or distance metrics between the soft labels and model predictions are used to train the model. However, soft labels, being maximum likelihood estimates (MLE) of the underlying distribution based on observed samples, might not provide an accurate approximation to the unknown distribution when the number of observations (annotations) is limited. Also, although adopting soft labels, those methods still focus on obtaining a “correct” label (i.e., pursuing improved classification accuracy).
So far, the calibration of emotion models has not been extensively studied. In this study, we introduce a novel approach which provides not only better emotion content estimation but also a reliable measure of the model’s prediction confidence.

heading: 3 Detecting NMA as OOD by quantifying emotion classification uncertainty
text: As explained in the introduction and confirmed experimentally in Section 7.1, training an emotion classifier with NMA utterances grouped into an additional class degrades the classification performance. This section studies an alternative method. The emotion classifier is trained on MA utterances and NMA utterances are treated as OOD samples. By quantifying uncertainty in emotion classification, the model is expected to output a high uncertainty score when encountering ambiguous emotions, indicating that the utterance doesn’t belong to any predefined MA class.

heading: 3.1 Limitation of modelling class probabilities with the softmax activation function
text: A neural network model classifier transforms the continuous logits at the output layer into class probabilities by a softmax function. The model prediction can thus be interpreted as a categorical distribu-
tion with the discrete class probabilities associated with the model outputs. The model is then optimised by maximising the categorical likelihood of the correct class, known as the cross-entropy loss.
However, the softmax activation function is known to have a tendency to inflate the probability of the predicted class due to the exponentiation applied to transform the logits, resulting in unreliable uncertainty estimations (Gal and Ghahramani, 2016; Guo et al., 2017). Furthermore, crossentropy is essentially MLE, a frequentist technique lacking the capability to infer the variance of the predictive distribution.
In the following section, we estimate the model uncertainty using evidential deep learning (EDL) (Sensoy et al., 2018) which places a secondorder probability over the categorical distribution.

heading: 3.2 Quantify uncertainty in emotion classification by evidential deep learning
text: Consider an emotion class label as a one-hot vector y where yk is one if the emotion belongs to class k else zero. y is sampled from a categorical distribution η where each component ηk corresponds to the probability of sampling a label from class k:
y ∼ P(y|η) = Cat(η) = ηykk . (1)
To model the probability of the predictive distribution, we assume the categorical distribution is sampled from a Dirichlet distribution:
η ∼ p(η|α) = Dir(η|α) = 1 B(α)
K∏
k=1
ηαk−1k
(2) where B(·) is the Beta function, αk is the hyperparameter of the Dirichlet distribution. α0 =∑K
k=1 αk is the Dirichlet strength. The output of a standard neural network classifier is a probability assignment over the possible classes and the Dirichlet distribution represents the probability of each such probability assignment, hence modelling second-order probabilities and uncertainty.
Subjective logic (Jsang, 2018) establishes a connection between the Dirichlet distribution and the belief representation in Dempster–Shafer belief theory (Dempster, 1968), also known as evidence theory. Consider K classes each associated with a belief mass bk and an overall uncertainty mass u, which satisfies u+ ∑K k=1 bk = 1. The belief mass assignment corresponds to the Dirichlet hyperparameter αk: bk = (αk − 1)/α0, where ek = αk−1
is usually termed evidence. The overall uncertainty can then be computed as:
u = K
α0 . (3)
A neural network fΛ is trained to predict Dir(η(i)|α(i)) for a given sample x(i) where Λ is the model parameters. The network is similar to standard neural networks for classification except that the softmax output layer is replaced with a ReLU activation layer to assure non-negative outputs, which is taken as the evidence vector for the predicted Dirichlet distribution: fΛ(x(i)) = e(i). The concentration parameter of the Dirichlet distribution can be calculated as α(i) = fΛ(x(i)) + 1. Given Dir(η(i)|α(i)), the estimated probability of class k can be calculated by:
E[η(i)k ] = α (i) k
α0(i) . (4)

heading: 3.2.1 Training
text: For brevity, superscript i is omitted in this section. Given one-hot label y and predicted Dirichlet Dir(η|α), the network can be trained by maximising the marginal likelihood of sampling y given the Dirichlet prior. Since the Dirichlet distribution is the conjugate prior of the categorical distribution, the marginal likelihood is tractable:
P(y|α) = ∫ P(y|η)p(η|α)dη
=
∫ ∏
k
ηykk 1
B(α)
∏
k
ηαk−1k
= B(α+ y)
B(α) =
∏K k=1 α yk k α0 ∑K k=1 yk .
(5)
It is equivalent to training the model by minimising the negative log marginal likelihood:
LNLL = K∑
k=1
yk(log(α0)− log(αk)). (6)
Following (Sensoy et al., 2018), a regularisation term is added to penalise the misleading evidence:
LR = KL(Dir(η|α̃)||Dir(η|1)), (7) where Dir(η|1) denotes a Dirichlet distribution with zero total evidence and α̃ = y+ (1− y)⊙α is the Dirichlet parameters after removal of the non-misleading evidence from predicted α. This penalty explicitly enforces the total evidence to shrink to zero for a sample if it cannot be correctly classified. The overall loss is L = LNLL + λLR where λ is the regularisation coefficient.

heading: 4 Emotion distribution estimation
text: As illustrated in Figure 1, the majority vote class is not sufficient for fine-grained emotion representations. In this section, we describe emotion by a distribution instead of a single class label.
Consider an input utterance x(i) associated with Mi labels from human annotators {y(i)m }Mim=1 where ym = [ym1, . . . , ymK ] is a one-hot vector. Instead of representing the emotion content by the majority vote class, we propose estimating the underlying emotion distribution η based on the observations {y(i)m }Mim=1. The emotion classification problem is thus re-framed as a distribution estimation problem. In contrast to the “soft label” method in Section 2 which approximates the emotion distribution of each x(i) solely based on D(i) = {y(i)m }Mim=1 by MLE and trains the model to learn this proxy in a supervised manner, the proposed approach meta-learns a distribution estimator fΛ across all data points Dmeta = {D(i)}Ni=1 where N is the number of utterances in training. This uses the knowledge about the emotion expression and annotation variability across different utterances.
For brevity, superscript i is omitted thereafter. Assume {ym}Mm=1 are samples drawn from a multinomial distribution. Let ŷ = ∑M m=1 ym represent the counts of each emotion class:
{ym}Mm=1 ∼ P(y|η) = Mult(η,M) (8)
Mult(η,M) = Γ(M + 1)
∏K k=1 Γ(ŷk + 1) ηŷkk . (9)
The categorical distribution in Eqn. (1) is the special case when M = 1.
The network is trained by maximising the marginal likelihood of sampling {ym}Mm=1 given the predicted Dirichlet prior Dir(η|α):
P({ym}Mm=1|α) = ∫ P({ym}Mm=1|η)p(η|α)dη
= Γ(M + 1)
∏K k=1 Γ(ŷk + 1)
∏K k=1 α ŷk k α0 ∑K k=1 ŷk . (10)
The multinomial coefficient is independent of α, we thus verify that LNLL in Eqn. (6) can be generalised to the distribution estimation framework by replacing one-hot majority label y with ŷ:
LNLL∗ = K∑
k=1
ŷk(log(α0)− log(αk)). (11)
The regulariser in Eqn. (7) is then replaced with:
LR1 = KL(Dir(η|α̂)||Dir(η|1)) (12) where α̂ = ȳ+(1−ȳ)⊙α and ȳ = 1M ∑M m=1 ym is the soft label. An alternative regulariser is proposed in order to explicitly regularise the predicted multinomial distribution:
LR2 = KL(ȳ||E[η]). (13) Hence, we have extend the EDL method described in Section 3.2 for classification to quantify the uncertainty in distribution estimation, with the original method (Sensoy et al., 2018) being a special case when M = 1 and ŷ becomes the one-hot majority label y. In addition, it’s worth noting that the proposed approach does not require a fixed number of annotators for every utterance and can easily generalise to a large number of annotators (i.e., for crowd-sourced datasets). There has been previous work on emotion distribution estimation using another variant of Dirichlet prior network (DPN) (Wu et al., 2022). Note that although both EDL* and DPN involve a Dirichlet distribution, EDL* has a different problem formulation to DPN. The key differences between EDL* and DPN are explained in detail in Appendix H.

heading: 5 Evaluation metrics
text: The proposed method is evaluated in terms of majority prediction, uncertainty estimation, OOD detection, and distribution estimation.
Majority prediction. Majority prediction for MA utterances is evaluated by classification accuracy (ACC) and unweighted average recall (UAR) which is the sum of class-wise accuracy divided by the number of classes.
Uncertainty estimation. Model calibration is evaluated by expected calibration error (ECE) (Naeini et al., 2015) and maximum calibration error (MCE) (Naeini et al., 2015). ECE measures model calibration by computing the difference in expectation between confidence and accuracy. Predictions are partitioned into Q bins equally spaced in the [0,1] range and ECE can be computed as follows:
ECE = Q∑
q=1
|Bq| n |Acc(Bq)− Conf(Bq)| . (14)
MCE is a variation of ECE which measures the largest calibration gap:
MCE = max q∈{1,...,Q} |Acc(Bq)− Conf(Bq)| . (15)
OOD detection. The area under the receiver operating characteristic (AUROC) and the area under the precision-recall curve (AUPRC) are used to evaluate the performance of OOD detection. The estimated uncertainty is used as a decision threshold for both AUROC and AUPRC. The baseline is 50% for AUROC and is the fraction of positives for AUPRC. NMA utterances are set as the positive class to detect.
Distribution estimation. Emotion distribution estimation performance is measured by the negative log-likelihood (NLL) of sampling human annotations from the predicted multinomial distribution.

heading: 6 Experimental setup

heading: 6.1 Baselines
text: The proposed methods were compared to the following baselines:
• MLE: a deterministic classification network with softmax activation trained by the crossentropy loss between the majority vote label and model predictions;
• MCDP: a Monte-Carlo dropout (Gal and Ghahramani, 2016) model with a dropout rate of 0.5 which is forwarded 100 times to obtain 100 samples during testing;
• Ensemble: an ensemble (Lakshminarayanan et al., 2017) of 10 MLE models with the same structure trained by bagging;
• MLE+: a MLE model with NMA as an extra class.
An additional baseline for distribution estimation:
• MLE*: the “soft label” approach mentioned in Section 2 which is trained by minimising KL divergence between the soft label ȳ and predictions. It is an extension of MLE from one-hot majority vote labels to soft labels.
The system described in Section 3.2 is denoted as “EDL”. “EDL*(R1)” and “EDL*(R2)” refers to the systems proposed in in Section 4 using regularisation terms defined in Eqn. (12) and Eqn. (13) respectively. Uncertainty estimation of EDL models are computed by Eqn. (3) while max probability is used as confidence measure for other methods.
Conformer 1
Conformer 2
Conformer 12
…
…
…
…𝑤𝑤1
𝑤𝑤2
𝑤𝑤12 …
Mean pooling
FC
ReLU
Dir(𝜼𝜼|𝜶𝜶)
𝜶𝜶𝜼𝜼𝒚𝒚 𝐾𝐾
α1 αK… 𝑒𝑒1 𝑒𝑒K…
p(𝒚𝒚|𝜼𝜼)𝒚𝒚
p 𝐲𝐲 𝜶𝜶 = ∫ p 𝐲𝐲 𝛈𝛈 Dir 𝛈𝛈 𝛂𝛂 d𝛈𝛈
U SM
-3 00
M
Figure 2: Illustration of the model structure.

heading: 6.2 Datasets
text: Two publicly available datasets were used in the experiments. The IEMOCAP corpus (Busso et al., 2008) is one of the most widely used emotion datasets. It consists of 10,039 English utterances from 5 dyadic conversational sessions. Each utterance was evaluated by at least three human annotators. Only 16.1% of utterances have an allannotators-agreed emotion label. The emotion distribution is represented using a five-dimensional categorical distribution, including happy (merged with excited), sad, neutral, angry, and others. The “others” category includes all emotions not covered in the previous four categories which is dominated by frustration (92%). 14.2% of the utterances don’t have a majority agreed emotion class label.
The CREMA-D corpus (Cao et al., 2014) contains 7,442 English utterances from 91 actors. Actors spoke from a selection of 12 sentences using one of six different emotions (anger, disgust, fear, happy, neutral and sad). The dataset was annotated by crowd-sourcing. Ratings based on audio alone were used in this work. Utterances have 9.21 ratings on average. 5.1% of utterances have an allannotators-agreed emotion label and 8.7% don’t have a majority agreed emotion class label.
Both datasets were divided into an MA subset and an NMA subset. All methods were trained only on MA data except for MLE+ where 25% of NMA utterances were reserved for testing and the rest were included in training. For IEMOCAP, Session 5 was reserved for testing, and Sessions 1-4 were split into training and validation with a ratio of 4:1. For the CREMA-D dataset, the MA subset was split into train, validation, test in the ratio 70 : 15 : 15 following Ristea and Ionescu (2021).

heading: 6.3 Model structure
text: The backbone structure used in this paper is illustrated in Figure 2 which follows an upstreamdownstream paradigm (Bommasani et al., 2021). The upstream model uses the universal speech model (USM) (Zhang et al., 2023) with 300M parameters which contains a CNN-based feature extractor and 12 Conformer (Gulati et al., 2020) encoder blocks of dimension 1024 with 8 attention heads. The structure of the downstream model follows SUPERB (Yang et al., 2021), a benchmark for evaluating pre-trained upstream models, which performs utterance-level mean-pooling followed by a fully-connected layer. The pre-trained upstream USM model is frozen. The downstream model computes the weighted sum of the hidden states extracted from each layer of the upstream model. The backbone structure has been shown to outperform state-of-the-art methods for emotion classification (see Table 5 in Appendix C). The implementation details for model training can be found in Appendix A.

heading: 7 Results
text: This section presents experimental results of the three approaches for handling ambiguous emotion: incorporating NMA as an extra class (Section 7.1),
detecting NMA as OOD (Section 7.2), and representing emotion as distributions (Section 7.3). The average of three runs are reported for all results.

heading: 7.1 Including NMA as an additional category degrades the performance
text: The first approach, which trains an emotion classifier with NMA as an extra class, is denoted “MLE+” in Table 1 and 2. Some of the NMA utterances are included in MLE+ training while the remainder are used for testing. Therefore, OOD detection is evaluated only on NMA (test) data for MLE+. The results reveal that the addition of the NMA class has a detrimental impact on the classification performance of the original MA emotion classes. Comparing to MLE, MLE+ observes a ∼23% relative decrease in both ACC and UAR on IEMOCAP and a ∼20% relative decrease in ACC and UAR on CREMA-D. The confusion matrices of the MLE+ model can be found in Appendix D, which shows that NMA itself is challenging to predict and it also confuses the model when predicting classes such as neutral, sad, frustrated, and disgust.

heading: 7.2 Detecting NMA as OOD
text: The proposed EDL-based method is compared to baselines in Tables 1 and 2 on the IEMOCAP and CREMA-D datasets respectively. First, as shown
by the values of ACC and UAR, the proposed method demonstrates comparable classification performance to the baselines, suggesting that the extension for uncertainty estimation does not undermine the model’s capabilities. Although the Ensemble achieves the highest accuracy on CREMA-D, it involves training 10 individual systems. The proposed method achieves overall the best classification performance with only a tenth of the computational cost of Ensemble during both training and testing. In addition, the proposed method offers superior model calibration, as shown by the lowest values of ECE and MCE. It also outperforms the baselines in effectively identifying NMA as OOD samples, as shown by the highest AUROC and AUPRC values.
Figure 3 shows the change of accuracy when samples with uncertainty larger than a threshold are excluded. The model tends to provide less accurate predictions when it is less confident about its prediction, shown by the decrease of classification accuracy when the uncertainty threshold increases, which demonstrates the effectiveness of uncertainty prediction.

heading: 7.3 Estimating emotion distribution
text: The proposed EDL* methods were first evaluated in terms of majority class prediction. The results of distribution-based methods on classification of MA data are shown in Table 3. Compared to the classification-based methods in Table 1 and Table 2, it can be seen that EDL* does not reduce the performance of emotion classification (in terms of ACC and UAR) and model calibration (in terms of ECE and NCE) on MA data. This indicates that the information of the majority class is retained when representing emotion as a distribution. Note that
when representing emotion as a distribution, it is no longer appropriate to consider NMA utterances as OOD samples, as illustrated by case (b) and (c) in Figure 1. Although still trained only on MA data, the proposed distribution-based system shows good generalisation ability in predicting the emotion distribution of NMA data, which we will see shortly. When encountering NMA data during testing, instead of simply returning “I don’t know”, the proposed system can provide reliable estimation of its emotional content, which is a key benefit.
The proposed EDL* methods were then evaluated regarding distribution estimation. Table 4 compares EDL* to the baselines in terms of the negative log likelihood of sampling target labels from the predicted emotion distribution. As can be seen from the table, EDL* produce improved distribution estimation, achieving the smallest NLL values on both MA and NMA data. Among the two EDL* methods employing different regularisation terms, EDL* with R2 (defined in Eqn. (13)), which directly applies regularisation to the predicted distribution, exhibits better distribution estimation with-
0.5 1.0
1
2
3
out sacrificing model calibration. A reject option was then evaluated for NLL (instead of accuracy) to examine model calibration. For a well-calibrated model, an increase in the NLL value, which is associated with poorer distribution estimation, is expected when the model becomes less confident. Figure 4 visualises the change of NLL for MA data and NMA data when uncertainty increases. For MA data, i.e. the type of data that has been seen by the models during training, most methods can successfully reject uncertain samples except for MLE and Ensemble, as shown by an increase in NLL values when the uncertainty threshold increases. However, for NMA data which the model hasn’t seen in training, only the EDL* methods exhibit the ability to demonstrate an increasing trend in NLL values.
The proficiency of the proposed EDL* methods for estimating the emotion distribution and providing reliable confidence predictions, demonstrates the method’s capacity to estimate both aleatoric uncertainty (Matthies, 2007; Der Kiureghian and Ditlevsen, 2009; Hu et al., 2024), arising from data complexity (i.e., the ambiguity of emotion expression), and epistemic uncertainty, corresponding to the amount of uncommitted belief in subjective logic.

heading: 7.4 Case study
text: Emotion distributions estimated by different methods are visualised against the label distributions for two representative examples in Figure 5. In general, distribution-based methods show superior performance for distribution estimation than
classification-based methods. In the case of utterance (a) which receives two “angry” labels and two “frustrated” labels, the proposed EDL* methods stands out by effectively capturing the tie between the emotions, whereas the predictions of classification-based methods tend to be predominantly skewed towards “frustrated”. As for utterance (b), where both “disgust” and “neutral” receive four votes, along with two votes for “angry” and one for “fear”, the emotion distributions predicted by the EDL* methods also show a similar pattern. These examples show that the proposed method can not only provide a more comprehensive emotion representation but also better reflect the variability of human opinions. Additional examples can be found in Appendix J and Appendix K.

heading: 8 Conclusions
text: This paper re-examines the emotion classification problem, starting with an exploration of ways to handle data with ambiguous emotions. It is first shown that incorporating ambiguous emotions as an extra class reduces the classification performance of the original emotion classes. Then, evidence theory is adopted to quantify uncertainty
in emotion classification which allows the classifier to output “I don’t know” when it encounters utterances with ambiguous emotion. The model is trained to predict the hyperparameters of a Dirichlet distribution, which models the second-order probability of the probability assignment over emotion classes. Furthermore, to capture finer-grained emotion differences, we transform the emotion classification problem into an emotion distribution estimation problem. All annotations are taken into account rather than only the majority opinion. A novel approach is proposed which extends standard EDL to quantify uncertainty in emotion distribution estimation. Experimental results show that given an utterance with ambiguous emotion the proposed approach is able to provide a comprehensive representation of its emotion content as a distribution with a reliable uncertainty measure.
Ethics Statement
In this work, all human annotations used for training were taken from existing publicly available corpora. No new human annotations were collected.
In subjective tasks like emotion recognition, there is usually no single “correct” answer. The conventional approach of imposing a single ground truth through majority voting may overlook valuable nuances within each annotator’s evaluation and the disagreements between them, potentially resulting in the under-representation of minority views. This study, instead of exclusively relying on the majority vote, integrates emotion annotations from all annotators for each utterance during model training. It is hoped that this work could contribute to a more inclusive representation of human opinions.
Limitations
The proposed approach requires the raw labels from different human annotators for each sentence to be provided by the datasets. Although validated only for emotion recognition, the proposed method could also be applied to other tasks with disagreements in subjective annotations, which will be investigated in future work.
Different people perceive emotion differently and hence the motivation of the paper is to handle such ambiguity. It may be that if annotators were to also provide confidence ratings during annotation, which is not the case in the emotion datasets we have used, then this information could be used to
weight the observations when estimating the emotion distribution.

heading: Acknowledgements
text: Work performed while W.W. was an intern at Google. W.W.’s PhD research is supported by the Cambridge Trust.

heading: B Discussion of statistical significance
text: The results reported in Section 7 are the average of three runs with different seeds. Regarding ECE, MCE, AUROC and AUPRC, the improvements in the results in bold are consistent across all three runs. For ACC and NLL, the differences are statistically significant with p < 0.05.

heading: C Comparing the backbone structure to SOTA models
text: The USM-based backbone structure is evaluated following the setup of the emotion recognition task of the SUPERB benchmark (Yang et al., 2021): four-way emotion classification (happy, sad, angry, neutral) on IEMOCAP dataset with leave-onesession-out five-fold cross validation. The USM300M model is compared to multiple state-of-theart models of similar size. Results are shown in Table 5. Except for the USM-300M model used in the paper, all other results are quoted from the cited papers. As shown in the table, the USM-based backbone structure outperforms other state-of-theart methods2 and yields the highest accuracy.

heading: D Confusion matrices of MLE+
text: As described in Section 7.1, including NMA as an extra class reduces the classification performance. This section analyses the confusion matrices of the MLE+ model, shown in Figure 6. It can be seen from the bottom right entry that NMA itself is challenging to predict, possibly because it essentially contains a mix of different emotion content. The last column demonstrates that grouping these utterances into one class can confuse the model,
1https://github.com/google/paxml 2https://superbbenchmark.org/leaderboard
particularly for the classes neutral, sad, frustrated, and disgust.

heading: E Alternative activation functions
text: As described in Section 3.2, ReLU is used as the output activation function in EDL to make sure the evidence is non-negative. This section compares the use of different activation functions including ReLU, softplus and exponential functions. The three activation functions are plotted in Figure 7.
As shown in Table 6, using exponential function tends to result in less effective model calibration, shown by the largest ECE and MCE values. It also produces worse performance for NMA detection, shown by the smallest AUROC and AUPRC. Figure 8 shows the reject option for accuracy of EDL with different activation functions. A drop in accuracy when the uncertainty threshold increases from 0 to 0.1 is observed for model using exponential activation. This indicates that exponential activation tends to lead to smaller uncertainty.0 1
0.00
0.25
0.50
0.75
1.00
Pr ob
ab ili
ty
The empirical cumulative distribution function (ECDF) of uncertainty and entropy on IEMOCAP and CREMA-D are plotted in Figure 9 and Fig-
ure 10 respectively. It can be seen that exponential activation leads to smaller uncertainty and entropy, which echos the statement in Section 3.1 that exponential activation tends to inflate the probability of the correct class.
0 1 Entropy
0.00
0.25
0.50
0.75
1.00
Pr ob
ab ili
ty
EDL(ReLU) EDL(Exp) EDL(Softplus)
0.0 0.5 1.0 Uncertainty
0.0
0.5
1.0
Pr ob
ab ili
ty
(a) ECDF of uncertainty
0 1 Entropy
0.0
0.5
1.0
Pr ob
ab ili
ty
(b) ECDF of entropy
Figure 9: Empirical CDF of uncertainty (left) and entropy (right) on IEMOCAP for EDL method with different activation functions.
0 1 Entropy
0.00
0.25
0.50
0.75
1.00
Pr ob
ab ili
ty

heading: F Analysis of regularisation coefficient
text: This section analyses the effect of regularisation coefficient λ in Eqn. (7) for EDL. The empirical CDF of uncertainty and entropy when different regularisation coefficient was used is plotted in in Figure 11 and Figure 12 for IEMOCAP and CREMA-D respectively. We observed that larger lambda values lead to a larger entropy and uncertainty. This aligns with the definition of the regularisation term in Eqn. (7) which tends to enforce a flat prior with small evidence.

heading: G Reject option for NLL on CREMA-D
text: This section shows the reject option for NLL on CREMA-D dataset. The change of NLL for MA data and NMA data when the uncertainty threshold increases are shown in Figure 13. For a wellcalibrated model, an increase in the NLL value is expected when the model becomes less confident. Similar to the findings in Figure 4, most methods are effective for rejecting uncertain samples in the MA data, as shown by an increase in NLL values when the uncertainty threshold increases. However, only the EDL* methods are successful for NMA data.
0.5 1.0
1
2
3

heading: H Differences between the proposed EDL* method and Dirichlet prior networks
text: The proposed EDL* method is similar to the Dirichlet prior network (DPN) used in Wu et al. (2022) but essentially different. Both methods involve a Dirichlet distribution, but the difference lies in the problem formulation and the connection of the Dirichlet distribution to evidence theory.
(i) Problem formulation. In DPN, individual emotion class labels provided by human annotators are treated as one-hot categorical distributions ({ηm}Mm=1) and the model is trained by maximising the likelihood of sampling those one-hot categorical distributions given the Dirichlet prior.
LNLLDPN = log p({ηm}Mm=1|α) (16)
However, the proposed EDL* method preserves the target emotion labels as discrete class labels ({ym}Mm=1) which are drawn from an unknown categorical likelihood (η). The model is then trained by maximising the likelihood of sampling discrete labels given the Dirichlet prior by marginalising out all possible categorical distributions.
LNLLEDL* = log P({ym}Mm=1|α)
= log ∫ P({ym}Mm=1|η)p(η|α)dη (17)
Therefore, the problem setting and NLL loss for two methods are essentially different. A drawback of DPN is that it greatly reduces the classification accuracy (as also pointed out by Wu et al. (2022)), while the proposed EDL* method improves over
DPN by preserving or even boosting the classification performance.
(ii) Connection to evidence theory. By connecting to evidence theory, the proposed method provides an explicit uncertainty score (Eqn. (3)) while DPN doesn’t. Therefore, DPN can only estimates aleatoric uncertainty (i.e., data complexity) but it cannot predict epistemic uncertainty (i.e., model confidence). In contrast, the proposed EDL* method provides superior performance in estimating both aleatoric and epistemic uncertainty, as explained in the last paragraph of Section 7.3.
The performance of DPN and EDL* are compared in the Table 7 and Table 8 (results taken for EDL* from Table 3 and Table 4 with the addition of DPN results). It can be seen that the proposed EDL* methods outperform DPN in all metrics on both datasets.

heading: I Analysis of Instances When OOD Detection Fails
text: This section includes examples and analysis of particular utterances where OOD detection is problematic and compares these examples across the techniques discussed. It is shown that distributionbased methods improve over the classificationbased systems in handling complex ambiguous emotions.
Consider the following false negative case where the OOD detection model fails to detect an NMA sample. Utterance “Ses04M_impro02_F024” from the IEMOCAP dataset has two “angry” labels and two “frustrated” labels as shown in Figure 14. The EDL system predicts this utterance as “frustrated” with a belief mass of 0.567 and an overall uncertainty score of 0.433, which reveals that the system fails to detect the utterance as NMA.
A possible cause of this failure is that the model gets confused by MA utterances seen in the training that convey similar emotional content, such as “Ses05M_impro01_M014” whose annotations are shown in Figure 15 with an MA emotion class “frustrated”. Although one annotator considered it as “angry”, the MA ground-truth target was “frustrated” in a classification-based system.
Both utterances occur within a dyadic situation where two people disagree, with the speaker being the one who compromises, feeling unhappy and frustrated. Such similar emotional content may confuse a classification-based system to also predict the NMA utterance as frustrated. It is worth noting that data with the same distribution as “Ses04M_impro02_F024”, which has tied votes, is not included during the training of a classificationbased model because there is no majority vote available to serve as ground truth.
This complex emotional expression can be better described by the distribution-based EDL* systems. The predicted distribution of the MA utterance is shown in Figure 16 and the predicted distribution of the NMA utterance can be found in Figure 5(a). It can be seen that the classification-based methods produce a similar distribution for the two utterances, with “frustrated” being dominant. However, the proposed EDL* methods can better match the label distribution and distinguish between these two
cases. Although not been trained on NMA data, the EDL* methods are still capable of providing accurate predictions of its emotional content. This is a key benefit of the distribution-based approaches.
Next, we provide a typical false positive instance where an MA utterance is mis-detected as OOD. The MA utterance “1087_IEO_FEA_LO” from the CREMA-D dataset has four “neutral”, two “sad”, and three “fear” human labels as in Figure 17. The NMA utterance “1052_ITH_FEA_XX” has four “neutral”, two “sad”, and four “fear” human labels as in Figure 18. The OOD system successfully predicts the NMA utterance as OOD with an overall uncertainty of 0.691 while also predicting the MA utterance as an OOD sample with an overall uncertainty of 0.623.3 This failure is possible because the MA utterance “1087_IEO_FEA_LO” contains a complex mixture of emotions shown by the rather flat label distribution similar to “1052_ITH_FEA_XX”, which confuses the OOD detection system. Note that the MA class “neutral” in Figure 17 comprises only 44+2+3 × 100% = 44.4% of the annotations and hence is not an absolute majority, which reduces the severity of this detection error.
3Assume the OOD detection threshold is taken as 0.5.
Again, the distribution-based EDL* methods show superior capability in handling such complex cases. The predicted distribution of the MA utterance is shown in Figure 19 and the predicted distribution of the NMA utterance can be found in Figure 21(b). For the MA utterance, although human opinions diverge, the classification-based methods only capture the majority prediction, with the predicted distribution being dominated by “neutral”. However, the emotion distribution predicted by the proposed EDL* methods retains the probability for “sad” and “fear” which accounts for the minority human opinions. Therefore, we show that the proposed EDL* method improves over the OOD system by providing a more comprehensive representation of emotional content as well as a more inclusive representation of human opinions.

heading: J Further visualised examples: IEMOCAP
text: This section shows more examples on IEMOCAP. Aligning with the findings in Section 7.4, EDL* methods show better estimation of emotion distribution.

heading: K Further visualised examples: CREMA-D
text: This section shows more examples of CREMA-D. As can be seen, EDL* methods can better approximate the distribution of emotional content of an utterance.

heading: Future Work
text: Although validated only for emotion recognition, the proposed method could also be applied to other tasks with disagreements in subjective annotations, which will be investigated in future work. Different people perceive emotion differently and hence the motivation of the paper is to handle such ambiguity. It may be that if annotators were to also provide confidence ratings during annotation, which is not the case in the emotion datasets we have used, then this information could be used to
weight the observations when estimating the emotion distribution