Modelling Variability in Human Annotator Simulation (2024)

Abstract: Human annotator simulation (HAS) serves as a cost-effective substitute for human evaluation tasks such as data annotation and system assessment. It is important to incorporate the variability present in human evaluation into HAS, since it helps capture diverse subjective interpretations and mitigate potential biases and overrepresentation. This work introduces a novel framework for modelling variability in HAS. Conditional softmax flow (S-CNF) is proposed to model the distribution of subjective human annotations, which leverages diverse human annotations via meta-learning. This enables efficient generation of annotations that exhibit human variability for unlabelled input. In addition, a wide range of evaluation metrics are adopted to assess the capability and efficiency of HAS systems in predicting the aggregated behaviours of human annotators, matching the distribution of human annotations, and simulating the inter-annotator disagreements. Results demonstrate that the proposed method achieves state-of-the-art performance on two real-world human evaluation tasks: emotion recognition and toxic speech detection.

heading: 1 Introduction
text: Human evaluation is fundamental to machine learning research. It guides processes such as data annotation and model assessment, including for instance perceptual quality evaluation (Ma et al., 2015; Talebi and Milanfar, 2018; Ramesh and Sanampudi, 2022), annotation generation for weak supervision (Ratner et al., 2016; Wu et al., 2022a), and model optimization based on human preference (Schatzmann et al., 2007; Gür et al., 2018; Ruiz et al., 2019; Shi et al., 2019; Chen et al., 2019; Lin et al., 2021). Collecting human annotations or evaluations often requires substantial resources and may expose human annotators to distressing
*Equal contribution. Code available: https://github.com/W-Wu/HAS_CNF
and harmful content in sensitive tasks (e.g., toxic speech detection, suicidal risk prediction, and depression detection). This inspires the exploration of human annotator simulation (HAS) as a scalable and cost-effective alternative, which facilitates large-scale dataset evaluation, benchmarking, and system comparisons.
Variability is a unique aspect of real-world human evaluation. Individual variations in cognitive biases, cultural backgrounds, and personal experiences (Hirschberg et al., 2003; Wiebe et al., 2004; Haselton et al., 2015) can lead to variability in human interpretation (Maniati et al., 2022). It has been argued that achieving a single deterministic “ground truth” in subjective tasks like human evaluation is not feasible, nor essential (Alm, 2011; Wu et al., 2022b). Therefore, HAS should incorporate the variability present in human evaluation rather than solely relying on majority opinions. This mitigates potential biases and over-representation in scenarios where dominant opinions could potentially overshadow minority viewpoints, thus promoting fairness and inclusivity.
This work investigates modelling subjective human annotation distributions and simulating human-like annotations. We propose a novel framework which formulates HAS as a zero-shot density estimation problem. A new model, conditional softmax flows (S-CNFs), is proposed which leverages diverse human annotations via meta-learning. This enables efficient generation of annotations that exhibit human variability for unlabelled input. To the best of our knowledge, this is the first work that incorporates human variability into HAS without requiring human annotators to be dynamically involved in the process, while remaining scalable for large crowd-sourced datasets. Moreover, a range of evaluation metrics are adopted to assess the capability and efficiency of HAS systems regarding prediction of the majority opinion, estimation of the human annotation distribution, and simulation of the
1139
inter-annotator disagreements. The proposed approach is evaluated on two real-world applications: emotion recognition and toxic speech detection. Empirical results demonstrate that the proposed method achieves state-of-the-art performance on modelling variability in HAS.

heading: 2 Human Annotator Simulation (HAS)

heading: 2.1 The Variability in Human Evaluation is Valuable
text: Each individual’s perception of the world is unique and influenced by their physical state and cognitive biases. This leads to diverse and subjective interpretations. Such subjectivity can be manifest in various tasks such as emotion recognition (Hirschberg et al., 2003; Mihalcea and Liu, 2006), perceptual quality assessment (Wiebe et al., 2004; Seshadrinathan et al., 2010), and user experience evaluation (Zen and Vanderdonckt, 2016). Rather than seeking to reduce the variability in annotations, it is important to account for annotators’ subjective interpretations when designing a human annotator simulator. The importance of variability in HAS can be demonstrated by the following examples:
Revealing data ambiguity. Incorporating the variability in human perception empowers HAS to reveal potential ambiguity or complexity in data, providing valuable insights for further analysis2.
Mitigating bias and over-representation. Incorporating the variability in human judgements prevents HAS from being biased towards a certain perspective and ignoring minority viewpoints, leading to a more inclusive representation of opinions where all viewpoints are given due consideration (Dixon et al., 2018; Hutchinson et al., 2020).
Improving model alignment. Optimization based on human feedback has led to superior performance on tasks such as text generation (Christiano et al., 2017; Ouyang et al., 2022; Rafailov et al., 2023), which aligns the behaviour of language models with human preferences. HAS could be helpful in this task, as it is an efficient and cost-effective alternative to generating human feedback.
2Taking emotion perception as an example, there are certain cases that convey fairly clear emotional expressions (e.g., laughing) and most annotators agree that the speaker is happy. However, there also exist cases where the emotion is more subtle and human opinions can easily diverge. For instance, in a dyadic situation where two people disagree, with the speaker being the one who compromises, some people would perceive the emotion as frustrated while others may interpret it as angry. These types of data contain ambiguous emotion that is inherently more complex to deal with.

heading: 2.2 Problem Formulation
text: Denote an event as di, which consists of a descriptor (e.g., an utterance or text) xi and a set of Mi human annotations Di = {η(m)i }Mim=1 for xi. Note that different events may be labelled by different sets of annotators. Given a dataset of training events D = {(xi,Di)}Ni=1, HAS aims to learn the conditional annotation distribution p(ηi|xi) given the observations Di of ηi provided by different annotators. For a unseen test descriptor x∗, HAS can then predict p(η∗|x∗) to simulate human-like annotations D∗ = {η(m)∗ }M∗m=1 in a way that reflects how it would be labelled by human annotators.

heading: 2.3 Related Work
text: Prior work mainly investigated three approaches to simulating human annotations.
The first approach uses a single proxy variable η′i (e.g., majority vote) to summarize all annotations for each descriptor xi (Kim et al., 2013; Djuric et al., 2015; Patton et al., 2016; Poria et al., 2017). This creates a proxy dataset D′ = {(xi,η′i)}Ni=1 and converts HAS into a supervised learning problem, which is usually solved by fitting a discriminative model to estimate the conditional distribution for the proxy variable. During testing, given an unseen descriptor x∗, the model predicts the proxy variable η′∗ for x∗. Clearly, modelling a single proxy variable as in this approach fails to take into account the subjectivity and diversity in human behaviour and perception. Other work incorporated the variance of human annotations into the proxy variable (Deng et al., 2012; Prabhakaran et al., 2012; Plank et al., 2014; Dang et al., 2017; Han et al., 2017; Leng et al., 2021). However, all these approaches still focus on obtaining the “correct” label (e.g., aiming for improved prediction accuracy) and minimizing the discrepancy among annotators (e.g., reducing “noise” in annotations) rather than embracing inter-annotator disagreements.
The second approach explicitly models the behaviours of different annotators using different individual models in an ensemble or different heads in a single model (Fayek et al., 2016; Chou and Lee, 2019; Davani et al., 2022). This approach is computationally feasible only when the number of annotators is relatively small and when a sufficient quantity of annotation is available for each annotator, which is not applicable to large crowd-sourced datasets (Lotfian and Busso, 2019; Mathew et al., 2021) that are common in real-world applications.
The third approach approximates subjective probability distributions using Markov chain Monte Carlo (MCMC) with people (Sanborn and Griffiths, 2007; Harrison et al., 2020), which requires human annotators to be involved in the process in a dynamic setting. These methods present the descriptor x∗ to human participants and asks them to provide a sequence of decisions D∗ following the Metropolis-Hastings acceptance rule. The annotation distribution p(η∗|x∗) is then estimated based on D∗. In other words, this requires access to human annotations D∗ for estimating the annotation distribution for each x∗, and there is no obvious way to transfer information between different events. Therefore, these methods cannot be applied to simulate annotation distributions for unlabelled test descriptors.

heading: 3 A Meta-learning Framework for HAS
text: This paper proposes a novel framework for HAS that meta-learns a conditional softmax flow (SCNF) to estimate the human annotation distribution p(η|x) across all training events in D. The proposed model learns to learn (i.e., meta-learns) how to estimate the underlying distribution of human annotations Di for any given descriptor xi by leveraging the diverse human annotations, rather than designing a proxy variable to summarize Di as in the first approach described in Sec. 2.3. Unlike the second approach in Sec. 2.3 which separately models each individual human annotator with a different model, our method is compatible with large crowd-sourced datasets since it amortizes across annotators with a single S-CNF model. Moreover, our model is a zero-shot human annotation simulator which can estimate the human annotation distribution p(η∗|x∗) for any unseen test descriptor x∗ without access to any human annotations D∗ for x∗, in contrast to the third method in Sec. 2.3 which requires human annotators to be dynamically involved in the process of labelling x∗.

heading: 3.1 A Latent Variable Model for HAS
text: The proposed framework for HAS is realized by a latent variable model3:
pθ(y|x) = ∫∫
p(y|v)pϕ(v|z)pΛ(z|x)dvdz, (1)
where the conditional prior pΛ(z|x) learns to summarize useful information about the input descriptor x and encode the possible disagreements over x among different human annotators, which is helpful for the likelihood pϕ(y|z) = ∫ p(y|v)pϕ(v|z)dv to simulate human-like annotations for x. The proposed zero-shot human annotator simulator is illustrated in Figure 1. Specifically, the conditional prior is modelled by a conditional factorized Gaussian distribution pΛ(z|x) = N (z|µΛ(x), diag(σ2Λ(x))) whose mean µΛ(x) and variance σ2Λ(x) are parameterized by a neural network gΛ with parameters Λ.
The intermediate variable v is obtained by a deterministic invertible transformation pϕ(v|z) = δ(v−fϕ(z)), where fϕ(z) is parameterized by an invertible neural network with parameters ϕ, and δ(·) is the multivariate Dirac delta function. This results in a conditional normalizing flow (CNF):
pθ(v|x) = ∫ δ(v − fϕ(z))pΛ(z|x)dz
= pΛ ( f−1ϕ (v) ∣∣∣x ) ∣∣∣∣∣det ( ∂f−1ϕ (v) ∂v )∣∣∣∣∣ , (2)
where det(·) denotes the determinant operator, ∂f−1ϕ (v)/∂v denotes the Jacobian matrix of f−1ϕ (v), and θ := {ϕ,Λ} denotes all parameters in this base CNF. This modelling choice has the advantage of having tractable marginal likelihood as in Eqn. (2) while not restricting the intermediate variable v to a specific type of distribution as in previous methods (e.g., Gaussian (Han et al., 2017)
3For clarity, we use different notations for human annotations η and model outputs y.
and Student’s-t (Wu et al., 2023)), thus offering enhanced tractability, flexibility and generality. In addition, samples can be efficiently drawn from this model by first drawing z ∼ pΛ(z|x) from the conditional prior and then computing the deterministic flow transformation v = fϕ(z).
Finally, the output variable y is obtained by augmenting the intermediate variable v with the transformation p(y|v). For continuous annotations, the identity transformation p(y|v) = δ(y− v) is used. However, real-world human evaluation tasks often involve discrete annotations (e.g., human annotators are usually instructed to choose from a predefined set of options or scores). In the next section, a new model class is introduced to accommodate discrete annotations.

heading: 3.2 Conditional Softmax Flows (S-CNFs)
text: We propose a new model class called conditional softmax flow (S-CNF) to accommodate discrete annotations. The workflow of S-CNF is illustrated in Figure 2. S-CNFs augment the base CNFs by applying the softmax function p(y|v) = δ (y − softmax(v)) to transform the continuous intermediate variable v into categorical probabilities y. Let c be a categorical variable with probability P(c = k|y) = yk (k = 1, · · · ,K), which represents the categorical annotation for an descriptor x, with P(c = k|v) =∫ ykδ (y − softmax(v)) dy = softmax(v)k. The
marginal likelihood of S-CNF is given by
Pθ(c = k|x) = ∫ P(c = k|v)pθ(v|x)dv (3)
where pθ(v|x) is the marginal likelihood of the base CNF defined in Eqn. (2). Since the marginal likelihood of the S-CNF given in Eqn. (3) is analytically intractable due to the softmax transformation, we propose to approximate it using variational inference (Wainwright et al., 2008) with a learnable mean-field Gaussian variational posterior qΩ(v|y) = N (v|µΩ(y),diag(σ2Ω(y))), whose mean µΩ(y) and variance σ2Ω(y) are parameterized by a neural network hΩ with parameters Ω. This can be seen as a probabilistic inverse of the softmax transformation p(y|v). Applying Jensen’s inequality to the log marginal likelihood of the SCNF in Eqn. (3), we obtain a tractable evidence lower bound (ELBO):
log Pθ(c = k|x) ≥ EqΩ(v|y)[log P(c = k|v) + log pθ(v|x)− log qΩ(v|y)]. (4)
It is worth noting that the softmax flow likelihood P(c = k|v) = softmax(v)k places non-zero probability mass for every category k = 1, · · · ,K, which is different from argmax flow (Hoogeboom et al., 2021) whose likelihood only places probability mass for a single category. From a modelling perspective, softmax flow has a better capacity to represent the variability and uncertainty in human annotations. From an optimization perspective, the ELBO for softmax flow is always welldefined, whereas the ELBO for argmax flow is not defined when the model output does not match the human annotation in which case the log-likelihood would be log(0) and requires additional thresholding tricks to fix (Hoogeboom et al., 2021).

heading: 3.3 A Meta-learning Objective for S-CNFs
text: Using the variational approximation defined in Eqn. (4), the loss L(θ,Ω;di) for S-CNF on a single event di can be defined as the average negative ELBO evaluated on the set of the human annotations Di = {η(m)i }Mim=1 for the corresponding descriptor xi:
L(θ,Ω;di) =
− 1 Mi
Mi∑
m=1
E qΩ(v|η (m) i )
[ K∑
k=1
η (m) i,k log P(ci = k|v)
+ log pθ(v|xi)− log qΩ(v|η (m) i )
] , (5)
where the expectation over the variational posterior is approximated by Monte Carlo simulation with the reparameterization trick (Kingma and Welling, 2014). Following the episodic training scheme (Vinyals et al., 2016; Snell et al., 2017; Chen et al., 2023), we treat density estimation on each event as a learning problem and randomly sample a subset of such learning problems to train on at each step during meta-training. This results in a metalearning objective across the training events in D:
Lmeta(θ,Ω;D) = Edi∼p(D)[L(θ,Ω;di)], (6)
where p(D) denotes the uniform distribution over D. Intuitively, this objective maps each human annotation to the latent space of the corresponding descriptor by the S-CNF during meta-training, which helps the model to build a diverse latent representation that captures the variability in human annotations across different descriptors.
At test time, the S-CNF can simulate human-like annotations for an unseen, unlabelled descriptor x∗ by first drawing v(m)∗ ∼ pθ(v|x∗) from the base CNF then applying the softmax function y(m)∗ = softmax(v (m) ∗ ) for m = 1, · · · ,M∗, where M∗ denotes the number of annotations to be simulated. Note that each sample of S-CNF is a categorical distribution with probabilities y(m)∗ . More details can be found in Appendix D.

heading: 4 Experimental Setup
text: The proposed framework for variability-aware HAS is evaluated on two real-world applications for speech and natural language processing: emotion class labelling and toxic speech detection. A wide range of evaluation metrics are adopted to assess the performance of HAS systems.

heading: 4.1 Evaluation Tasks and Datasets
text: Emotion class labelling. The highly subjective perception of emotion often results in disagreements among human annotators. Most emotion datasets employ multiple annotators to label each utterance while prior works typically use the majority vote as the ground-truth (Busso et al., 2008; Poria et al., 2019; Wu et al., 2021). A variability-aware HAS system better handles different opinions among human annotators and enhances the fairness of emotion class annotation. MSP-Podcast (Lotfian and Busso, 2019) is one of the largest publicly available datasets in speech emotion recognition, which contains natural English speech from podcast record-
ings annotated using crowd-sourcing. Each utterance has 6.7 annotations on average. Release 1.6 was used in our experiments, which contains 50k+ utterances from 1k+ speakers. The standard splits of training, validation and test were used. Emotion labels were grouped into five categories: angry, sad, happy, neutral, and other. 16.5% of the utterances do not have a majority agreed emotion class.
Toxic speech detection aims to filter out harmful language, which is crucial for respectful online environments and healthy communications among users. A variability-aware HAS system accounts for comprehensive understanding of hate speech, which is a good substitute for human annotators to reduce their exposure to distressing and harmful content. HateXplain (Mathew et al., 2021) was used in our experiments, which is a popular dataset for toxic speech detection, which contains 20k+ text posts from Twitter and Gab. These posts are labelled using crowd-sourcing with the commonly used three-category annotation: hate, offensive, normal. Each post is annotated by three annotators. Cases where all three annotators choose a different class (919 out of 20,148 posts) were originally excluded from the standard split of the dataset. We incorporate these cases into our training, validation, and test sets in an 8:1:1 ratio to better reflect the inter-annotator disagreements, resulting in 16,118 posts for training, 2,014 for validation, and 2,016 for testing.

heading: 4.2 Evaluation Metrics
text: A range of metrics are adopted to assess the empirical performance of HAS systems in terms of majority prediction, distribution matching, and human variability simulation.
Majority prediction. Classification accuracy (Acc) for the majority vote is evaluated for all test inputs with majority-agreed human annotations.
Distribution matching. Negative log likelihood (NLL) is used to evaluate how well the model estimates the human annotation distribution: NLLall = − 1N ∑N i=1( 1 Mi ∑Mi m=1 log pθ(η (m) i |xi)).
Inter-annotator disagreement simulation. Apart from evaluating the goodness of fit, additional metrics are adopted to explicitly measure how well the model simulates the variability and disagreements in human annotations: (i) the root mean squared error of the standard deviations of the annotations for all test inputs: RMSEs = √ 1 N ∑N i=1 (σi − si)2, where
σi = 1 K ∑K k=1 √ 1 Mi ∑Mi m=1(η (m) i,k − η̄i,k)2, si =
1 K ∑K k=1
√ 1
M∗
∑M∗ m=1(y (m) i,k − ȳi,k)2, K denotes
the number of classes, η̄ is the average of human annotations, y is a simulated annotation, ȳ is the average of simulated annotations, and M∗ is the number of simulated annotations; (ii) the absolute error of the average standard deviations of the annotations for all test inputs: E(s̄) = |σ̄ − s̄|, where σ̄ = ∑N i=1 σi and s̄ = ∑N i=1 si; (iii) Fleiss’s kappa
(κ) (Fleiss, 1971), a real number between −1 and +1, with −1 indicating no observed agreement and +1 indicating perfect agreement. The absolute error between the kappas of human annotations (κ) and simulated annotations (κ̂) for all test inputs is reported: E(κ̂) = |κ̂− κ|.

heading: 4.3 Baselines
text: The proposed S-CNF method is compared to baselines of various types such as ensemble methods, Bayesian methods, and conditional generative models. This includes deep ensemble (Ensemble) (Lakshminarayanan et al., 2017), Monte-Carlo dropout (MCDP) (Gal and Ghahramani, 2016), Bayes-bybackprop (BBB) (Blundell et al., 2015), conditional variational autoencoder (CVAE) (Kingma and Welling, 2014), conditional argmax flow (ACNF) (Hoogeboom et al., 2021), and Dirichlet prior network (DPN) (Malinin and Gales, 2018; Wu et al., 2022b). We fit each method to all available human annotations for all utterances in the training set, tune hyperparameters on the validation set, and report performance on the test set. M∗ = 100 samples are used to compute evaluation metrics at test time. Ensemble only consists of 10 systems due to its expensive computational cost. Details about the configuration of the baseline models can be found in Appendix E.

heading: 4.4 Backbone Architecture
text: All compared methods use the same upstreamdownstream feature encoder gΛ to extract features from descriptors. The upstream model (Bommasani et al., 2021) is pre-trained on a large amount of unlabelled data to learn universal representations. WavLM (Chen et al., 2022) and RoBERTa (Liu et al., 2019) are used as the pre-trained upstream models for speech and text descriptors respectively. The downstream model consists of two Transformer encoder blocks followed by two fully connected (FC) layers, which are fine-tuned to target specific applications. The invertible flow model fθ
uses three real NVP blocks (Dinh et al., 2017) and the variational encoder hΩ contains an FC layer. Details about the structure and implementation can be found in Appendix D and Appendix F.

heading: 5 Results and Analysis
text: The proposed method is evaluated according to the setup in Sec. 4. The evaluation results along with case study of representative examples demonstrates the superior capability of the proposed method in capturing the aggregated behaviours of human annotators, matching the distribution of human annotations, and simulating the variability of human interpretation. For each metric, mean value with standard error over three independent runs are reported for all methods.

heading: 5.1 Performance
text: Table 1 compares all methods in terms of majority prediction and distribution matching. The Ensemble achieves the best majority prediction accuracy (Acc) at the cost of training 10 independent systems. The proposed S-CNF achieves the second-best majority prediction accuracy with only a tenth of the computational cost of Ensemble during training and testing. Despite this, we stress that achieving the highest accuracy is not the goal of HAS since accuracy only measures the majority prediction performance and ignores variability
Emotion Training (sec) Testing (sec)
MCDP 7.20±0.10E+03 1.82±0.01E+04 Ensemble 1.46±0.00E+05 1.67±0.01E+03
BBB 7.55±0.01E+03 1.79±0.01E+04 DPN 6.80±0.01E+03 2.90±0.01E+02 A-CNF 7.04±0.02E+03 2.31±0.07E+02 S-CNF 6.99±0.00E+03 2.12±0.02E+02
Toxic Training (sec) Testing (sec)
MCDP 2.42±0.02E+02 5.99±0.02E+02 Ensemble 2.39±0.01E+03 4.00±0.04E+01
BBB 3.22±0.01E+02 5.79±0.01E+02 DPN 1.92±0.01E+02 2.67±0.02E+01 A-CNF 3.14±0.04E+02 1.40±0.11E+01 S-CNF 2.63±0.02E+02 1.37±0.09E+01
Table 3: Computational wall-clock time. The number M∗ of annotations to simulate is set to 10 for ensemble and 100 for all other methods.
in the annotations. More importantly, S-CNF is the best at matching the distributions of human annotations (in terms of NLLall) among all compared methods4. Table 2 reports the test results for all compared methods regarding inter-annotator disagreement annotator simulation. S-CNF again outperforms all compared methods in modelling the variability in human annotations, evident by the smallest RMSEs, E(s̄) and E(κ̂).

heading: 5.2 Computational Time Cost
text: The computational time cost of all of compared methods for the two tasks studied in the paper are shown in Table 3. Denote M∗ as the number of annotations to be simulated. The Ensemble model with M∗ members involves training and testing M∗ individual models, which costs M∗× more training
4Despite a tiny overlap in the error margin of MCDP and S-CNF for emotion class labelling, S-CNF consistently outperforms MCDP for all three runs.
0.8 1.0 1.2
0.590
0.595 Acc
0.8 1.0 1.2
1.390
1.395
NLLall
0.8 1.0 1.2
0.20
0.25
s
0.8 1.0 1.2 0.00
0.05
(s)
0.8 1.0 1.2 0.1 0.2 0.3 0.8 1.0 1.2 0.00 0.05
( )
0.8 1.0 1.2
0.590
0.595 Acc
0.8 1.0 1.2
1.390
1.395
NLLall
0.8 1.0 1.2
0.20
0.25
s
0.8 1.0 1.2 0.00
0.05
(s)
0.8 1.0 1.2 0.1 0.2 0.3 0.8 1.0 1.2 0.00 0.05
( )
(a) Emotion class labelling
0.8 1.0 1.2
0.670
0.675
Acc
0.8 1.0 1.2 0.83 0.84 0.85
NLLall
0.8 1.0 1.2 0.1
0.2
0.3 s
0.8 1.0 1.2 0.00
0.05
(s)
0.8 1.0 1.2
0.50
0.75
0.8 1.0 1.2 0.0
0.1
( )
0.8 1.0 1.2
0.670
0.675
Acc
0.8 1.0 1.2 0.83 0.84
0.85
NLLall
0.8 1.0 1.2 0.1
0.2
0.3 s
0.8 1.0 1.2 0.00
0.05
(s)
0.8 1.0 1.2
0.50
0.75
0.8 1.0 1.2 0.0
0.1
( )
(b) Toxic speech detection
Figure 3: The effect of prior tempering on S-CNF. The x-axis corresponds to the prior temperature.
time and M∗× more testing time. MCDP and BBB require M∗ forward passes during testing to generate M∗ samples and therefore cost M∗× more testing time. All other methods require a single forward pass. S-CNF has slightly longer training time than DPN due to Monte Carlo sampling procedure for estimating the ELBO while being the most efficient for testing.

heading: 5.3 Adjusting the Variability of S-CNFs by Prior Tempering
text: One advantage of S-CNF is that its sample variability can be easily controlled on demand without re-training. This is achieved by tempering the stan-
dard deviation of pΛ(z|x) at test time. Figure 3 explores the effect of prior tempering on the performance. Overall, the trend is clear that the simulated annotations become more diverse as the temperature increases, shown by the increase in the average standard deviation (s̄) and the decrease in Fleiss’s kappa (κ̂) of simulated annotations. As the temperature decreases, simulated annotations tends to concentrate more around the mean. The default temperature value 1 used during training (i.e., no tempering) achieves the best trade-off among majority prediction accuracy (Acc), distribution matching (NLLall), and inter-annotator disagreement simulation (in terms of E(s̄) and E(κ̂)). In addition, prior tempering in S-CNF covers a wider range of dynamics than adjusting the dropout rate in MCDP. More details can be found in Appendix H.

heading: 5.4 Case Study
text: To better illustrate the properties of the annotations simulated by different methods, we visualize the simulated distributions against the ground-truth distributions for three representative examples in Figure 4 (more case study examples can be found in Appendix I). Overall, the mean of the samples generated by S-CNF aligns the best with the average human label, indicating its superior performance in estimating the aggregated behaviours of human annotators. In addition, the samples generated by S-CNF are the most diverse among all compared methods, which manage to simulate the variability of the behaviours of different individual human annotators. In sharp contrast, the samples generated by all the other methods highly concentrate around their sample means. The visualized result for each example is analyzed below.
In case (a), human annotators reach a consensus. The majority of samples generated by S-CNF exhibit prominent peaks aligned with the ground-truth emotion class “neutral”. In contrast, many samples generated by A-CNF peak at other emotion classes.
In case (b), human opinions diverge. The majority of samples generated by S-CNF are sharp categorical distributions peaking at one of the two majority emotion classes “happy” and “neutral”. Additionally, a few samples generated by S-CNF peak at the emotion class “angry”, which manages to simulate the minority viewpoint held by some annotators. Very few human annotators attribute this utterance to the emotion classes “sad” and “other”, and S-CNF likewise produces scarce samples peaking at these classes.
0.04 0.02 0.00 0.02 0.04
0.04
0.02
0.00
0.02
0.04
In case (c), five human annotators give distinct emotion labels, resulting in a tie in the label means. The tie comes from annotators’ diverse individual perceptions of the emotion rather than consensus on its ambiguity. S-CNF is the only model that can simulate both the diverse behaviours of different individual annotators and the aggregated behaviour of all annotators since the individual samples are sharp categorical distributions peaking at one of the five emotion classes and the mean of the samples aligns well with the label mean.
A case study of toxic speech detection exhibits similar trends and can be found in Appendix J.

heading: 6 Conclusions
text: This paper studied human annotator simulation (HAS), a cost-effective alternative to generating human-like annotations for automatic data labelling and model evaluation. A novel framework is proposed to incorporate the variability of human evaluations into HAS. This framework leverages diverse annotations to estimate the distribution of human annotations by meta-learning a conditional softmax flow (S-CNF) on large crowd-sourced datasets. This overcomes the drawbacks of prior work and enables efficient generation of annotations that exhibit human variability for unlabelled test inputs. The proposed method clearly and consistently outperformed a wide range of methods on emotion class labelling and toxic speech detection, achieving the best performance for human annotation distribution matching and inter-annotator disagreement simulation. It is hoped that the proposed method could help mitigate unfair biases and overrepresentation in HAS and reduce the exposure of human annotators to potentially harmful content, thus promoting ethical AI practices.
Limitations
This work focuses on categorical annotations, which is commonly used during human evaluation. Other types of annotations can be accommodated by designing suitable corresponding output transformations p(y|v).
The proposed S-CNF is tested for two representative tasks: emotion recognition and toxic speech detection with speech and text as input respectively. We believe that the proposed method can also be general to other tasks, which is kept for future research directions.
Ethics Statement
In this work, all human annotations used for training were taken from existing publicly available corpora, and no new human annotations were collected.
It is hoped that this work could play a part in promoting ethical AI practice. Firstly, it has been shown that the proposed HAS system can capture the inherent variability in human judgements and help mitigate biases and the issue of overrepresentation, thus producing a more inclusive representation of human opinions. The proposed HAS system also has the potential to minimize human annotators’ exposure to offensive and/or
hateful content in some evaluation tasks such as HateXplain. However, as with most research in machine learning, new modelling techniques could be used by bad actors to cause harm more effectively, but we do not see how the proposed HAS system is more concerning than any other method in this regard.

heading: Acknowledgements
text: W. W. is supported by a Cambridge International Scholarship from the Cambridge Trust. W. C. acknowledges funding via a Cambridge Trust Scholarship and a Cambridge University Engineering Department Studentship under grant G105682 NMZR/089 supported by Huawei R&D UK. This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service (www.hpc.cam.ac.uk) funded by EPSRC Tier2 capital grant EP/T022159/1.
The MSP-Podcast data was provided by The University of Texas at Dallas through the Multimodal Signal Processing Lab. This material is based upon work supported by the National Science Foundation under Grants No. IIS-1453781 and CNS-1823166. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or The University of Texas at Dallas.

heading: A The Sources of Variability in Human Evaluation
text: Human perception refers to the process by which individuals interpret and make sense of the sensory information they receive from the environment. It involves the integration of sensory data, cognitive processes, emotions, and previous experiences. Subjective perception emphasizes that each individual’s perception of the world is unique and influenced by their internal mental states, beliefs, attitudes, and past experiences. As a result, people can interpret and react to the same stimuli differently, leading to diverse and subjective perceptions.
Each person’s sensory organs, such as eyes and ears, may have slight variations in sensitivity and acuity, leading to different perceptions of the same stimuli. Cognitive biases, the inherent mental shortcuts or tendencies that influence how humans perceive and process information, can lead to difference in judgment and decision-making. People’s past experiences, cultural norms, and upbringing also shape their perceptions. Different cultural backgrounds can lead to distinct interpretations of the same event, leading to diverse reaction. The variability in humans can manifest in various tasks such as colour perception, emotion recognition, art appreciation, and feedback preferences.
Embracing and understanding the variability of human perception is vital for various research fields such as psychology, neuroscience, humancomputer interaction, and so on, and has practical implications in designing human-centered systems and promoting empathy and diversity. It helps create products and interfaces that cater to diverse user needs and preferences in fields like humancomputer interaction and user experience design. Being aware of the variability of perception is crucial in ethical decision-making. It help ensures that different perspectives and cultural sensitivities are considered, which helps identify and address potential biases that might disproportionately affect certain groups or lead to unfair outcomes.

heading: B Derivations
text: Detailed derivations for the training objectives on a single event di = {xi,Di} where Di = {η(1)i , · · · ,η (M) i } are presented in this section. For the simplicity of notations, the subscription i in our derivations will be omitted without ambiguity where possible. The meta-learning objectives presented in the paper are obtained by averaging such
single-task objectives across tasks.
B.1 Objective Function for the Base CNF
Denote the empirical human annotation distribution as pm(y|x) = δ ( y − η(m) ) , m = 1, · · · ,M and model output distribution as pθ(y|x). The average KL divergence between them over all M human annotations for this input x is given by:
L(θ;d) = 1 M
M∑
m=1
KL (pm(y|x) ∥ pθ(y|x))
= 1
M
M∑
m=1
∫ pm(y|x) log
pm(y|x) pθ(y|x) dy
= − 1 M
M∑
m=1
∫ pm(y|x) log pθ(y|x)dy + const
= − 1 M
M∑
m=1
log pθ(η (m)|x) + const
Minimizing this KL objective is equivalent to maximizing the average log likelihood log pθ(η
(m)|x) over all human annotations as presented in the paper.
B.2 Objective Function for S-CNF
For categorical annotations, each label η(m) represents the probabilities of all categories in the categorical human annotation distribution: η(m) = [η
(m) 1 , · · · ,η (m) K ], where η (m) k = Pm(c = k|x). Denote the model output distribution as Pθ(c|x). The average KL divergence between them over all M human annotations for this input x is given by:
Lexact(θ;d) = 1 M
M∑
m=1
KL (Pm(c|x) ∥ Pθ(c|x))
= 1
M
M∑
m=1
K∑
k=1
Pm(c = k|x) log Pm(c = k|x) Pθ(c = k|x)
= − 1 M
M∑
m=1
K∑
k=1
Pm(c = k|x) log Pθ(c = k|x)
+ const
= − 1 M
M∑
m=1
K∑
k=1
η (m) k log Pθ(c = k|x) + const,
where the marginal likelihood is lower bounded using variational inference:
log Pθ(c = k|x) = log ∫ P(c = k|v)pθ(v|x)dv
= log ∫ qΩ(v|η)
P(c = k|v)pθ(v|x) qΩ(v|η) dv
≥ ∫ qΩ(v|η) log P(c = k|v)pθ(v|x)
qΩ(v|η) dv
= EqΩ(v|η) [log P(c = k|v) + log pθ(v|x) − log qΩ(v|η)] .
Therefore, the final negative ELBO objective is obtained by
Lexact = − 1 M
M∑
m=1
K∑
k=1
η (m) k log Pθ(c = k|x)
≤ − 1 M
M∑
m=1
K∑
k=1
η (m) k EqΩ(v|η(m)) [log P(c = k|v)
+ log pθ(v|x)− log qΩ(v|η(m)) ]
= − 1 M
M∑
m=1
EqΩ(v|η(m))
[ K∑
k=1
η (m) k log P(c = k|v))
+ log pθ(v|x)− log qΩ(v|η(m)) ]
= L(θ,Ω;d),
where
log P(c = k|v) = logsoftmax(v)k,
log pθ(v|x) = pΛ ( f−1θ (v)|x ) ∣∣∣∣∣det ( ∂f−1θ (v) ∂v )∣∣∣∣∣ ,
log qΩ(v|η(m)) = N (v|µΩ(η(m)), diag(σ2Ω(η(m)))).
B.3 The Negative Log Likelihood (NLLalli ) for Categorical Annotations
The marginal likelihood of S-CNF is intractable, which can be approximated using Monte-Carlo simulation: Pθ(c = k|x) = ∫ P(c = k|v)pθ(v|x)dv
= Epθ(v|x) [P(c = k|v)]
≈ 1 Q
Q∑
j=1
P(c = k|vj), {vj}Qj=1 ∼iid pθ(v|x)
= 1
Q
Q∑
j=1
softmax(vj)k, {vj}Qj=1 ∼iid pθ(v|x)
= ȳk,
where ȳ = 1Q ∑Q j=1 softmax(vj) = 1 Q ∑Q j=1 yj which is the average of the simulated categorical distributions. Let η̄ = 1M ∑M m=1 η
(m) be the average label.
Then, the NLLalli for a single input xi is given by
NLLalli = − 1
M
M∑
m=1
K∑
k=1
η (m) i,k log Pθ(c = k|xi)
≈ − 1 M
M∑
m=1
K∑
k=1
η (m) i,k log ȳi,k
= − K∑
k=1
η̄i,k log ȳi,k,
which is the cross entropy between the averaged label and averaged sample.

heading: C Emotion Label Processing for MSP-Podcast
text: In MSP-Podcast, each annotator can choose from ten emotion classes to label the primary emotion of an utterance: Angry, Sad, Happy, Surprise, Fear, Disgust, Contempt, Neutral, Other. Although only one option is allowed, they can say other and define their own emotion class which can be more than one. During label processing, the original other class is split into sub-classes depending on the manual defined label and merged with the predefined labels. The grouping details are shown as follows: (i) Angry includes angry, disgust, contempt, annoyed; (ii) Sad includes sad, frustrated, disappointed, depressed, concerned; (iii) Happy includes happy, excited, amused; (iv) Neutral includes neutral; (v) Other includes all other emotion subclasses not listed above.

heading: D Model Structure Details
text: The procedure of sampling from and optimizing S-CNF are summarized in Algorithm 1 and 2.
A neural-network-based encoder gΛ is built to model µΛ(x),σ2Λ(x) given input x where Λ is the model parameters. gΛ follows an upstreamdownstream paradigm. The upstream model is pretrained on large amount of unlabelled data to learn universal representations. The downstream model uses the learned representation from the upstream model for specific applications.
For tasks involving speech as input (i.e., emotion class labelling), WavLM (Chen et al., 2022) is used
Algorithm 1 Sampling from S-CNF Input: x Output: Categorical probability y Compute µΛ(x),σ2Λ(x) = gΛ(x) Sample z ∼ N (µΛ(x), diag(σ2Λ(x)) Compute v = fθ (z) Compute y = softmax(v)
Algorithm 2 Optimizing S-CNF
Input: x,D = {η(1), · · · ,η(M)} Output: ELBO LELBO on dataset D for m = 1, · · · ,M do
Compute µΩ(η(m)),σ2Ω(η (m)) = hΩ(η (m)) for j = 1, · · · , Q do
Sample vj ∼ qΩ(v|η(m)) Compute L(m)j =
−∑Kk=1 η (m) k log P(c = k|vj)+log pθ(vj |x)− log qΩ(vj |η(m)) end for Compute LELBOm = 1Q ∑Q j=1 L (m) j end for Compute LELBO = 1M ∑M m=1 LELBOm
as the upstream model. WavLM is a speech foundation model pre-trained by self-supervised learning that takes raw waveform as input. The waveform is encoded by a CNN encoder followed by multiple Transformer encoders. The BASE+ version5 of the model is used in this work which has 12 Transformer encoder blocks with 768-dimensional hidden states and 8 attention heads. The parameters of the pretrained WavLM are frozen and the weighted sum of the outputs of the 12 Transformer encoder blocks is used as the speech embeddings feeding into the downstream model.
RoBERTa (Liu et al., 2019) is used as upstream model to encode text input for toxic speech detection, which is a robustly optimized model of BERT (Devlin et al., 2019). RoBERTa is a Transformer-based language model pretrained on a large corpus of English data with the masked language modelling objective. The BASE version6 was used in the work which has 12 Transformer layers, 768 hidden units, 12 attention heads, and 125 million parameters.
The downstream model consists of two Transformer encoder blocks followed by two FC layers.
5https://huggingface.co/microsoft/ wavlm-base-plus
6https://huggingface.co/roberta-base
The Transformer encoder layers has a hidden dimension of 128 and four attention head. The output layer contains two heads to predict the mean and standard deviation of the latent distribution pΛ(z|x). The invertible flow model fθ uses three real NVP blocks (Dinh et al., 2017) of dimension 64. The variational encoder hΩ for S-CNF contains a FC layer of dimension 64 and two output heads for the mean and standard deviation of the variational distribution qΩ(v|y).

heading: E Detailed Configuration of All Compared Methods
text: Ensemble consists of 10 systems initialized and trained using different random seeds. MCDP uses dropout rate of 0.4. A standard Gaussian prior is used for BBB. A modified version of EDL is used (Wu et al., 2023) which is trained by maximizing the per-observation-based marginal likelihood with a modified regularization term. Ensemble, MCDP, BBB, EDL use the same model structure as gΛ apart from removing the output head for predicting variance of latent distribution. A modified version of DPN (Wu et al., 2022b) is used which is trained by interpolating per-observation-based marginal likelihood with KL divergence. The coefficient of KL term is set to 5.0 for emotion class labelling and 2.0 for toxic speech detection. CVAE has the same gΛ structure as S-CNF for modelling p(z|x), and two 64-d FC layers are used for encoder and decoder. A-CNF has identical model structure as S-CNF.
F Implementation Details
The system was implemented using PyTorch with the SpeechBrain (Ravanelli et al., 2021) and normflows (Stimper et al., 2023) toolkit. The Adadelta optimizer was used with an initial learning rate of 1.2 for emotion class labelling and 0.05 for speech quality assessment. The NewBob learning rate scheduler was used with annealing factor 0.8 and patience 1. The system was trained for 30 epochs and the model with the best validation performance was used for testing. The number of ELBO samples was set to 20. All experiments were run with three different seeds and the average and standard error are reported.
Emotion class labelling RMSEs MAEs E(s̄)
MCDP 0.305 0.233 0.206 Ensemble 0.277 0.222 0.166
BBB 0.284 0.226 0.178 CVAE 0.333 0.244 0.244 DPN 0.297 0.236 0.191
A-CNF 0.223 0.209 0.046 S-CNF 0.218 0.198 0.015
Toxic speech detection RMSEs MAEs E(s̄)
MCDP 0.297 0.242 0.122 Ensemble 0.290 0.220 0.105
BBB 0.279 0.229 0.115 CVAE 0.345 0.208 0.208 DPN 0.299 0.220 0.178
A-CNF 0.274 0.232 0.062 S-CNF 0.263 0.206 0.002
Table 4: Analysis of standard deviation of simulated samples.

heading: G Analysis of Standard Deviation of Simulated Samples
text: It has been observed in Sec. 5.1 that flow models tend to have a larger difference between RMSEs and E(s̄). This section provides detailed analysis to this observation. Let N be the number of test utterances. Three std-related metrics are computed: (i) RMSE between std of predictions and human la-
bels: RMSEs = √
1 N ∑N i=1 (si − σi)2; (ii) Mean
absolute error between std of predictions and std of human labels: MAEs = 1N ∑N i=1 |si−σi|; (iii) Absolute error between average std of predictions and average std of human labels E(s̄) = |s̄i − σ̄i|. Results are shown in Table 4. The flow model tends to have larger discrepancy between MAEs and E(s̄). According to the triangular inequality:
E(s̄) = ∣∣∣∣∣ 1 N N∑
i=1
si − 1
N
N∑
i=1
σi ∣∣∣∣∣
= ∣∣∣∣∣ 1 N N∑
i=1
(si − σi) ∣∣∣∣∣ ≤ 1 N N∑
i=1
|si − σi| = MAEs
which show that E(s̄) is a lower bound of MAEs. The equality condition is satisfied when all samples are uniformly either greater than or less than the compared value. Therefore, a larger discrepancy between these two values indicates that the
0.04 0.02 0.00 0.02 0.04
0.04
0.02
0.00
0.02
0.04
Label MCDP Ensemble BBB CVAE DPN A-CNFS-CNF
0 20 80 10040 60 0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Utterance idx
St d
Figure 5: Standard deviation of simulated samples for emotion class labelling.
standard deviation of some samples exceeds that of the labels, while for others, it is lower. A smaller discrepancy indicates that the standard deviation of samples tend to be consistently larger of smaller than that of the labels. In Figure 5, 100 test utterances are randomly selected and the std of samples generated by different models are plotted, which supports the above conclusion. The proposed SCNF has the best performance for matching the diversity of human annotations.

heading: H Adjusting the Variability of CNFs by Prior Tempering
text: Sec. 5.3 has explored the effect of prior tempering on the performance. More details are provided in this section. Table 5 shows the effect of prior tempering on the performance of S-CNF. Table 6 shows the effect of dropout rate on the performance of MCDP. When the temperature increases or dropout rate increases, the simulated annotations become more diverse, shown by the increase in the average standard deviation (s̄) and the decrease in Fleiss’s kappa (κ̂) of simulated annotations. Comparing two tables, it can be seen that prior tempering in CNF is more efficient and covers a wider range of dynamics than adjusting the dropout rate in MCDP.

heading: I Further visualised examples: Emotion Class Labelling
text: This section shows additional visualized examples for emotion class labelling when human annotators reach a consensus (Figure 6 (a)(b)), diverge (Figure 6 (c)(d)), and give distinct labels (Figure 6 (e)). Aligned with the findings in Sec. 5.4, the proposed S-CNF can better simulate the aggregated behaviour as well as the variability of human annotations in all cases.

heading: J Further visualised examples: Toxic Speech Detection
text: This section shows visualized examples for toxic speech detection when all three human annotators provide the same label (Figure 7 (a)(b)), one of them gives a different label (Figure 7 (c)(d)), and all three annoators give distinct labels (Figure 7 (e)). Similar to the observations of the emotion class labelling task, the proposed S-CNF can better simulate the aggregated behaviour as well as the variability of human annotations in all cases.

heading: Future Work
text: We believe that the proposed method can also be general to other tasks, which is kept for future research directions