Establishing Reliability in Crowdsourced Voice Feedback (CVF)
and Evaluating the Reliability in the Absence of Ground-Truth
Aashish Jain∗

Sudeeksha Murari†aajain@amazon.com
Amazon Alexa
Boston, MA, USA

sud@amazon.com
Amazon Alexa
Boston, MA, USA

ABSTRACTIntelligent Voice Assistant (IVA) systems, such as Alexa, Google
Assistant and Siri, allow us to interact with them using just the voicecommands. IVAs can elicit voice feedback directly from the users
and use their responses to improve the various components of IVAs.
One concern with using such crowdsourced voice feedback (CVF)
data is the reliability of feedback itself such as background noise or
disingenuous feedback. In this paper, we propose ways to establish
confidence scores to indicate the reliability of the CVF data. We
build a probabilistic Bayesian Belief Network (BBN) model, whichuses the CVF data as training dataset. Since human annotation of
the CVF data can be expensive, we explore ways to evaluate such amodel without human labeled data. We propose several metrics that(i) do not need any ground-truth, (ii) can be simply computed using
the CVF data, and (iii) can reliably measure the model performanceto output confidence scores indicating reliability.

CCS CONCEPTS• Mathematics of computing → Bayesian networks; • Computing methodologies → Model verification and validation;
Uncertainty quantification; Bayesian network models.

KEYWORDS
intelligent voice assistant, crowdsourced voice feedback, Bayesian
belief network, ground-truth data, feedback reliability
ACM Reference Format:
Aashish Jain and Sudeeksha Murari. 2021. Establishing Reliability in Crowdsourced Voice Feedback (CVF) and Evaluating the Reliability in the Absence of Ground-Truth. In Proceedings of Data-Efficient Machine Learning Workshop at KDD (DeMaL’21). ACM, New York, NY, USA, 9 pages.
https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

INTRODUCTION

In order to collect feedback on specific aspects of an intelligent
Voice Assistant (IVA), a system like Crowdsourced Voice Feedback∗ Corresponding author† Work done while at Amazon Alexa.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
DeMaL’21, August 2021, Virtual© 2021 Association for Computing Machinery.
ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn(CVF) can be valuable. CVF is a system that can target specificinteractions based on domain/intent, device type or other aspectsof a request made to an IVA. Typically, when a user makes a request
to an IVA, the IVA responds with an action or with a response.
The user is then asked a question by CVF about the interaction
to gauge if the IVA rendered a satisfactory response/action. TheseCVF questions can be generic or very specific to the interaction,and users’ responses are recorded and used to improve the IVA. An
example of such an interaction, along with the CVF question canbe seen in Fig. 1. Other than binary questions, there are numericuser’s original utterance
What time is it?

IVA’s response
It is 4 PM.

Was that helpful?
IVA’s feedback
Yes, thank you!
user’s feedback utterance question
Figure 1: An example of a user-IVA interaction
feedback questions as well, such as “how did I do on a scale of 1-5?".
The users’ typical responses to CVF can be broadly categorized as
yes, no, maybe, silence or some numeric response. Annoyed users
also say things like “shut up" or “go away" or completely ignore
the CVF question and ask for something else. In this work, weonly focus on the binary questions and responses are assigned a
feedback type of yes, no and other. In the example shown in Fig. 1,
the feedback type of the response is yes. The other feedback typecontains several scenarios such as when a user (i) provides uncertain
feedback (“maybe" or “I don’t know") (ii) provides the feedback in anoisy background such that the feedback cannot be understood, and(iii) ignores the feedback question. In the Fig. 1, we also illustrate
what we refer to as original and feedback utterances, as shown on
the first and fourth lines, respectively.
In this paper, we propose to (i) use Bayesian Belief Network(BBN) [20, 26] to establish the reliability of crowdsourced voice
feedback (CVF) data in a frugal manner without human annotatedtraining data, and (ii) evaluate CVF confidence detection model in
the absence of ground-truth data. The BBN model can also be used
to add interpretability to the model predictions by providing estimates of marginal contributions of Automated Speech Recognition(ASR), Natural Language Understanding (NLU) etc., towards theDeMaL’21, August 2021, Virtual

three feedback types for a given interaction. However, interpretation of the predictions is not the focus of this paper. This paper
focuses on the former, i.e. establishing reliability in CVF data andevaluating model in the absence of the ground-truth data. In an IVA
system, millions of CVF data-points are collected each month. Thisdata is then used as training/testing data in models or to generate
health metrics around IVA performance. Often, there are concerns
on reliability of CVF data due to sarcastic responses or heavy background noise etc. To address this, a BBN model is used to predict
feedback type and confidence scores on user’s feedback. With acontinuous confidence score for each feedback, noisy data can befiltered-out or reliable data can be filtered-in.

2

RELATED WORK AND OUR CONTRIBUTION

Any type of crowdsourced data comes with some level of noise
and there have been efforts in the past to clean the labels in crowdsourced data [6, 10], with the purpose to use them for training
supervised models. Another research area in dealing with noisylabelled data, not just limited to crowdsourced data, is to estimate
the uncertainty in the labels so that more uncertain data points canbe kept separate from more confident data points, and there is a dedicated framework for that as well [24]. There is research on learningwith noisy data itself, for example using deep neural networks [7].
In all these areas, the focus has been mainly on training the model,
i.e. getting reliable training data with minimized noise and building
models that have minimal influence due to noise in the training
data. However, it is difficult to curate a reliable ground-truth data
set for model evaluations using these approaches. In the absence ofground-truth data, it becomes challenging to compare models andselect the best one for deployment. There is an idea called reverse
testing [5], which does not rely on the ground-truth data. Anotheridea is in the context of interpretable machine learning, where thefocus is on explanation, rather than prediction, and novel evaluation
methods were proposed in the absence of the ground-truth data[36]. A more sustainable solution to evaluate models in the absenceof ground-truth data is to have a set of reliable success metrics,
and there exists work in which some metrics were proposed thatcan be used in the absence of the ground-truth data, though only
for binary classification problems [11]. In these research attempts,
reliable labelled data sets were already present to train the modelin the first place. We do not find any research on dealing withboth issues (i.e. noisy crowdsourced data for training models andabsence of ground-truth for model evaluation) for crowdsourced
voice feedback domain.
We have the challenges of (i) not having reliable labelled datato train the model to make predictions on both feedback type andconfidence scores on them, and (ii) not having the ground-truth data
to evaluate that model. Having dependence on the ground-truth
data is not desirable due to the cost of human annotations and risk
to user privacy. In our work, we introduce new metrics and also use
existing metrics, that can be computed using only the CVF data, anddo not require any ground-truth data. We propose to use these CVF
data based metrics to replace ground-truth data based metrics, usedin traditional supervised learning, like 𝐹 1 or ROC AUC. In order toshow that these CVF data based metrics are reliable and can indeed
be good proxies for ground-truth based metrics (as if we had a

Aashish Jain and Sudeeksha Murari

ground-truth data set available), we curate a small human annotatedground-truth data set to compare these metrics with ground-truth
data based metrics. Moreover, we also define some metrics, thatuse both CVF data and ground-truth data, that can be used toevaluate the predicted confidence scores in the voice feedback, as
opposed to only the feedback type prediction. This allows us toestablish confidence on model predicted confidence scores. To thebest of our knowledge, this is the first work to (i) build a BBN basedmodel using CVF data to predict feedback type and associatedconfidence scores and (ii) present reliable metrics to measure theperformance of crowdsourced voice feedback confidence detection
model in the absence of ground-truth data. Last, we also present
hyper-parameters, used in building the BBN model, that can betuned to achieve optimal model performance for applications suchas this.

3

DATA

Usually, the IVAs use spoken language understanding (SLU) basedmodels [4, 18, 29] to detect domain, intent and slots, which are inturn used to automatically tag user’s feedback types as yes, no orother. Throughout this paper, we will refer to this as user feedback.
The confidence on the user feedback may depend on many factors
such as (i) feedback type determination by the SLU models (ii) users’
sentiment scores [19], (iii) SLU model hypothesis confidence scores(iv) ASR model hypothesis confidence scores [16, 17, 22] and manyother run-time meta signals that are collected during the user-IVA
interaction. We use these signals to build a BBN based model inthis work. Note that we do not use any lexical based signals in thiswork. For most of the signals, we use them corresponding to boththe original and the feedback utterance as illustrated in the example
shown in Fig. 1. All these signals in a single record, consisting ofboth the original and the feedback utterances, forms one sample of
the CVF dataset.
We collected one month of CVF data from IVA users, whereusers are not identifiable. From that data, we subsampled 8.5 × 106
samples (Dataset A). Each data sample consists of two utterances,including the original and feedback utterance, as we illustrated inFig. 1 above. We split the CVF dataset into training and test sets,leaving 7.5×106 samples as the CVF training set and 1×106 samples
as the CVF test set.
To demonstrate that the newly designed metrics computed on
the CVF test set represent the true model performance, we study
correlations between CVF test set metrics and ground-truth data
based metrics. For this aspect of validating the metrics we propose,
we randomly select 3.1 × 103 CVF data samples from a different
time window than Dataset A. We annotate this smaller dataset togenerate the feedback type labels on each CVF data sample, andrefer to this as ground-truth test set.

4

METHODOLOGYA CVF is useful in understanding whether a rendered experiencethrough an IVA was satisfactory or not. For this purpose, the feedback type represents the success or failure of an experience with
the IVA. We believe that a Bayesian Belief Network (BBN) is well
suited for not only producing confidence scores indicating the reliability of user feedback, but it also helps understand the factors

Establishing Reliability in Crowdsourced Voice Feedback (CVF) and Evaluating the Reliability in the Absence of Ground-Truth

that contribute to a user responding yes or no to a CVF question
like “did I play the correct song". This paper focuses on building a
BBN based model to establish reliability in the user feedback, andmore importantly on introducing metrics to measure the model
performance in the absence of ground-truth data. The BBN model
predicts the potential user response by generating outputs of probabilities for each feedback type (yes, no, other). We then compare
the outputs with the actual responses provided by users to assess
confidence scores. For instance, if the model predicts yes with aprobability of 0.8 and the user responds “yes", our confidence in the
feedback response is 0.8. If our model predicts yes with a probabilityof 0.7 and no with a probability of 0.3, and the user response is “no",the confidence score is 0.3.

4.1

Bayesian Belief Network Model

A Bayesian belief network (BBN) is a directed acyclic graph (DAG)
based framework that is used to model the joint probability distribution P of a set of variables in a data set [20, 26]. The DAG is definedas 𝐺 = (𝑉 , 𝐸), with 𝑉 representing a set of nodes or variables and𝐸 ⊆ 𝑉 × 𝑉 representing edges in the network. The joint probability
distribution of all variables |𝑉 | is the set of parameters Θ in the
BBN, and so the BBN is represented as the pair (𝐺, Θ). This network
effectively captures the independence relationships between thevariables X = {𝑋 1 ...𝑋 𝑁 }, with graphical separation in 𝐺, which
also corresponds to the conditional independency in probability,
leading to the following factorization [20, 21]ÖP(X|𝐺, Θ) =𝑃 (𝑋 𝑣 |Π𝑋 𝑣 , Θ𝑋 𝑣 ),
(1)
𝑣 ∈𝑉

where, Π𝑋 𝑣 is the parent node of the variable 𝑋 𝑣 . Here, the joint
probability distribution of X (with parameters Θ) can be thoughtof as a multiplication of the local distribution of each variable 𝑋 𝑣(with parameters Θ𝑋 𝑣 ) conditional on each variable’s parents Π𝑋 𝑣 .
Note that for a node, there can be a single or multiple or even no
parents. The factorization (Eq. 1) is valid only when we do not haveany missing data.
Depending on whether the structure of the BBN is known or
unknown, and if the data is full or partial (i.e. with missing values),
there are four possible learning approaches [3]. In this work, welearn the structure from the data, hence we use the unknown structure scenario. We also treat our data prior to training (see Sec. 4.2)
so that there is no missing values, and so our data is full. The BBN
model is learned using the data in two broad steps. In the first step,
the DAG structure 𝐺 of the BBN is learned using the data, which
captures the dependence structure in the data.
Once, the structure learning is finished and we find 𝐺, the second
step is to learn the parameters Θ, to finish learning the pair (𝐺, Θ),i.e. the BBN . Usually, the parameters are learned either using
maximum likelihood or Bayes method [31].

4.2

Training

We first pre-process the CVF training data for learning the structure
and the parameters of the BBN. The BBN can be learned using datathat contains only discrete variables or only continuous variables
or mix of both. In this work, we use bnlearn [31, 32], a popular R
package for learning BBNs. This package supports continuous anddiscrete BBNs but not the hybrid. Since feedback type is discrete,

DeMaL’21, August 2021, Virtual

we transform all continuous variables into discrete variables, totrain a discrete BBN. The number of levels, 𝑛𝑙 , that each continuous
variable is discretized into discrete variable, is a hyper-parameter.
There are some missing values in some of the variables, and we
replace them with a unique level (e.g. “missing"), to make the data
full. We believe that a missing field can affect our confidence inthe user feedback and therefore treat missing values as a discrete
category in our training data. From a reliability perspective, missing
data are useful information in establishing confidence.
The pre-processed data is used to first learn the structure of the
BBN. We use various constraint-based [2, 9, 14, 23, 27, 33, 34, 37],
score-based [28] (using Bayesian Information Criterion (BIC) [30] as
scoring metric), hybrid [12, 14, 35] and local discovery [8] learningalgorithms to learn the structure. For each model, we generate five
network structures by applying nonparametric bootstrap to thetraining data, and then compute the joint strength of all possibleedges. This bootstrapping procedure allows us to generate moregeneric network structure. Finally, for the parameter learning, we
use the Bayes method.

4.3

Inferencing

We make all the probabilistic inferences using gRain package [15],which allows to compute conditional probability of any variableconditioned on any other set of variables, for a given sample. Unlikebnlearn, which supports approximate inferencing, gRain package
supports exact inferencing. In our use-case, we want to compute theprobability of the feedback type variable 𝑋 𝐹 , for all three feedback
types 𝑋 𝐹 𝑡 with 𝑡 = {𝑦𝑒𝑠, 𝑛𝑜, 𝑜𝑡ℎ𝑒𝑟 }, conditioned on the values of allother variables. We define that conditional probability as 𝑃 (𝑋 𝐹 𝑡 |X ′ ),where, X ′ is 𝑋 1, ...𝑋 𝑗 , ...𝑋 𝑁 with 𝑗 ≠ 𝐹 . The conditional probability
is used as confidence score 𝐶 𝑡 on feedback type 𝑡, in other words,
𝐶 𝑡 = 𝑃 (𝑋 𝐹 𝑡 |X ′ ); 𝑡 ∈ {𝑦𝑒𝑠, 𝑛𝑜, 𝑜𝑡ℎ𝑒𝑟 },
with

Í

𝑡𝐶

(2)

𝑡 = 1. We can also use argmax 𝐶 𝑡 to predict the feedback𝑡

type 𝑡 ′ , like a classification problem. Note that, to distinguish theuser feedback type from model predicted feedback type, we denotethem by 𝑡 and 𝑡 ′ , respectively. When the predicted feedback type𝑡 ′ (which is the one with the highest 𝐶 𝑡 ) is the same as the user
feedback response 𝑡, 𝐶 𝑡 is retained as our confidence score on the
feedback received. If 𝐶 𝑡 is a low number, it does not mean thatthe user was incorrect, it means that based on the signals we use,
it is difficult to explain the feedback received, and therefore weput lower confidence in that data point. These data points are notreadily usable and need additional inspection to understand thefeedback provided. Having model’s prediction of the feedback type,
and the confidence score on the user’s feedback type defined, wenext discuss the model evaluation methods in the next section.

5

MODEL EVALUATION METHODS

For model evaluation, we propose a combination of new and existing metrics that do not need ground-truth data, but only the
CVF data, and we call these CVF based metrics (MC ). For validationof the MC metrics, we also compute traditional evaluation metrics (MG ) on a small human labeled ground-truth data, which wascurated only to show correlations between MC and MG metrics.

DeMaL’21, August 2021, Virtual

5.1

CVF Based Metrics (MC )We define 11 CVF based metrics and denote them using 𝑀𝑖𝐶 , where𝑖 = 1...11. These metrics are based on (i) BBN structure, (ii) agreement between user’s feedback type 𝑡 and model predicted feedback
type 𝑡 ′ , and (iii) user’s sentiment scores. We summarize all the MC
metrics in Table 1, and explain them in detail below.
5.1.1 BBN Accuracy. There are information theory based metrics
such as Akaike Information Criterion (AIC) [1] and likelihood based
metrics such as BIC [30], that can be used to score how well theBBN fits the data. These metrics are based on log likelihood of a
BBN given a dataset, and so the accuracy of the BBN depends on
the size of the data itself [13]. As a result, these metrics do notprovide the “absolute" accuracy but rather the “relative" accuracyof a BBN for a given data set. An alternative approach to measure
the accuracy of a BBN was proposed [25], where the accuracy
was approximated as conditional independencies present in the
structure of the BBN. This alternate accuracy is called the Network
Conditional Independencies Mutual Information (NCIMI). The idea
behind this metric is to look at all relationships in the networkthat are conditionally independent, and then compute the mutual
information (MI) dependency measure between the variables using
the data. The non-zero values of the MI tell us about the inaccuracies
in the model. The NCIMI is simply the summation of all possible
MIs in the network (computed only on conditionally independent
variable pairs). To perform model comparisons, we compute mean
statistics of NCIMI. Based on this NCIMI mean statistics, we define
two metrics: (i) 𝑀1𝐶 = mean of NCIMI on the CVF test set, (ii) 𝑀2𝐶= median of NCIMI on the CVF test set. The smaller the values ofthese metrics, the more accurate the BBN is.
5.1.2 Classification Accuracy Assuming User Feedback As GroundTruth. If we assume that the user’s feedback is totally reliable andthere is no noise (which is not the case), we can compute the agreement between the user feedback type 𝑡 and the model predicted
feedback type 𝑡 ′ . We compute accuracy to represent this agreementand denote this metric as 𝑀3𝐶 . This metric represents the hypothesisthat the users that provide feedback are correct most of the time,
which was validated using a small sample annotation. This is not
used as a standalone metric to evaluate the model.
5.1.3 Extreme Sentiment Based Metrics. The speech sentimentscores are computed on both the original and feedback utterance
using the sentiment score model [19], which generates three typesof sentiment scores: (i) activation, which captures if the user is
excited or calm, (ii) valence, which represents positive or negative
sentiment and (iii) satisfaction, which also measures the positiveor negative experience. We find that activation is quite uniformly
distributed across the feedback classes, however, satisfaction andvalence distributions are sensitive to feedback type. It was also
found in another study that the activation scores are not correlatedwith customer satisfaction [19]. The idea behind extreme sentimentbased metrics is to assess what the model predicts when the sentiment scores are extreme. We posit that for extreme positive andnegative sentiment scores, user’s feedback type 𝑡 mostly should be
yes and no, respectively. Before defining extreme sentiment based
metrics, we first discuss how we subset the data to get the extreme
sentiment feedback test sets, both extreme negative sentiment (ENS)

Aashish Jain and Sudeeksha Murari

test set and extreme positive sentiment (EPS) test set. We consider
EPS test set feedback as those ones for which both the satisfaction
and the valence scores are greater than 𝜇𝑠 + 𝑘𝜎𝑠 and 𝜇 𝑣 + 𝑘𝜎𝑣 , respectively. Here, 𝜇𝑠 and 𝜇 𝑣 are the means and 𝜎𝑠 and 𝜎𝑣 are thestandard deviations of satisfaction and valence, respectively. Similarly, for ENS test set, we consider satisfaction and valence scores
to be smaller than 𝜇𝑠 − 𝑘𝜎𝑠 and 𝜇 𝑣 − 𝑘𝜎𝑣 , respectively. We use 𝑘 = 3
in this work as that provides sufficiently large ENS and EPS test
set, while maintaining the sentiment extremeness, as found in aninternal study. Having setup the two test sets for EPS and ENS
feedback, we next define extreme sentiment based metrics.
Table 1: Summary of MC metrics. The hypothesis that a metric should be higher or lower for a better model performanceis indicated by ↑ and ↓, respectively, in the description.
MC

Description

𝑀1𝐶
𝑀2𝐶
𝑀3𝐶
𝑀4𝐶
𝑀5𝐶
𝑀6𝐶
𝑀7𝐶
𝑀8𝐶
𝑀9𝐶
𝐶
𝑀10
𝐶
𝑀11

NCIMI mean on CVF test set ↓NCIMI median on CVF test set ↓Accuracy between 𝑡 and 𝑡 ′ ↑Mean of the confidence scores on yes class in ENS test set ↓Recall of no class in ENS test set ↑Rate of false negatives due to yes class in ENS test set ↓Relative reduction in yes samples in ENS test set ↑Mean of the confidence scores on no class in EPS test set ↓Recall of yes class in EPS test set ↑Rate of false negatives due to no class in EPS test set ↓Relative reduction in no samples in EPS test set ↑Extreme Negative Sentiment Based Metrics: In the extreme
negative sentiment (ENS) test set, we posit that the model would
generate lower confidence scores on the yes class and hence would
have lower 𝐶 𝑡 for yes. To capture this, we define 𝑀4𝐶 = 𝐶¯ 𝑦𝑒𝑠 (ENS)
as a metric to represent the mean of the confidence scores on theyes class in the ENS test set, and we posit that it’s value should
be smaller for a better model. If we focus on the no class in the
ENS test set, then we can make some arguments that if the user
feedback type 𝑡 is no, then the model should also predict no. In otherwords, if we consider 𝑡 as reference and 𝑡 ′ as model hypothesis, thefalse negatives should be much smaller for no. It is possible thatsome of the no from users would be predicted as other feedback
types by the model but it is important that the model predicts
mainly no and other feedback types but not yes. It is fine if there
are some false positives for no, as model would be stealing fromother feedback types to no. Based on these arguments, the recallof no class could be an important metric, and we denote that as
𝑀5𝐶 = recall𝑛𝑜 (ENS) which should become higher as the modelgets better. Along with 𝑀5𝐶 , we can also keep track of how manytimes the model predicted yes when the user feedback was no. We
FN𝑦𝑒𝑠 capture this information using 𝑀6𝐶 = 𝑛,𝑛𝑜 ENS , which is the𝑛𝑠ratio of the number of false negatives due to yes, FN𝑦𝑒𝑠 , to the total
number of no responded by users in ENS test set 𝑛𝑛,𝑛𝑜, and we𝑠
hypothesize that this metric’s value should be lower. The ENS test
set should have as small number of yes feedback as possible, so even
if the user’s feedback is yes, the model should predict no. We use𝑛,𝑦𝑒𝑠𝑛,𝑦𝑒𝑠 ′𝑛,𝑦𝑒𝑠
𝑀7𝐶 = 100 × (𝑛𝑠
− 𝑛𝑠
)/𝑛𝑠
(ENS) to denote the relative

Establishing Reliability in Crowdsourced Voice Feedback (CVF) and Evaluating the Reliability in the Absence of Ground-Truth

reduction in the number of yes feedback, when going from 𝑡 to 𝑡 ′ ,𝑛,𝑦𝑒𝑠𝑛,𝑦𝑒𝑠 ′
where 𝑛𝑠
and 𝑛𝑠are the number of yes feedback in ENS
test set as responded by users and predicted by model, respectively.
The value of this metric should be higher for better models.
Extreme Positive Sentiment Based Metrics: Similar to ENS
𝐶 for thetest set based four metrics, we define metrics 𝑀8𝐶 − 𝑀11
EPS test set, by simply switching the feedback polarity from yes
to no and vice versa. As a result, in the EPS test set, we have (i)𝑀8𝐶 = 𝐶¯𝑛𝑜 (EPS) as the mean of the confidence scores on the no
𝐶 =class, (ii) 𝑀9𝐶 = recall𝑦𝑒𝑠 (EPS) as the recall of yes class, (iii) 𝑀10

FN𝑛𝑜 
𝑝,𝑦𝑒𝑠 EPS as the ratio of the number of false negatives due to𝑛𝑠𝑝,𝑦𝑒𝑠no, FN𝑛𝑜 , to the total number of yes responded by users 𝑛𝑠
in
′
𝑝,𝑛𝑜
𝑝,𝑛𝑜
𝑝,𝑛𝑜
𝐶
EPS test set, and (iv) 𝑀11 = 100 × (𝑛𝑠− 𝑛𝑠)/𝑛𝑠 (EPS) as
the relative reduction in the number of no feedback, when going𝑝,𝑛𝑜𝑝,𝑛𝑜 ′
from 𝑡 to 𝑡 ′ , where 𝑛𝑠
and 𝑛𝑠are the number of no feedbackin the EPS test set as responded by users and predicted by model,
𝐶 should be lower,
respectively. The values of metrics 𝑀8𝐶 and 𝑀10
𝐶 should be higher, for betterand values of metrics 𝑀9𝐶 and 𝑀11models.

5.2

Ground-Truth Based Metrics (MG )Having human annotated ground-truth data with yes, no, other feedback type labels and the model predicted feedback type with confidence scores (Eq. 2), we can compute several metrics to measure
the model performance. However, we emphasize that MG metrics
are generated only for establishing the efficacy of MC metrics, as
stated in Sec. 5. The MG metrics are described in the Table 2. The
Table 2: List of ground-truth based metrics . The values of all
metrics should be higher for better model performance, as
indicated by ↑ in the description.
MG

Description

𝑀1𝐺
𝑀2𝐺
𝑀3𝐺
𝑀4𝐺
𝑀5𝐺
𝑀6𝐺

Accuracy ↑Mean of precision across three feedback types ↑Mean of recall across three feedback types ↑Mean of F1 scores across three feedback types ↑Mean of ROC AUC across three feedback types ↑Mean of differences in confidence scores between ground-truthlabel and 𝑡 ↑Mean of confidence scores on ground-truth label ↑For 𝑖 = 1..4, computed between the ground-truth label and user
feedback type 𝑡 (rather than model predicted feedback type 𝑡 ′ ),and hence is a constant number for each 𝑖

𝑀7𝐺
𝑢
𝑀𝑖𝐺

metrics 𝑀1𝐺 - 𝑀5𝐺 are well established, so we only elaborate 𝑀5𝐺 𝑀7𝐺 in more detail. We use the model to get confidence scores onboth user’s feedback type 𝑡 and ground-truth feedback type using
Eq. 2. For instance, if 𝑡 is yes, we use the 𝐶 𝑡 of yes as it’s confidence
score; if for the same sample, the ground-truth feedback type label
is no, the 𝐶 𝑡 of no produced by the model is it’s confidence score.
We expect that the confidence score generated by the model for the
ground-truth feedback type label, in general, should be greater thanthat generated by the same model for the user provided feedback

DeMaL’21, August 2021, Virtual

type 𝑡, as that way the model is closer to ground-truth than theuser feedback. Therefore, we can compute the difference between
the confidence scores on ground-truth label and 𝑡 for all samples in
the ground-truth test set, and the mean of those differences is what
we refer to as 𝑀6𝐺 . Higher value of this metric should correspondto better model performance. We also assume that the mean of theconfidence scores on the ground-truth feedback type label, acrossall samples, should be higher for better models, and this value isrepresented by 𝑀7𝐺 . It is also important to mention that the metrics
𝑀1𝐺 - 𝑀4𝐺 rely only on the feedback types, i.e. classification labels,
whereas 𝑀5𝐺 - 𝑀7𝐺 are computed using confidence scores.

6

RESULTS AND DISCUSSION

In the absence of the ground-truth data, we proposed several metrics under MC in Sec.5.1 that can be used to evaluate model performance. In order to show that MC metrics can be trusted, we
carry out correlation studies between MG and MC metrics. A goodcorrelation would mean that some of MG metrics can indeed beproxied by some of MC metrics. To obtain correlations between
MG and MC metrics, we build several models by varying three
hyper-parameters, to study the effect of them on the metrics in MG
and MC . These three hyper-parameters are: (i) the structure learning algorithms SLA, (ii) the number of levels 𝑛𝑙 that is used while
discretizing the continuous variables into categorical variables with𝑛𝑙 levels, and (iii) the number of samples in the training data 𝑛𝑠 . For
computing correlations, we look at the trends of different metrics
as function of all the models generated from varying those three
hyper-parameters.
Before computing correlations, we first look at the trends ofsome of the MG metrics (since we already curated a small groundtruth data set specifically for this work) for various models that𝑀𝐺
we built. Fig. 2 presents the results for 𝐺𝑖 𝑢 for all models. Here,𝑀𝑖𝑢𝑖 = 1..4 correspond to classification based metrics, and 𝑀𝑖𝐺 metricis computed between ground-truth feedback type and user feedback type 𝑡, and so it is independent of the model choice. The firstpanel in Fig. 2 shows the results by varying the structure learning algorithm (SLA) and the SLA Index 1..14 indicates the following algorithms: {chow.liu, fast.iamb, gs, h2pc, hc, iamb,
iamb.fdr, inter.iamb, mmhc, mmpc, pc.stable, rsmax2,
si.hiton.pc, tabu}, which are implemented in bnlearn [31, 32].
The variation with respect to the number of levels 𝑛𝑙 , used in discretizing the continuous variables to discrete variables, is shownin the second panel, where we vary 𝑛𝑙 from 1..19. The third panel
displays the variations in the number of samples 𝑛𝑠 in the training
data set, where we vary it from 1 × 103 − 7.5 × 106 . In all three variations, we observe that none of those four MG metrics outperformu
MG metrics. This means that user’s feedback 𝑡 is more closer tothe ground-truth compared with model predicted feedback type 𝑡 ′ ,𝑀𝐺although with minute differences as the values of 𝐺𝑖 𝑢 are close𝑀𝑖
to 1. For classification purpose, the model may not even be needed,
however, the purpose of this work is to establish the confidence
scores on the user’s feedback which is not possible to get directlyfrom users.

DeMaL’21, August 2021, Virtual

Aashish Jain and Sudeeksha Murari

1.0

1.0

0.9
●

● ●

●

● ●
●

●

●

0.8

●

1.0
●●

0.9

●

0.8

u

MG
i

●

●●●●●●

●●

0.9

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3
1

3

5 7 9 11 13
SLA Index●

7

10 13 16 19
nl

●

●●

●
●

0.4
4

●

●

●●
●

0.3
1

●●

● ●

0.5
●

●

●

●

●● ●

0.8

●●

●

MG
i

●●●●

103

104

105
ns

106

107

Figure 2: Results for normalized 𝑀𝑖𝐺 , where 𝑖 = 1..4, for all the models generated by varying three different hyper-parameters,
shown in three panels: (i) first panel: Structure learning algorithm (SLA) index with 14 SLAs (ii) second panel: number of levels
used while discretizing continuous variables to discrete variables, 𝑛𝑙 with values 1..19, and (iii) third panel: number of samples𝑢in the training data set, 𝑛𝑠 with values 1 × 103 − 7.5 × 106 . 𝑀𝑖𝐺 is computed between the ground-truth feedback type and the user
feedback type 𝑡, and is constant for each 𝑖.
Markers: □ : (𝑖 = 1), ^ : (𝑖 = 2), △ : (𝑖 = 3), ◦ : (𝑖 = 4)
For the SLA variation experiment, we fix 𝑛𝑠 = 2 × 106 and𝑛𝑙 = 5. From the SLA variation experiment, we find that gs, iamb,si.hiton.pc, which are constraint-based algorithms and rsmax2
which is a hybrid method, do not perform well for our problem, if
we consider 𝑀1𝐺 , 𝑀2𝐺 and 𝑀4𝐺 metrics. Depending on what metricwe consider, there are different best models. However, considering
the mean 𝐹 1 score, i.e. 𝑀4𝐺 , iamb.fdr, inter.iamb, pc.stable,
which are constraint-based algorithms, hc, tabu, which are scorebased algorithms and h2pc, which is a hybrid algorithm, perform
well for our problem. To make the selection of the best SLA, welook at the training time and we find the following training times in
AWS’s m5.24xlarge EC2 instance: iamb.fdr: 140s, inter.iamb:
71s, pc.stable: 570s, hc: 92s, tabu: 120s and h2pc: 1, 710s. As aresult, inter.iamb is the best choice for SLA. For the 𝑛𝑙 variation
experiment, we fix 𝑛𝑠 = 2 × 106 and SLA to be inter.iamb. We seethat the values of all four metrics increase initially, and then theylevel-off with minor fluctuations, with elbow point around 𝑛𝑙 = 5.
Based on this observation, we pick 𝑛𝑙 = 5 in other experiments as
higher 𝑛𝑙 increases the computational burden. For the 𝑛𝑠 variation
experiment, we fix SLA to be inter.iamb, and find that all four
metrics do not change much with 𝑛𝑠 , except for data points around𝑛𝑠 = 1 × 104 .
Having discussed the results based on MG metrics, we now
present results on MC metrics, and also look at the correlations.
Fig. 3 presents the patterns of these two normalized metrics asa function of the model index in the top three panels. Here, themodel index 1...14 refers to all the SLAs, in the same order discussed
above for Fig. 2, and 15...33 model index refers to 𝑛𝑙 variations with
𝑛𝑙 = 1...19, and the index 34...54 refers to 𝑛𝑠 variations with 𝑛𝑠 in
the range of 1 × 103 − 7.5 × 106 . A table, listing all three hyperparameters for all models, can be found in the Appendix. We cansee negative correlations between 𝑀8𝐶 and 𝑀6𝐺 for all three experiments. This is expected because as the model becomes better, themean of the differences between confidence scores of the groundtruth feedback type and the user feedback type should increase, but
the mean of the confidence scores of user’s no feedback in the ENS
test set should decrease. To quantify the correlation, we compute
Spearman’s rank correlation coefficient 𝑟 between the two metrics.
For the pair 𝑀8𝐶 − 𝑀6𝐺 , we find 𝑟 = −0.90 (p-value: 1 × 10−7 ), −0.67(p-value: 2.4 × 10−3 ), −0.80 (p-value: 1.6 × 10−5 ) for SLA, 𝑛𝑙 and𝑛𝑠 variation experiments, respectively. We can also combine all
the models from three different variation experiments to get 54
models or data points to compute 𝑟 , and we find 𝑟 = −0.91 (p-value:< 1 × 10−8 ). We present another example for the pair 𝑀3𝐶 − 𝑀7𝐺 in
the bottom three panels in Fig. 3, where 𝑀3𝐶 is the accuracy between
user feedback type 𝑡 and model predicted feedback type 𝑡 ′ , and 𝑀7𝐺
is the mean of the confidence scores on ground-truth feedback type,
and we see positive correlations for all three experiments. Again,as the model gets better, both 𝑀3𝐶 and 𝑀7𝐺 should increase. We find𝑟 = 0.79 (p-value: 7.2 × 10−4 ), 0.53 (p-value: 0.022), 0.80 (p-value:
1.4 × 10−5 ) for SLA, 𝑛𝑙 and 𝑛𝑠 variation experiments, respectively.
Considering all 54 models, for this pair, we find 𝑟 = 0.87 (p-value:< 1 × 10−8 ). We also note that the best models found for all threevariation experiments, based on MG metrics in Fig. 2, are also thebest models found using MC metrics. For example, for 𝑛𝑙 variation experiment, we see the best results around 𝑛𝑙 = 5 for both
MG and MC metrics. Therefore, we can conclude that the MC metrics are indeed useful in determining the optimal values for these
hyper-parameters.
Considering all 54 models, we computed 𝑟 for all 77 pairs between
MG and MC . The results for 𝑟 for all those pairs are shown in Fig. 4.
The statistically significant correlations that have p-values less thanor equal to 0.05 are shown using the black font color, and we find
41 such pairs. Out of those 41 pairs, there are 10 pairs for which|𝑟 | > 0.7 with p-values < 1 × 10−9 (highlighted using the black box
in Fig. 4) that we reckon point to strong correlations.

Establishing Reliability in Crowdsourced Voice Feedback (CVF) and Evaluating the Reliability in the Absence of Ground-Truth

M C8n l −variations

SLA −variations

1.0

●

●

● ●

●

● ● ●

●

●

●

●

●

● ●

● ● ●

0.5

●

Max(M C8 ) 0.0

● ●

−0.5
●

0.9

M C3

● ● ● ●●

● ●● ● ● ●

●

●

● ●

● ●

●

●

● ●

● ●

● ● ●

0.0 Max(M 6 )●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ●

−0.5
●

● ● ●

● ● ● ● ● ● ● ● ● ●0.9●

Max(M C3 ) 0.7

● ● ●

7

●

● ●

0

5

10

MG
7

0.7 Max(M G)

●

●
●

0.5

MG
6
G

●
●

●

1.0
0.5

● ● ●

●
● ●

●

n s −variations

● ● ●

● ●

●

DeMaL’21, August 2021, Virtual

0.5

15

20

25

30

35

40

45

50

55

Model Index

CVF data based metrics, M

CFigure 3: Top panels: Normalized 𝑀8𝐶 and 𝑀6𝐺 for all models generated by varying three different hyper-parameters: (i) Structurelearning algorithm (SLA), (ii) number of levels used while discretizing continuous variables to discrete variables, 𝑛𝑙 , and (iii)number of samples in the training data set, 𝑛𝑠 , with a common x-axis indicating the Model Index explained in the text. Bottom
panels: Normalized 𝑀3𝐶 and 𝑀7𝐺 for all models generated by varying three different hyper-parameters with a common x-axis.
The normalization is done by simply diving the values by maximum of respective array.
Markers: −■− CVF based metric, − ⃝ − Ground-truth based metric.
MC
11

0.22

0.02

0.38

0.07

0.36

0.3

0

MC
10

−0.21

−0.06

−0.38

−0.11

−0.34

−0.28

−0.07

MC
9

0.11

−0.17

0.4

−0.11

0.39

0.27

−0.17

1.0

MC
8

−0.35

−0.1

−0.58

0.06

−0.84

−0.91

−0.01

0.5

MC
7

0.52

0.28

0.64

0.17

0.72

0.72

0.19

MC
6

−0.5

−0.34

−0.55

−0.22

−0.62

−0.61

−0.32

−1.0

MC
5

0.57

0.29

0.85

0.2

0.65

0.69

0.27

p−value

MC
4

−0.41

−0.17

−0.6

−0.02

−0.8

−0.88

−0.04

MC
3

0.6

0.85

0.1

0.81

−0.02

−0.09

0.87

MC
2

a low
a med
a high

0.04

0.27

−0.1

0.27

−0.13

−0.28

0.51

MC
1

−0.24

−0.07

−0.21

−0.08

−0.11

−0.2

0.23

MG
1

MG
2

MG
3

MG
4

MG
5

MG
6

MG
7

r

0.0
−0.5

G

Ground−truth data based metrics, M

Figure 4: Results for Spearman’s rank correlation coefficient𝑟 between all 77 MG and MC metrics pairs. The color scale on
cells indicates 𝑟 with green as 𝑟 = 1, red as 𝑟 = −1 and white as𝑟 = 0. The p-value for each 𝑟 statistic is indicated using threefont colors in each cell, with labels “low" (black) as p-value≤ 0.05, “med" (gold) as 0.05 < p-value ≤ 0.1 and “high" (grey)as p-value > 0.1. Cells that are highlighted within black box
refer to pairs with |𝑟 | > 0.7 with p-values < 1 × 10−9 .

On those ten strongly correlated pairs, we see six MG metrics:(i) 𝑀2𝐺 , (ii) 𝑀3𝐺 , (iii) 𝑀4𝐺 , (iv) 𝑀5𝐺 , (v) 𝑀6𝐺 , and (vi) 𝑀7𝐺 and five
MC metrics: (i) 𝑀3𝐶 , (ii) 𝑀4𝐶 , (iii) 𝑀5𝐶 , (iv) 𝑀7𝐶 , and (v) 𝑀8𝐶 . Thismeans that we can rely on those five MC metrics (in the absenceof ground-truth data) only if the true success metric to measure

the model performance is one (or more) of those six MG metrics.
For example, if the objective is to maximize the mean of ROC AUC
across all feedback types (𝑀5𝐺 ), as if we had ground-truth data, thenwe can proxy that using one of the three non-ground-truth based
metrics 𝑀4𝐶 , 𝑀7𝐶 or 𝑀8𝐶 . Since we see the highest correlation of
𝑀5𝐺 with 𝑀8𝐶 , which is the mean of the confidence scores on user
feedback in the ENS test set, we recommend to use 𝑀8𝐶 . However, if
the objective is to optimize some other metrics (from 𝑀2𝐺 , 𝑀3𝐺 , 𝑀4𝐺 ,𝑀5𝐺 , 𝑀6𝐺 , 𝑀7𝐺 ), then Fig. 4 can be used to select the most reasonable
MC choice.

7

CONCLUSIONSWe proposed a way to leverage a BBN model to establish confidence
scores for crowdsourced voice feedback (CVF) data. We carriedout several experiments by varying different hyper-parameters,such as SLA, 𝑛𝑙 and 𝑛𝑠 , and presented their effects on the model
performance using some existing and new metrics identified inthis paper. We discussed 11 metrics that can be computed without
ground-truth data (MC metrics), and merely using the CVF data.
We studied correlations of those 11 metrics with 7 ground-truth
based metrics (MG , for which we curated a small ground-truth
test set), where all those metrics were computed while carrying
out hyper-parameter experiments mentioned above. On those 77
pairs, we found 10 metrics pairs between MC -MG that had strong
correlations with absolute value of Spearman’s rank correlation
greater than 0.7. Using the correlation results, we showed that one
or more of five MC metrics (𝑀3𝐶 , 𝑀4𝐶 , 𝑀5𝐶 , 𝑀7𝐶 , and 𝑀8𝐶 ) can be used
to measure the model performance in the absence of the groundtruth data, depending on what ground-truth metrics (such as (i)mean of precision, (ii) mean of recall, (iii) mean of F1, (iv) mean of
ROC AUC, across all feedback types, and (v) mean of differences in
confidence scores between ground-truth label and user feedback

DeMaL’21, August 2021, Virtuallabel, and (vi) mean of confidence scores on ground-truth label) wewant to optimize.

ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their insightfulcomments.

REFERENCES[1] Hirotogu Akaike. 1973. Information Theory and an Extension of the Maximum
Likelihood Principle. Springer New York, New York, NY, 199–213.
[2] Constantin F. Aliferis, Alexander Statnikov, Ioannis Tsamardinos, Subramani
Mani, and Xenofon D. Koutsoukos. 2010. Local Causal and Markov Blanket
Induction for Causal Discovery and Feature Selection for Classification Part I:
Algorithms and Empirical Evaluation. Journal of Machine Learning Research 11,
7 (2010), 171–234.
[3] Irad Ben-Gal. 2008. Bayesian Networks. American Cancer Society. https://doi.
org/10.1002/9780470061572.eqr089[4] Adam L. Berger, Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A
Maximum Entropy Approach to Natural Language Processing. Computational
Linguistics 22, 1 (1996), 39–71. https://www.aclweb.org/anthology/J96-1002[5] Dheeraj Bhaskaruni, Fiona Moss, and Chao Lan. 2018. Estimating Prediction
Qualities without Ground Truth: A Revisit of the Reverse Testing Framework.
49–54. https://doi.org/10.1109/ICPR.2018.8545706[6] Pierce Burke and Richard Klein. 2020. Confident in the Crowd: Bayesian Inference
to Improve Data Labelling in Crowdsourcing. 1–6. https://doi.org/10.1109/
SAUPEC/RobMech/PRASA48453.2020.9041099[7] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. 2019. Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels.
In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan
Salakhutdinov (Eds.). PMLR, 1062–1070.
[8] C. Chow and C. Liu. 1968. Approximating discrete probability distributions withdependence trees. IEEE Transactions on Information Theory 14, 3 (1968), 462–467.
https://doi.org/10.1109/TIT.1968.1054142[9] Diego Colombo and Marloes H. Maathuis. 2014. Order-Independent ConstraintBased Causal Structure Learning. Journal of Machine Learning Research 15, 116(2014), 3921–3962. http://jmlr.org/papers/v15/colombo14a.html[10] Mohamad Dolatshah, Mathew Teoh, Jiannan Wang, and Jian Pei. 2018. Cleaning
crowdsourced labels using oracles for statistical classification. Proceedings of the
VLDB Endowment (2018), 376–389.
[11] Maksym Fedorchuk and Bart Lamiroy. 2017. Binary Classifier Evaluation Without
Ground Truth. In 2017 Ninth International Conference on Advances in Pattern
Recognition (ICAPR). 1–6. https://doi.org/10.1109/ICAPR.2017.8593175[12] Nir Friedman, Iftach Nachman, and Dana Pe’er. 1999. Learning Bayesian Network
Structure from Massive Datasets: The "Sparse Candidate" Algorithm. 206–215.
https://doi.org/10.13140/2.1.1125.2169[13] Nir Friedman and Zohar Yakhini. 1996. On the Sample Complexity of Learning
Bayesian Networks. In Proceedings of the Twelfth International Conference on
Uncertainty in Artificial Intelligence (Portland, OR) (UAI’96). Morgan Kaufmann
Publishers Inc., San Francisco, CA, USA, 274–282.
[14] Maxime Gasse, Alex Aussem, and Haytham Elghazel. 2014. A hybrid algorithm
for Bayesian network structure learning with application to multi-label learning.
Expert Systems with Applications 41, 15 (Nov 2014), 6755–6772. https://doi.org/
10.1016/j.eswa.2014.04.032[15] Søren Højsgaard. 2012. Graphical Independence Networks with the gRain
Package for R. Journal of Statistical Software 46, 10 (2012), 1–26. https://doi.org/10.18637/jss.v046.i10[16] Hui Jiang. 2005. Confidence measures for speech recognition: A survey. SpeechCommunication 45, 4 (2005), 455–470. https://doi.org/10.1016/j.specom.2004.12.
004[17] Kaustubh Kalgaonkar, Chaojun Liu, Yifan Gong, and Kaisheng Yao. 2015. Estimating confidence scores on ASR results using recurrent neural networks. In 2015
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
4999–5003. https://doi.org/10.1109/ICASSP.2015.7178922[18] Shubham Kapoor and Caglar Tirkaz. 2019. Bootstrapping NLU Models with
Multi-task Learning.
[19] Yelin Kim, Joshua Levy, and Yang Liu. 2020. Speech Sentiment and Customer
Satisfaction Estimation in Socialbot Conversations. In Proc. Interspeech 2020.
1833–1837. https://doi.org/10.21437/Interspeech.2020-2890[20] Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models: Principles
and Techniques. MIT Press.
[21] Steffen L. Lauritzen. 1996. Graphical Models. Oxford University Press.
[22] Qiujia Li, Preben Ness, Anton Ragni, and Mark Gales. 2019. Bi-directional Lattice
Recurrent Neural Networks for Confidence Estimation. In ICASSP 2019 - 2019

Aashish Jain and Sudeeksha MurariIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
6755–6759. https://doi.org/10.1109/ICASSP.2019.8683488[23] Dimitris Margaritis. 2003. Learning Bayesian Network Model Structure From Data.
Ph.D. Dissertation. Carnegie Mellon University.
[24] Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. 2021. Confident Learning:
Estimating Uncertainty in Dataset Labels. J. Artif. Intell. Res. 70 (2021), 1373–1411.
[25] Alexandros Pappas and Duncan F. Gillies. 2002. A New Measure for the Accuracyof a Bayesian Network. In MICAI 2002: Advances in Artificial Intelligence. Springer
Berlin Heidelberg, Berlin, Heidelberg, 411–419.
[26] Judea Pearl. 2009. Probabilistic reasoning in intelligent systems : Networks ofplausible inference. Morgan Kaufmann, San Francisco, Calif. Example for
explaining away.
[27] Jose M. Peña. 2008. Learning Gaussian Graphical Models of Gene Networks withFalse Discovery Rate Control. In Evolutionary Computation, Machine Learningand Data Mining in Bioinformatics, Elena Marchiori and Jason H. Moore (Eds.).
Springer Berlin Heidelberg, Berlin, Heidelberg, 165–176.
[28] Stuart J. Russell and Peter Norvig. 2009. Artificial Intelligence: A modern approach(3 ed.). Pearson.
[29] Ruhi Sarikaya, A. Paul Crook, Alex Marin, Minwoo Jeong, Jean-Philippe Robichaud, Asli Çelikyilmaz, Young-Bum Kim, Alexandre Rochette, Zia Omar Khan,
Xiaohu Liu, Daniel Boies, Tasos Anastasakos, Zhaleh Feizollahi, Nikhil Ramesh,
H. Suzuki, Roman Holenstein, Elizabeth Krawczyk, and Vasiliy Radostev. 2016.
An overview of end-to-end language understanding and dialog management for
personal digital assistants. 2016 IEEE Spoken Language Technology Workshop(SLT) (2016), 391–397.
[30] Gideon Schwarz. 1978. Estimating the Dimension of a Model. The Annals of
Statistics 6, 2 (1978), 461 – 464. https://doi.org/10.1214/aos/1176344136[31] Marco Scutari. 2010. Learning Bayesian Networks with the bnlearn R Package.
Journal of Statistical Software 35, 3 (2010), 1–22. https://doi.org/10.18637/jss.v035.
i03[32] Marco Scutari. 2017. Bayesian Network Constraint-Based Structure Learning
Algorithms: Parallel and Optimized Implementations in the bnlearn R Package.
Journal of Statistical Software 77, 2 (2017), 1–20. https://doi.org/10.18637/jss.v077.
i02[33] Ioannis Tsamardinos, Constantin Aliferis, and A Statnikov. 2003. Time andSample Efficient Discovery of Markov Blankets And Direct Causal Relations. (042003), 673–678. https://doi.org/10.1145/956750.956838[34] Ioannis Tsamardinos, Constantin F. Aliferis, and Alexander R. Statnikov. 2003.
Algorithms for Large Scale Markov Blanket Discovery.. In FLAIRS Conference,
Ingrid Russell and Susan M. Haller (Eds.). AAAI Press, 376–381.
[35] Ioannis Tsamardinos, Laura Brown, and Constantin Aliferis. 2006. The MaxMin Hill-Climbing Bayesian Network Structure Learning Algorithm. Machine
Learning 65 (10 2006), 31–78. https://doi.org/10.1007/s10994-006-6889-7[36] Fan Yang, Mengnan Du, and Xia Hu. 2019. Evaluating Explanation Without Ground Truth in Interpretable Machine Learning. arXiv e-prints, Article
arXiv:1907.06831 (July 2019), arXiv:1907.06831 pages. arXiv:1907.06831 [cs.LG][37] Sandeep Yaramakala and Dimitris Margaritis. 2005. Speculative Markov Blanket
Discovery for Optimal Feature Selection. In Proceedings of the Fifth IEEE International Conference on Data Mining (ICDM ’05). IEEE Computer Society, USA,
809–812. https://doi.org/10.1109/ICDM.2005.134

A

MODEL HYPER-PARAMETERSEstablishing Reliability in Crowdsourced Voice Feedback (CVF) and Evaluating the Reliability in the Absence of Ground-Truth

Table 3: List of hyper-parameters for all models generated.
Model Index
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

SLA

𝑛𝑙

𝑛𝑠

chow.liu
fast.iamb
gs
h2pc
hc
iamb
iamb.fdr
inter.iamb
mmhc
mmpc
pc.stable
rsmax2
si.hiton.pc
tabuinter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb
inter.iamb

5
5
5
5
5
5
5
5
5
5
5
5
5
5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5

2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
2 × 106
1 × 103
2 × 103
5 × 103
7 × 103
8 × 103
9 × 103
1 × 104
1.1 × 104
1.2 × 104
1.5 × 104
2 × 104
5 × 104
7 × 104
1 × 105
2 × 105
5 × 105
7 × 105
1 × 106
2 × 106
5 × 106
7.5 × 106

DeMaL’21, August 2021, Virtual