SlowGuess/ACL_2020_OCR huggingface dataset:
Estimating Predictive Uncertainty for Rumour Verification Models
Elena Kochkina
University of Warwick, UK The Alan Turing Institute, UK E.Kochkina@warwick.ac.uk
Maria Liakata
Queen Mary University of London, UK  
The Alan Turing Institute, UK  
University of Warwick, UK  
mliakata@turing.ac.uk
The inability to correctly resolve rumours circulating online can have harmful real-world consequences. We present a method for incorporating model and data uncertainty estimates into natural language processing models for automatic rumour verification. We show that these estimates can be used to filter out model predictions likely to be erroneous, so that these difficult instances can be prioritised by a human fact-checker. We propose two methods for uncertainty-based instance rejection, supervised and unsupervised. We also show how uncertainty estimates can be used to interpret model performance as a rumour unfolds.
One of the greatest challenges of the information age is the rise of pervasive misinformation. Social media platforms enable it to spread rapidly, reaching wide audiences before manual verification can be performed. Hence there is a strive to create automated tools that assist with rumour resolution. Information about unfolding real-world events such as natural disasters often appears in a piece-wise manner, making verification a time-sensitive problem. Failure to identify misinformation can have a harmful impact, thus it is desirable that an automated system aiding rumour verification does not only make a judgement but that it can also inform a human fact-checker of its uncertainty.
Deep learning models are currently the state-of-the-art in many Natural Language Processing (NLP) tasks, including rumour detection (Ma et al., 2018), the task of identifying candidate rumours, and rumour verification (Li et al., 2019; Zhang et al., 2019), where the goal is to resolve the veracity of a rumour. Latent features and large parameter spaces of deep learning models make it hard to interpret a model's decisions. Increasingly researchers are investigating methods for understand-
ing model predictions, such as through analysing neural attention (Vaswani et al., 2017) and studying adversarial examples (Yuan et al., 2019). Another way to gain insights into a model's decisions is via estimating its uncertainty. Understanding what a model does not know can help us determine when we can trust its output and at which stage information needs to be passed on to a human (Kendall and Gal, 2017).
In this paper, rather than purely focusing on the performance of a rumour verification model, we estimate its predictive uncertainty to gain understanding of a model's decisions and filter out the cases that are 'hard' for the model. We consider two types of predictive uncertainty: data uncertainty (aleatoric) and model uncertainty (epistemic). The approach we adopt requires minimal changes to a given model and is relatively computationally inexpensive, thus making it possible to apply to various architectures.
- We are the first to apply methods for uncertainty estimation to the problem of rumour verification. We show that removing instances with high uncertainty filters out many incorrect predictions, gaining performance improvement in the rest of the dataset.
- We propose a supervised method for instance removal that combines both aleatoric and epistemic uncertainty and outperforms an unsupervised approach.
- We propose a way to analyse uncertainty patterns as a rumour unfolds in time. We make use of this to study the relation between the stance expressed in response tweets and fluctuation in uncertainty at the time step following a response.
- We explore the relationship between uncertainty estimates and class labels.
6964
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6964-6981 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2.1 Rumour Verification
A rumour is a circulating story of questionable veracity, which is apparently credible but hard to verify, and produces sufficient skepticism/anxiety so as to motivate finding out the actual truth (Zubiaga et al., 2018). Rumour detection and verification in online conversations have gained popularity as tasks in recent years (Zubiaga et al., 2016; Ma et al., 2016; Enayet and El-Beltagy, 2017). Existing works aim to improve performance of supervised learning algorithms that classify claims, leveraging linguistic cues, network- and user-related features, propagation patterns, support among responses and conversation structure (Derczynski et al., 2017; Gorrell et al., 2018). Due to the nature of the task, each rumour can be considered as a new domain and existing models struggle with generalisability. Here we employ model-agnostic methods of uncertainty estimation that can provide performance improvements and insight on the working of the models to inspire further development.
2.2 Related Work on Uncertainty Estimation
There is a growing body of literature which aims to estimate predictive uncertainty of deep neural networks (DNNs) (Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Malinin and Gales, 2018). Gal and Ghahramani (2016) have shown that application of Monte-Carlo (MC) Dropout at testing time can be used to derive an uncertainty estimate for a DNN. Lakshminarayanan et al. (2017) estimate model uncertainty by using a set of predictions from an ensemble of DNNs, while Malinin and Gales (2018) propose a specialised framework, Prior Networks, for modelling predictive uncertainty. Here we focus on the dropout method proposed by Gal and Ghahramani (2016) as it is computationally inexpensive, relatively simple and does not interfere with model training.
Within NLP Xiao and Wang (2018) have used aleatoric (Kendall and Gal, 2017) and epistemic (Gal and Ghahramani, 2016) uncertainty estimates for Sentiment analysis and Named Entity Recognition. Dong et al. (2018) used a modification of Gal and Ghahramani (2016) method to output confidence scores for Neural Semantic Parsing.
Rumour Verification is a task where levels of certainty play a crucial role because of the potentially high impact of erroneous decisions. Moreover, unlike other tasks, it is a time-sensitive problem: as
Figure 1: branch-LSTM model
new information comes to light the level of certainty is expected to change giving insights into a model's predictions. We therefore explore the dynamics of uncertainty as a discussion unfolds in section 6.3. Note that data and model uncertainty should not be confused with uncertainty expressed by a user in a post. Automatically identifying levels of uncertainty expressed in text is a challenging NLP task (Jean et al., 2016; Vincze, 2015), which could be complementary to predictive uncertainty in the case of rumour verification.
Active Learning and Uncertainty: Uncertainty estimates could be used in an Active Learning (AL) setup. This would involve using uncertainty estimates over the model's predictions to select instances whose manual labelling and addition to the training set would yield the most benefit (Olsson, 2009). Active learning has been applied to various NLP tasks in the past (Settles and Craven, 2008). More recently Siddhant and Lipton (2018) have shown that Bayesian active learning by disagreement, using uncertainty estimates provided either by Dropout (Gal and Ghahramani, 2016) or Bayes-by-Backprop (Blundell et al., 2015) significantly improves over i.i.d. baselines and usually outperforms classic uncertainty sampling on a number of NLP tasks and datasets. Bhattacharjee et al. (2017, 2019) applied AL to identifying misinformation in news and social media. Our work could be applied in an AL setup to close the loop in incrementally training a model for misinformation using predictive uncertainty.
3.1 Rumour Verification Model
We describe the rumour verification model which forms the basis of our experiments. This served as
6965
Figure 2: Example of a conversation from the PHEME dataset. Branches are highlighted as lines connecting the tweets.
a competitive baseline model (branch-LSTM) for a Semeval task on rumour verification (RumourEval 2019) (Gorrell et al., 2018) $^{1}$ . To process a conversation discussing a rumour while preserving some of the structural relations between the tweets, a tree-like conversation is split into branches, i.e. linear sequences of tweets, as shown in Figure 2. Branches are then used as training instances for a branch-LSTM model consisting of an LSTM layer followed by several ReLU layers and a softmax layer (default base of e and temperature of 1) that predicts class probabilities. Here we use outputs from the final time steps (see Figure 1). Given a training instance, branch of tweets  $x_{i}$ ,  $i \in [1,..,N]$ , where  $N$  is the number of branches, and the label  $y_{i}$ , represented as one-hot vector of size  $C$ , where  $C$  is the number of classes, the loss function  $l_{1}$  (categorical cross entropy) is calculated as follows:
$$
u _ {i} = f \left(x _ {i}\right)
$$
$$
v _ {i} = W _ {v} u _ {i} + b _ {v}
$$
$$
p _ {i} = \operatorname {s o f t m a x} (v _ {i}) = \frac {e ^ {v _ {i}}}{\sum_ {k = 1} ^ {C} e ^ {v _ {i} ^ {k}}}
$$
$$
l _ {1} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {C} y _ {i} ^ {k} \log p _ {i} ^ {k},
$$
where  $u_{i}$  is an intermediate output of layers prior to the softmax layer,  $v_{i}$  is logits, and  $p_{i}$  are predicted class probabilities for a training instance  $x_{i}$ . To obtain predictions for each of the conversation trees we average class probabilities for each of the branches in the tree. In this case tweets are represented as the average of the corresponding word2vec word embeddings, pre-trained on the Google News dataset (300d) (Mikolov et al., 2013).
3.2 Uncertainty Estimation
We consider two types of uncertainty as described in Kendall and Gal (2017): data uncertainty (aleatoric) and model uncertainty (epistemic). Data uncertainty is normally associated with properties of the data, such as imperfections in the measurements. Model uncertainty on the other hand comes from model parameters and can be explained away given enough (i.e. an infinite amount of) data.
We also use the output of the softmax layer to measure the confidence of the model. There are four common ways to calculate uncertainty using the output of the softmax layer: Least Confidence Sampling, Margin of Confidence, Ratio of Confidence and Entropy (Munro, 2019). Here we use the highest class probability as a confidence measure and refer to it as 'softmax'. Using other strategies lead to similar conclusions (see appendices).
3.2.1 Data Uncertainty
We assume aleatoric uncertainty to be a function of the data that can be learned along with the model (Kendall and Gal, 2017). Conceptually, this input-dependent uncertainty should be high when it is hard to predict the output given a certain input.
In order to estimate aleatoric uncertainty associated with input instances, we add an extra output to our model that represents variance  $\sigma$ . We then incorporate  $\sigma$  into the loss function according to Kendall and Gal (2017), in the following way.
$$
\sigma_ {i} = \text {s o f t p l u s} \left(W _ {\sigma} u _ {i} + b _ {\sigma}\right) = \ln \left(1 + e ^ {W _ {\sigma} u _ {i} + b _ {\sigma}}\right)
$$
Here we assume that predictions come from a normal distribution with mean  $v$  and variance  $\sigma$ . We sample  $v$ , distorted by Gaussian noise,  $T$  times, put each through a softmax layer and pass to a standard categorical cross entropy loss function to obtain a
<https://github.com/kochkinaelena/RumourEval2019
6966
mean over losses for all  $T$  samples.
$$
\begin{array}{l} d _ {t, i} = v _ {i} + \sqrt {\sigma_ {i}} * \epsilon , \epsilon \sim N (0, 1) \\ l _ {2} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {k = 1} ^ {C} y _ {i} ^ {k} l o g (s o f t m a x (d _ {t, i}) ^ {k}) \\ \end{array}
$$
Here  $l = w_{1}l_{1} + w_{2}l_{2}$  is the total loss. If the original prediction  $u$  was incorrect, we would need a high  $\sigma$  to have varied samples away from it and hence lower the loss. In the opposite case,  $\sigma$  should be small such that all samples yield a similar result, thus minimising the loss function.  $\sigma$  is chosen as the unbound variance in logit space, which, after the model is trained, approximates input-dependent variance. This method can be applied to a wide range of models, but since it changes the loss function, it is likely to affect a model's performance.
3.2.2 Model Uncertainty
To obtain epistemic uncertainty we use the approach proposed by Gal and Ghahramani (2016), which allows estimating uncertainty about a model's predictions by applying dropout at testing time and sampling from the approximate posterior. This approach requires no changes to the model, does not affect performance, and is relatively computationally inexpensive. We apply dropout at testing time N times and obtain N predictions. We evaluate the differences between them to obtain a single uncertainty value in the following ways:
Variation Ratio Each of the sampled softmax predictions can be converted into an actual class label. We then define epistemic uncertainty as the proportion of cases which are not in the mode category (the label that appears most frequently).
$$
v = 1 - N _ {m} / N _ {t o t a l},
$$
where  $N_{m}$  is the number of cases belonging to the mode category (most frequent class). Thus the variation ratio is 0 when all of the sampled predictions agree, indicating low model uncertainty. The upper bound would differ depending on the number of cases, but will not reach 1.
Entropy Given an array of predictions, we average over them and then calculate predictive entropy as follows:
$$
s = - \sum_ {i} p _ {i} \log p _ {i}.
$$
Variance Each prediction is a vector, the output of a softmax layer (entries in [0,1] which sum up to 1), of size equal to the number of classes. We calculate the variance across each dimension and then take the max value of variance as our uncertainty estimate.
3.3 Instance Rejection
We assume that instances yielding high predictive uncertainty values are likely to be incorrectly predicted. We therefore make use of predictive uncertainty to filter out instances and explore the tradeoff between model performance and coverage of a dataset. We perform instance rejection in two ways; unsupervised and supervised.
Unsupervised We remove portions of a dataset corresponding to instances with the highest uncertainty (separately for each uncertainty type).
Supervised We train a supervised meta-classifier on a development set using features composed of uncertainty estimates (aleatoric, variance, entropy, variation ratio), the averaged softmax layer output and the model's prediction to decide whether an instance is correctly predicted. We reject instances classified as incorrect and evaluate performance on the rest. We compare two strong baseline models for this task: Support Vector Machines (SVM) and Random Forest (RF). Supervised rejection allows us to leverage all forms of uncertainty together and also dictates the number of instances to remove.
Random We have compared the two instance rejection methods above against removing portions of the test set at random. The outcome of the rejection at random does not lead to consistent performance improvement (see appendix A).
3.4 Time-sensitive uncertainty estimates
Since rumour verification is a time-sensitive task, we have performed analysis of model uncertainty over time, as a rumour unfolds. As illustrated in Figure 3 we have deconstructed the timeline of the development of a conversation tweet by tweet, starting with just the source tweet (initiating the rumour) and adding one response at a time. We have then obtained model predictions and associated uncertainties for each sub-tree. As the difference between each sub-tree is a single tweet, we can track the development of uncertainty alongside the development of a conversation, and the effect each added response has.
6967
Figure 3: Development of a conversation tree over time and its decomposition into branches
3.5 Calibration
Uncertainty estimates obtained do not correspond to the actual probabilities of the prediction being correct, they instead order the samples from the least likely to be correct to the most likely. While the order provided by the scores is sufficient for unsupervised and supervised rejection, these scores can be on a different scale for different datasets and do not allow for direct comparison between models, i.e. they are not calibrated. Calibration refers to a process of adjusting confidence scores to correspond to class membership probabilities, i.e if  $N$  predictions have a confidence of 0.5, then  $50\%$  of them should be correctly classified in a perfectly calibrated case. Modern neural networks are generally poorly calibrated and hyper-parameters of the model influence the calibration (Guo et al., 2017). MC dropout uncertainty is thus also influenced by hyperparameters but can be calibrated using dropout probability (Gal, 2016).
To evaluate how well confidence scores are calibrated, one can use reliability diagrams and Expected Calibration Error (ECE) scores (Guo et al., 2017). ECE is obtained by binning  $n$  confidence scores into  $M$  intervals and comparing the accuracy of each bin against the expected one in a perfectly calibrated case (equal to the confidence of the bin):  $ECE = \sum_{m=1}^{M} \frac{|B_m|}{n} |acc(B_m) - conf(B_m)|$ . Confidence calibration can be improved using Calibration methods. These are post-processing steps that produce a mapping from existing scores to calibrated probabilities using a held-out set. Common approaches are Histogram binning, Isotonic regression and Temperature scaling (Guo et al., 2017).
In our experiments we use publicly available datasets of Twitter conversations discussing ru
# Posts	#Trees	T	F	U	NR
PHEME	33288	2410	1067	639	704	0
Twitter 15	40927	1374	350	336	326	362
Twitter 16	18770	735	189	173	174	199
Table 1: Number of posts, conversation trees and class distribution in the datasets (T - True, F - False, U - Unverified, NR - Non-Rumour).
mours. Table 1 shows the number of conversation trees in the datasets and the class distribution.
4.1 PHEME
We use conversations from the PHEME dataset discussing rumours related to nine newbreaking events. Rumours in this dataset were labeled as True, False or Unverified by professional journalists (Zubiaga et al., 2016). When conducting experiments on this dataset we perform cross-validation in a leave-one-event-out setting, i.e. using all the events except for one as training, and the remaining event as testing. This is a challenging setup, imitating a real-world scenario, where a model needs to generalise to unseen rumours. The number of rumours, the number of the corresponding conversations, as well as the class label distribution (true-false-unverified) vary greatly across events.
4.2 Twitter 15/16
The Twitter 15 and Twitter 16 datasets were made publicly available by Ma et al. (2017), and were created using reference datasets from MaMa et al. (2016) and Liu et al. (2015). Claims were annotated using veracity labels on the basis of articles corresponding to the claims found in rumour debunking websites such as snoopes.com and emergent.info. These datasets merge rumour detection and verification into a single four-way classification task, containing True, False and Unverified rumours as well as Non-Rumours. Both datasets are split into 5 folds for cross validation, and contrary to the PHEME dataset, folds are of approximately equal size with a balanced class distribution.
We perform cross-validation on all of the datasets. When choosing parameters, we choose one of the folds within each dataset to become the development set: CharlieHebdo in PHEME (large fold with balanced labels) and fold 0 in Twitter 15 and Twitter 16. We evaluate models using both accuracy and macro F-score due to the class imbalance in
6968
Figure 4: Unsupervised rejection of instances with the highest uncertainty and corresponding lowest confidence (softmax) values across 3 datasets. The Y-axis shows performance in terms of accuracy, on the X-axis the percentage of the remaining instances is shown.
All instances	Classifier	N removed	Supervised rejection	Unsupervised rejection
aleatoric	epistemic (variation ratio)	softmax
Accuracy	Macro F	Accuracy	Macro F	Accuracy	Macro F	Accuracy	Macro F	Accuracy	Macro F
PHEME	0.278	0.225	SVM	1057	0.399	0.196	0.306	0.216	0.35	0.235	0.332	0.239
RF	1179	0.378	0.235	0.311	0.217	0.346	0.227	0.329	0.236
Twitter 15	0.671	0.67	SVM	402	0.806	0.801	0.656	0.632	0.801	0.795	0.794	0.788
RF	504	0.834	0.829	0.662	0.624	0.836	0.828	0.818	0.811
Twitter 16	0.755	0.756	SVM	184	0.895	0.893	0.751	0.744	0.885	0.878	0.878	0.868
RF	197	0.897	0.892	0.755	0.747	0.887	0.878	0.884	0.873
Table 2: How rejecting instances using supervised and unsupervised methods affects model performance across datasets, in terms of both accuracy and macro F-score. Performance values were obtained in a separate set of experiments, by removing one of the folds from the training set, as supervised models needed an extra development set to be trained on.
the PHEME dataset<sup>2</sup>. During the cross-validation iterations each fold becomes a testing set once. We then aggregate model predictions from each fold, resulting in predictions for the full dataset, and use them to perform evaluation as well as unsupervised instance rejection based on uncertainty levels.
To perform supervised rejection we need to train a meta-classifier on a subset of data that was not used for training the rumour verification model. Therefore in a separate set of experiments we exclude one of the folds (development set) from training of the verification model. We run cross-validation with one less fold and at each step obtain predictions and uncertainty estimates for both the test fold and the development set. We then use the predictions and uncertainty values predicted for the instances in the development set as training instances in our rejection meta-models, which we then evaluate on each of the corresponding test folds, thus obtaining the combined predictions for all of the folds in the dataset except for the development. This set up corresponds to results shown in Table 2, as one of the folds was removed from train-
ing. The results are therefore not directly comparable to the ones in Figure 4 or in previous literature (Kochkina et al., 2018; Ma et al., 2018).
6.1 Unsupervised Rejection
Figure 4 shows the effect of applying unsupervised rejection (as explained in section 3.3). Each plot shows model performance in terms of accuracy, where the first bar of each plot shows model performance with all instances present and the following bars show performance for the corresponding percentage of remaining instances. Figure 4 shows the effect of unsupervised rejection using aleatoric and epistemic uncertainty (calculated as variation ratio, see section  $3.2.2)^{3}$ , as well as the softmax class probabilities as a measure of confidence (1-uncertainty). Initial performance using  $100\%$  of the data (Figure 4) on the PHEME dataset is markedly different to Twitter 15,16 due to the dataset and task-setup differences. On the Twitter 15 dataset branch-LSTM does not reach the state
3We performed experiments using variance and entropy values with similar outcomes (appendix A).
$^{2}$ https://github.com/kochkinaelena/Uncertainty4VerificationModels
(f)
Figure 5: Examples of uncertainty development over time for three conversations discussing rumours from the PHEME dataset. Each of the nodes is labeled with its predicted stance label: green - supporting, red - denying, blue - questioning and black - commenting. Predictions are in bold at the bottom, where F - False, T - True, U - Unverified.