SlowGuess/ACL_2020_OCR huggingface dataset:
Table 12 shows Expected Calibration Error (ECE) before and after the calibration process using the Histogram Binning method for all types of uncertainty. Figure 9 shows corresponding reliability diagrams (calibration curves). We use the experiment setup with one of the folds reserved as development set in order to train the calibration method. We convert uncertainty estimates  $u$  into confidence scores as  $1 - u$ , and for the aleatoric we normalise it to be in [0,1]. Calibration curves were plotted using the function from the scikit-learn package. Implementation of ECE scores and Histogram Binning were adapted from https://github.com/
6975
com/markus93/NN_calibration/blob/master/scripts/calibration/cal_methods.py.
F Datasets
Here we describe how to access the datasets used in the study. We use three publicly available datasets:
F.1 PHEME
The PHEME dataset can be downloaded here: https://figshare.com/articles/PHEME_dataset_for_Rumour_Detection_and_Veracity_Classification/6392078
F.2 Twitter 15,16
The Twitter 15,16 datasets can be downloaded here: https://www.dropbox.com/s/ 7ewzdrbelpmrnxu/rumdetect2017.zip?dl=0 It contains list of tweet ids belonging to the dataset. The split into folds for cross-validation is taken from here: https://github.com/majingCUHK/ Rumor_RvNN/tree/master/nfold
6976
%	# removed	Random	Aleatoric	Entropy	Variance	Variation ratio	LCS	MC	RC	E
100%	0	0.385	0.385	0.385	0.385	0.385	0.385	0.385	0.385	0.385
97.5%	60	0.384	0.391	0.388	0.387	0.386	0.386	0.387	0.387	0.385
95%	120	0.384	0.397	0.388	0.387	0.386	0.386	0.39	0.39	0.389
90%	240	0.382	0.412	0.385	0.387	0.387	0.387	0.389	0.389	0.389
85%	361	0.384	0.417	0.385	0.385	0.386	0.388	0.39	0.39	0.39
80%	481	0.381	0.427	0.385	0.385	0.387	0.388	0.387	0.387	0.387
70%	723	0.374	0.448	0.389	0.389	0.388	0.387	0.386	0.387	0.386
60%	964	0.370	0.481	0.387	0.396	0.394	0.389	0.377	0.377	0.376
50%	1205	0.376	0.528	0.389	0.392	0.391	0.386	0.382	0.378	0.381
Table 5: Performance (accuracy) after unsupervised rejection on PHEME dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer (as opposed to Entropy, Variance and Variation ratio that are based on multiple softmax samples).
%	# removed	Random	Aleatoric	Entropy	Variance	Variation ratio	LCS	MC	RC	E
100.0%	0	0.591	0.591	0.591	0.591	0.591	0.591	0.591	0.591	0.591
97.5%	34	0.589	0.599	0.603	0.599	0.602	0.601	0.601	0.601	0.6
95.0%	68	0.593	0.61	0.609	0.609	0.609	0.609	0.609	0.609	0.609
90.0%	137	0.592	0.63	0.625	0.627	0.622	0.621	0.621	0.620	0.622
85.0%	206	0.597	0.647	0.637	0.646	0.634	0.634	0.634	0.631	0.634
80.0%	274	0.599	0.668	0.648	0.665	0.657	0.630	0.631	0.633	0.630
70.0%	412	0.577	0.642	0.669	0.718	0.699	0.660	0.661	0.660	0.660
60.0%	549	0.596	0.64	0.679	0.77	0.765	0.684	0.684	0.684	0.684
50.0%	687	0.598	0.649	0.677	0.817	0.821	0.723	0.722	0.721	0.723
Table 6: Performance (accuracy) after unsupervised rejection on Twitter 15 dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer.
%	# removed	Random	Aleatoric	Entropy	Variance	Variation ratio	LCS	MC	RC	E
100.0%	0	0.788	0.788	0.788	0.788	0.788	0.788	0.788	0.788	0.788
97.5%	18	0.789	0.784	0.798	0.794	0.796	0.795	0.795	0.795	0.796
95.0%	36	0.787	0.783	0.808	0.805	0.805	0.800	0.801	0.800	0.804
90.0%	73	0.787	0.787	0.837	0.828	0.829	0.828	0.828	0.826	0.828
85.0%	110	0.786	0.787	0.856	0.85	0.856	0.848	0.848	0.848	0.848
80.0%	146	0.789	0.789	0.881	0.868	0.869	0.864	0.864	0.862	0.866
70.0%	220	0.794	0.794	0.905	0.907	0.901	0.905	0.905	0.905	0.905
60.0%	294	0.787	0.803	0.939	0.937	0.937	0.925	0.925	0.925	0.925
50.0%	367	0.78	0.81	0.954	0.957	0.957	0.951	0.951	0.951	0.954
Table 7: Performance (accuracy) after unsupervised rejection on Twitter 16 dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer.
%	EE	FE	GU	OT	PT	PM	SS	CH	GW
100%	0.429	0.062	0.459	0.589	0.240	0.325	0.588	0.353	0.139
90%	0.385	0.053	0.491	0.601	0.217	0.333	0.600	0.360	0.140
80%	0.417	0.030	0.469	0.615	0.196	0.356	0.608	0.370	0.131
70%	0.400	0.02	0.442	0.62	0.186	0.337	0.628	0.376	0.114
60%	0.333	0.023	0.378	0.638	0.181	0.342	0.653	0.391	0.112
50%	0.429	0.014	0.387	0.655	0.174	0.333	0.674	0.422	0.118
Table 8: Unsupervised rejection using variation ratio uncertainty estimates for each event-fold in the PHEME dataset. EE – Ebola-Essien; FE – Ferguson unrest; GU – Gurlitt; OT – Ottawa shooting; PT – Prince-Toronto; PM – Putin missing; SS – Sydney Siege; CH – Charlie Hebdo; GW – Germanwings crash.
%	Aleatoric	Variation ratio	Entropy	Variance	LCS	MC	RC	E
100%	0.385	0.385	0.385	0.385	0.385	0.385	0.385	0.385
90%	0.395	0.389	0.388	0.389	0.392	0.392	0.394	0.391
80%	0.397	0.390	0.393	0.390	0.392	0.392	0.392	0.392
70%	0.400	0.391	0.393	0.391	0.399	0.398	0.395	0.399
60%	0.393	0.401	0.399	0.405	0.399	0.399	0.396	0.399
50%	0.386	0.413	0.413	0.413	0.400	0.401	0.395	0.400
Table 9: Performance (accuracy) after per-fold unsupervised rejection on PHEME dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer (as opposed to Entropy, Variance and Variation ratio that are based on multiple softmax samples).
%	Aleatoric	Variation ratio	Entropy	Variance	LCS	MC	RC	E
100%	0.591	0.591	0.591	0.591	0.591	0.591	0.591	0.591
90%	0.566	0.625	0.625	0.622	0.619	0.619	0.619	0.623
80%	0.558	0.650	0.657	0.652	0.654	0.652	0.651	0.653
70%	0.569	0.699	0.697	0.699	0.675	0.675	0.673	0.674
60%	0.596	0.725	0.714	0.724	0.707	0.708	0.708	0.707
50%	0.603	0.753	0.753	0.756	0.741	0.741	0.741	0.741
Table 10: Performance (accuracy) after per-fold unsupervised rejection on Twitter 15 dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer (as opposed to Entropy, Variance and Variation ratio that are based on multiple softmax samples).
%	Aleatoric	Variation ratio	Entropy	Variance	LCS	MC	RC	E
100%	0.788	0.788	0.788	0.788	0.788	0.788	0.788	0.788
90%	0.783	0.830	0.833	0.821	0.816	0.818	0.821	0.816
80%	0.782	0.870	0.873	0.870	0.866	0.866	0.866	0.865
70%	0.784	0.898	0.902	0.903	0.902	0.902	0.900	0.902
60%	0.810	0.928	0.934	0.932	0.921	0.921	0.921	0.921
50%	0.835	0.954	0.962	0.957	0.938	0.940	0.949	0.949
Table 11: Performance (accuracy) after per-fold unsupervised rejection on Twitter 15 dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer (as opposed to Entropy, Variance and Variation ratio that are based on multiple softmax samples).
No calibration	Histogram Binning
S	A	VR	E	VAR	S	A	VR	E	VAR
PHEME	0.646	0.683	0.492	0.292	0.295	0.173	0.088	0.111	0.119	0.108
Twitter 15	0.265	0.333	0.216	0.119	0.144	0.056	0.039	0.062	0.065	0.066
Twitter 16	0.191	0.196	0.121	0.080	0.109	0.164	0.079	0.044	0.058	0.056
Table 12: Expected Calibration Error before and after applying calibration over uncertainty estimates. S - softmax (LCS), A - aleatoric uncertainty, VR - variation ratio, E - entropy, VAR - variance.
6978
(a) Aleatoric PHEME
(b) Aleatoric Twitter 15
(c) Aleatoric Twitter 16
(d) Epistemic PHEME
(e) Epistemic Twitter 15
(f) Epistemic Twitter 16
Figure 7: Effect of parameters on uncertainty estimates.
6979
(f) Epistemic Twitter 16
(g) Softmax PHEME
(h) Softmax Twitter 15
(i) Softmax Twitter 16
Figure 8: Boxplots showing uncertainty values grouped by the number of tweets in a conversation tree for 3 types of uncertainty estimates: aleatoric, epistemic, softmax. The Y-axis shows uncertainty (a-f) and confidence (g-i) values (a higher number indicates lower uncertainty). Numbers in bold show the number of conversations trees in each of the bins.
(i) Softmax Twitter 16
Figure 9: Reliability diagrams (calibration curves). X-axis shows confidence intervals, Y-axis shows accuracy at each interval (fraction of instances predicted correctly). Bottom plots show the number of instances in each interval. For both plots, blue - before calibration, red - after Histogram Binning.
6981