SlowGuess/ACL_2020_OCR huggingface dataset:
$k = 3$ . It has lower performance than  $k = 2$  across all models. Therefore for the subsequent experimental sections, we only report top  $k$  calibration performance using the heuristic- $k$  value only.
We use the confidence predictions of our fullfeature model  $+\mathrm{Rank} + \mathrm{Var} + \mathrm{top}k$  to re-rank the top $k$  predictions in the test set. Table 2 shows the sentence-level (entity of interest) accuracy for our re-ranked top prediction and the original model prediction.
3.2 Calibration for Named Entities
For Named Entity (NE) Recognition experiments, we use two NE annotated datasets, namely CoNLL 2003 and MADE 1.0. CoNLL 2003 consists of documents from the Reuters corpus annotated
with named entities such as Person, Location etc. MADE 1.0 dataset is composed of electronic health records annotated with clinical named entities such as Medication, Indication and Adverse effects.
The entity of interest for NER is the named entity span prediction. We define  $\phi(x)$  as predicted entity spans in  $\arg\max(k)$  label sequences predictions for  $x$ . We use BERT-base with token-level softmax output and marginal likelihood based training. The model uncertainty estimates for "Var" feature are computed by estimating the variance of length normalized MC-dropout samples of span marginals. Due to the similar trends in behavior of BERT and BERT+CRF model in POS experiments, we only use BERT model for NER. However, the span marginal computation can be easily extended to linear-chain CRF models. We also use the length of the predicted named entity as the feature "ln" in this experiment. Complete details about forecaster and baselines are in the Appendix. Value of heuristic- $k$  is 3 for all +Rank+Var+topk forecasters. We show ablation and baseline results for  $k = 3$  only. However, no other forecasters for any  $k \in \{2,3\}$  outperform our best forecasters in Table 3.
We use the confidence predictions of our "+"Rank+Var+top3" models to re-score the confidence predictions for all spans predicted in top-3 MAP predictions for samples in the test set. A threshold of 0.5 was used to remove span predictions with low confidence scores. Table 4 shows the Named Entity level (entity of interest) MicroF score for our re-ranked top prediction and the original model prediction. We see that re-ranked predictions from our models consistently improve the model f-score.
3.3 Calibration for QA Models
We use three datasets for evaluation of our calibration methods on the QA task. Our QA tasks are modeled as extractive QA methods with a single span answer predictions. We use three datasets to construct experiments for QA calibration. SQuAD1.1 and EMRQA (Pampari et al., 2018) are open-domain and clinical-domain QA datasets, respectively. We process the EMRQA dataset by restricting the passage length and removing unanswerable questions. We also design an out-of-domain evaluation of calibration using clinical QA datasets. We follow the guidelines from Pampari et al. (2018) to create a QA dataset
Calibration	SQuAD1.1(BERT)	EMRQA(bioBERT)	MADE 1.0(bioBERT)	MADE1.0(OOD)(bioBERT)
Platt	3.69±.16	5.07±.37	3.64±.17	15.20±.16
Calibrated Mean	2.95±.26	2.28±.18	2.50±.31	13.26±.94
+Var	2.92±.28	2.74±.15	2.71±.32	12.41±.95
Platt+top3	7.71±.28	5.42±.25	11.87±.19	16.36±.26
Calibrated Mean+top3	3.52±.35	2.11±.19	9.21±.25	12.11±.24
+Var+top3	3.56±.29	2.20±.20	9.26±.27	11.67±.27
+Var+lm+top3	3.54±.21	2.12±.19	6.07±.26	12.42±.32
+Rank+Var+top3	2.47±.18	1.98±.10	1.77±.23	12.69±.20
+Rank+Var+lm+top3	2.79±.32	2.24±.29	1.66±.27	12.60±.28
Table 5: ECE percentages for QA tasks SQuAD1.1, EMRQA and MADE 1.0. MADE 1.0(OOD) refers to the out-of-domain evaluation of a QA model that is trained and calibrated on EMRQA training and validation splits. The results are for top-1 MAP predictions on the test data. ECE standard deviation is estimated by repeating the experiments for 5 repetitions. BERT model's uncalibrated ECE for SQuAD1.1, EMRQA, MADE 1.0 and MADE 1.0(OOD) are  $6.24\%$ $6.10\%$ ,  $20.10\%$  and  $18.70\%$  respectively. heuristic- $k$  is 3 for all +Rank+Var+top $k$  forecasters.
Calibration	SQuAD1.1(BERT)	EMRQA(bioBERT)	MADE 1.0(bioBERT)	MADE1.0(OOD)(bioBERT)
Baseline	79.79±.08	70.97±.14	66.21±.18	31.62±.12
+Rank+Var+top3	80.04±.11	71.34±.22	66.33±.12	31.99±.11
+Rank+Var+lm+top3	80.03±.15	71.37±.26	66.33±.15	32.02±.09
Table 6: Table shows change in Exact Match Accuracy for QA datasets and our best proposed calibration method. The confidence score from the calibration method is used to re-rank the events  $E \in \mathcal{I}(s)$ . Standard deviation is estimated by repeating the experiments for 5 repetitions. Baseline refers to MC-dropout averaged (sample-size=10) output of model  $p_{\theta}$ . heuristic-k is 3 for all +Rank+Var+topk forecasters.
from MADE 1.0 (Jagannatha et al., 2019). This allows us to have two QA datasets with common question forms, but different text distributions. In this experimental setup we can mimic the evaluation of calibration methods in a real-world scenario, where the task specifications may remain the same but the underlying text source changes. Details about dataset pre-processing and construction are provided in the Appendix.
The entity of interest for QA is the top- $k$  answer span predictions. We use the "lm" perplexity as a feature in this experiment to analyze its behaviour in out-of-domain evaluations. We use a 2 layer unidirectional LSTM to train a next word language model on the EMRQA passages. This language model is then used to compute the perplexity of a sentence for the "lm" input feature to the forecaster. We use the same baselines as the previous two tasks.
Based on Table 5, our methods outperform the baselines by a large margin in both in-domain and
out-of-domain experiments. Value of heuristic-  $k$  is 3 for all  $+\mathrm{Rank} + \mathrm{Var} + \mathrm{topk}$  forecasters. We show ablation and baseline results for  $k = 3$  only. However, no other forecasters for any  $k\in \{2,3\}$  outperform our best forecasters in Table 5. Our models are evaluated on SQuAD1.1 dev set, and test sets from EMRQA and MADE 1.0. They show consistent improvements in ECE and Exact Match Accuracy.
Our proposed methods outperform the baselines in most tasks and are almost as competitive in others.
Features and top-  $k$  samples: The inclusion of top-  $k$  features improve the performance in almost all tasks when the rank of the prediction is included. We see large increases in calibration error when the top-  $k$  prediction samples are included in forecaster training without including the rank information in tasks such as CoNLL NER and MADE 1.0 QA. This may be because the  $k = 1,2,3$  predictions
Figure 1: Modified reliability plots (Accuracy - Confidence vs Confidence) on MADE 1.0 QA test. The dotted horizontal line represents perfect calibration. Scatter point diameter denotes bin size. The inner diameter of the scatter point denotes the number of positive events in that bin.
Figure 2: An example of named entity span from CoNLL dataset. Rank is  $k^{th}$  rank from top- $k$  MAP inference (Viterbi decoding). Mean Prob and Std is the mean and standard deviation of length-normalized probabilities (geometric mean of marginal probabilities for each token in the span). Calibrated confidence is the output of  $\text{Rank} + \text{Var} + \ln + \text{top3}$ .
may have similar model confidence and uncertainty values. Therefore a more discriminative signal such as rank is needed to prioritize them. For instance, the difference between probabilities of  $k = 1$  and  $k = 2$  MAP predictions for POS tagging may differ by only one or two tokens. In a sentence of length 10 or more, this difference in probability when normalized by length would account to very small shifts in the overall model confidence score. Therefore an additional input of rank  $k$  leads to a substantial gain in performance for all models in POS.
Our task-agnostic scheme of "Rank+Var+topk" based forecasters consistently outperform or stay competitive to other forecasting methods. However, results from task-specific features such as "lm" and "len" show that use of task-specific features can further reduce the calibration error. Our domain shift experimental setup has the same set of questions in both in-domain and out-of-domain datasets. Only the data distribution for the answer passage is different. However, we do not observe an improvement in out-of-domain performance by using "lm" feature. A more detailed analysis of task-specific features in QA with both data and question shifts is required. We leave further investigations of such schemes as our future work.
Choice of k is important : The optimal choice of  $k$  seems to be strongly dependent on the inherent properties of the tasks and its output event set. In all our experiments, for a specific task all
"By the close Yorkshire had turned that into a 37-run advantage but off-spinner Such had scuttled their hopes, taking four for 24 in 48 balls and leaving them hanging on 119 for five and praying for rain."
Entity	Rank	Mean Prob	Calibrated Confidence
off-spinner Such	Rank 1	90.08± 8.65	86.32
Such	Rank 2	89.02 ± 2.07	86.61
+Rank+Var+topk forecasters exhibit consistent behaviours with respect to the choice of  $k$ . In POS experiments, heuristic-  $k = 2$ . In all other tasks, heuristic-  $k = 3$ . Our heuristic-  $k$  models are the best performing models, suggesting that the heuristic described in Section 2.5 may generalize to other tasks as well.
Re-scoring : We show that using our forecaster confidence to re-rank the entities of interest leads to a modest boost in model performance for the NER and QA tasks. In POS no appreciable gain or drop in performance was observed for  $k = 2$ . We believe this may be due to the already high token level accuracy (above  $97\%$ ) on Penn Treebank data. Nevertheless, this suggests that our re-scoring does
not lead to a degradation in model performance in cases where it is not effective.
Our forecaster re-scores the top- $k$  entity confidence scores based on model uncertainty score and entity-level features such as entity lengths. Intuitively, we want to prioritize predictions that have low uncertainty over high uncertainty predictions, if their uncalibrated confidence scores are similar. We provide an example of such re-ranking in Figure 2. It shows a named entity span predictions for the correct span "Such". The model  $p_{\theta}$  produces two entity predictions "off-spinner Such" and "Such". The un-calibrated confidence score of "off-spinner Such" is higher than "Such", but the variance of its prediction is higher as well. Therefore the  $+Rank + Var + ln + top3$  re-ranks the second (and correct) prediction higher. It is important to note here that the variance of "off-spinner Such" may be higher just because it involves two token predictions as compared to only one token prediction in "Such". This along with the "ln" feature in  $+Rank + Var + ln + top3$  may mean that the forecaster is also using length information along with uncertainty to make this prediction. However, we see similar improvements in QA tasks, where the "ln" feature is not used, and all entity predictions involve two predictions (span start and end index predictions). These results suggest that use of uncertainty features are useful in both calibration and re-ranking of predicted structured output entities.
Out-of-domain Performance: Our experiments testing the performance of calibrated QA systems on out-of-domain data suggest that our methods result in improved calibration on unseen data as well. Additionally, our methods also lead to an improvement in system accuracy on out-of-domain data, suggesting that the mapping learned by the forecaster model is not specific to a dataset. However, there is still a large gap between the calibration error for within domain and out-of-domain testing. This can be seen in the reliability plot shown in Figure 1. The number of samples in each bin are denoted by the radius of the scatter point. The calibrated models shown in the figure correspond to “+Rank+Var+lm+top3” forecaster calibrated using both in-domain and out-of-domain validation datasets for forecaster training. We see that out-of-domain forecasters are over-confident and this behaviour is not mitigated by using data-uncertainty aware features like “lm”. This is likely due to a shift in model’s prediction error when
applied to a new dataset. Re-calibration of the forecaster using a validation set from the out-of-domain data seems to bridge the gap. However, we can see that the sharpness (Kuleshov and Liang, 2015) of out-of-domain trained, in-domain calibrated model is much lower than that of in-domain trained, in-domain calibrated one. Additionally, a validation dataset is often not available in the real world. Mitigating the loss in calibration and sharpness induced by out-of-domain evaluation is an important avenue for future research.
Uncertainty Estimation : We use MC-Dropout as a model (epistemic) uncertainty estimation method in our experiments. However, our method is not specific to MC-Dropout, and is compatible with any method that can provide a predictive distribution over token level outputs. As a result any bayesian or ensemble based uncertainty estimation method (Welling and Teh, 2011; Lakshminarayanan et al., 2017; Ritter et al., 2018) can be used with our scheme. In this work, we do not investigate the use of aleatoric uncertainty for calibration. Our use of language model features is aimed at accounting for distributional uncertainty instead of aleatoric uncertainty (Gal, 2016; Malinin and Gales, 2018). Investigating the use of different types of uncertainty for calibration remains as our future work.
We show a new calibration and confidence based re-scoring scheme for structured output entities in NLP. We show that our calibration methods outperform competitive baselines on several NLP tasks. Our task-agnostic methods can provide calibrated model outputs of specific entities instead of the entire label sequence prediction. We also show that our calibration method can provide improvements to the trained model's accuracy at no additional training or data cost. Our method is compatible with modern NLP architectures like BERT. Lastly, we show that our calibration does not over-fit on in-domain data and is capable of generalizing the calibration to out-of-domain datasets.
Research reported in this publication was supported by the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health under Award Number R01HL125089.
Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33-40.
Jochen Brocker. 2009. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 135(643):1512-1519.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence modeling for neural semantic parsing. arXiv preprint arXiv:1805.04604.
Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232.
Yarin Gal. 2016. Uncertainty in deep learning. University of Cambridge, 1:3.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1321-1330. JMLR.org.
Abhyuday Jagannatha, Feifan Liu, Weisong Liu, and Hong Yu. 2019. Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (made 1.0). Drug safety, 42(1):99-111.
Volodymyr Kuleshov and Percy S Liang. 2015. Calibrated structured prediction. In Advances in Neural Information Processing Systems, pages 3474-3482.
Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. 2018. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pages 2810-2819.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pages 6402-6413.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746.
Fei Li, Yonghao Jin, Weisong Liu, Bhanu Pratap Singh Rawat, Pengshan Cai, and Hong Yu. 2019. Finetuning bidirectional encoder representations from transformers (bert)-based models on large-scale electronic health record notes: An empirical study. *JMIR Med Inform*, 7(3):e14830.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. 2019. A simple baseline for bayesian uncertainty in deep learning. arXiv preprint arXiv:1902.02476.
Andrey Malinin and Mark Gales. 2018. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pages 7047-7058.
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114-119. Association for Computational Linguistics.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Cláudio A Naranjo, Usoa Busto, Edward M Sellers, P Sandor, I Ruiz, EA Roberts, E Janecek, C Domecq, and DJ Greenblatt. 1981. A method for estimating the probability of adverse drug reactions. *Clinical Pharmacology & Therapeutics*, 30(2):239-245.
Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv:1508.05154.
Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. arXiv preprint arXiv:1809.00732.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
J Platt. 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. Advances in Large Margin Classifiers, pages 61-74.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings, volume 6.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Sarah Sarabadani. 2019. Detection of adverse drug reaction mentions in tweets using elmo. In Proceedings of the Fourth Social Media Mining for Health Applications (# SMM4H) Workshop & Shared Task, pages 120-122.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schon. 2019. Evaluating model calibration in classification. arXiv preprint arXiv:1902.06977.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Max Welling and Yee W Teh. 2011. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtopicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical concept extraction with contextual word embedding. arXiv preprint arXiv:1810.10566.
A.1 Algorithm Details:
The forecaster construction algorithm is provided in Algorithm 1. The candidate events in Algorithm 1 are obtained by extracting top- $k$  label sequences for every output. The logits obtained from  $p_{\theta}$  are averaged over 10 MC-Dropout samples before being fed into the final output layer. We use the validation dataset from the task's original split to train the forecaster. The validation dataset is used to construct both training and validation split for the forecaster. The training split contains all top- $k$  predicted entities. The validation split contains only top-1 predicted entities.
A.2 Evaluation Details
We use the expected calibration error (ECE) score defined by (Naeini et al., 2015) to evaluate our calibration methods. Expected calibration error is a score that estimates the expected absolute difference between model confidence and accuracy. This is calculated by binning the model outputs into  $N$  ( $N = 20$  for our experiments) bins and then computing the expected calibration error across all bins. It is defined as
$$
E C E = \sum_ {i = 0} ^ {N} \frac {\left| B _ {i} \right|}{n} \left| a c c \left(B _ {i}\right) - c o n f \left(B _ {i}\right) \right|, \tag {4}
$$
where  $N$  is the number of bins,  $n$  is the total number of data samples,  $B_{i}$  is the  $i^{th}$  bin. The functions  $acc(.)$  and  $conf(.)$  calculate the accuracy and model confidence for a bin.
We use AllenNLP's wrapper with HuggingFace's Transformers code for our implementation. We use BERT-base-cased (Wolf et al., 2019) weights as the initialization for general-domain datasets and bio-BERT weights (Lee et al., 2019) as the initialization for clinical datasets. We use cased models for our analysis, since bio-BERT(Lee et al., 2019) uses cased models. A common learning rate of 2e-5 was used for all experiments. We used validation data splits provided by the datasets. In cases where the validation dataset was not provided, such as MADE 1.0, EMRQA or SQuAD1.1, we use  $10\%$
of the training data as the validation data. We use a patience of 5 for early stopping the model, with each epoch consisting of 20,000 steps. We use the final evaluation metric instead of negative log likelihood (NLL) to monitor and early stop the training. This is to reduce the mis-calibration of the underlying  $p_{\theta}$  model, since Guo et al. (2017) observe that neural nets overfit on NLL. The implementation for each experiment is provided in the following subsections.
A.3.1 Part-of-speech experiments
We evaluate our method on the Penn Treebank dataset (Marcus et al., 1994). Our experiment uses the standard training (1-18), validation(19-21) and test (22-24) splits from the WSJ portion of the Penn Treebank dataset. The un-calibrated output of our model for a candidate label sequence is estimated as
$$
\hat {p} = \frac {1}{M} \sum_ {M C - D r o p o u t} p _ {\theta} \left(y _ {1}, y _ {2}, \dots y _ {L} \mid x\right) ^ {\frac {1}{L}}, \tag {5}
$$