SlowGuess/ACL_2020_OCR huggingface dataset:
where  $M$  is the number of dropout samples. The  $L^{th}$  root accounts for different sentence lengths. Here  $L$  is the length of the sentence. We observe that this kind of normalization improves the calibration of both baselines and proposed models. We do not normalize the probabilities while reporting the ECE of uncalibrated models. We use two choice of  $p_{\theta}$  models, namely BERT and BERT+CRF. BERT only model adds a linear layer to the output of BERT network and uses a softmax activation function to produce marginal label probabilities for each token. BERT+CRF uses a CRF layer on top of unary potentials obtained from the BERT network outputs.
We use Platt Scaling (Platt, 2000) as the baseline calibration model. Our Platt scaling model uses the MC-Dropout average of length normalized probability output of the model  $p_{\theta}$  as input. The lower variance and length invariance of this input feature make Platt Scaling a very strong baseline. We also use a "Calibrated Mean" baseline using Gradient Boosted Decision Trees as our estimator with the same input feature as Platt.
A.3.2 NER Experiments
For CoNLL dataset, "testa" file was reserved for validation data and "testb" was reserved for test. For MADE 1.0 (Jagannatha et al., 2019), since validation data split was not provided we randomly selected  $10\%$  of training data as validation data.
1 https://github.com/huggingface/transformers
2The code for forecaster construction is available at https://github.com/abhyudaynj/ StructuredPredictionCalibrationNLP
Algorithm 1: Forecaster construction for model  $p_{\theta}$  with max rank  $k_{max}$ .
Input: Uncalibrated model  $p_{\theta}$ , Validation Dataset  $\mathbb{D} = \{(x^{(i)},y^{(i)}\}_{i = 0}^{\lvert\mathbb{D}\rvert},k_{max}$ .
Output: Forecaster  $F_{y}$
The length normalized marginal probability for a span starting at  $i$  and of length  $l$  is estimated as
$$
\hat {p} = \frac {1}{M} \sum_ {M C - D r o p o u t} p _ {\theta} \left(y _ {i}, y _ {2}, \dots y _ {i + l - 1} \mid x\right) ^ {\frac {1}{l}}. \tag {6}
$$
We use this as the input to both the baseline and proposed models. We observe that this kind of normalization improves the calibration of baseline and proposed models. We do not normalize the probabilities while reporting the ECE of uncalibrated models. We use BIO-tags for training. While decoding, we also allow spans that start with "I-" tag.
A.3.3 QA experiments
We use three datasets for our QA experiments, SQAUD 1.1, EMRQA and MADE 1.0. Our main aim in these experiments is to understand the behaviour of calibration and not the complexity of the tasks themselves. Therefore, we restrict the passage lengths of EMRQA and MADE 1.0 datasets to be similar to SQuAD1.1. We pre-process the passages from EMRQA to remove unannotated answer span instances and reduce the passage length to 20 sentences. EMRQA provides multiple question templates for the same question type (referred to as logical form in Pampari et al. (2018)). For each annotation, we randomly sample 3 question templates for our QA experiments. This is done to ensure that question types that have multiple question templates are not over-represented in the data.
For example, the question type for “’Does he take anything for her —problem—” has 49 available answer templates, whereas “How often does the patient take —medication—” only has one. So for each annotation, we sample 3 question templates for a question type. If the question type does not have 3 available templates, we up-sample. For more details please refer to Pampari et al. (2018).
EMRQA is a QA dataset constructed from named entity and relation annotations from clinical i2b2 datasets consisting of adverse event, medication and risk related questions (Pampari et al., 2018). We aim to also test the performance of our calibration method on out-of-domain test data. To do so, we construct a QA dataset from the clinical named entity and relation dataset MADE 1.0, using the questions and the dataset construction procedure followed in EMRQA. This allows us to have two QA datasets with common question forms, but different text distributions. This experimental setup enables us to evaluate how a QA system would perform when deployed on a new text corpus. This corresponds to the application scenario where a fixed set of questions (such as Adverse event questionnaire (Naranjo et al., 1981)) are to be answered for clinical records from different sources. Both EMRQA and MADE 1.0 are constructed from clinical documents. However, the documents themselves have different structure and language due to their different clinical sources, thereby mimicking
the real-world application scenarios of clinical QA systems.
MADE QA Construction MADE 1.0 (Jagannatha et al., 2019) is an NER and relation dataset that has similar annotation to "relations" and "medication" i2b2 datasets used in EMRQA. EMRQA uses an automated procedure to construct questions and answers from NER and relation annotations. We replicate the automated QA construction followed by Pampari et al. (2018) on MADE 1.0 dataset to obtain a corresponding QA dataset for the same. For this construction, we use question templates that use annotations that are common in both MADE 1.0 and EMRQA datasets. Examples of common questions are in Table 7. A full list of questions in MADE 1.0 QA is in "question_template.csv" file included in supplementary materials. The dataset splits for EMRQA and MADE QA are provided in Table 8.
Forecaster features Since we only consider single-span answer predictions, we require a constant number of predictions (answer start and answer end token index), for this task. Therefore we do not use the "ln" feature in this task. The uncalibrated probability of an event is normalized as follows and then used as input to all calibration models.
$$
\hat {p} = \frac {1}{M} \sum_ {M C - D r o p o u t} p _ {\theta} \left(y _ {s t a r t}, y _ {e n d} \mid x\right) ^ {1 / 2} \tag {7}
$$
Unlike the previous tasks, extractive QA with single-span output does not have a varying number of output predictions for each data sample. It always only predicts the start and end spans. Therefore using length normalized (where length is always 2) uncalibrated output does not significantly affect the calibration of baseline models. However, we use the length-normalized uncalibrated probability as our input feature to keep our base set of features consistent throughout the tasks. Additionally, in extractive QA tasks with non-contiguous spans, the number of output predictions can vary and be higher than 2. In such cases, based on our results on POS and NER, the length-normalized probability may prove to be more useful. The "Var" feature and "Rank" feature is estimated as described in previous tasks.
Input	Output	Example Question Form
Problem	Treatment	How does the patient manage her —problem—
Treatment	Problem	Why is the patient on —treatment—
Problem	Problem	Has the patient ever been diagnosed or treated for —problem—
Drug	Drug	Has patient ever been prescribed —medication—
Table 7: Examples of questions that are common in EMRQA and MADE QA datasets.
Dataset Name	Train	Validation	Test
EMRQA	74414	8870	9198
MADE QA	99496	14066	21309
Table 8: Dataset size for the MADE dataset QA pairs that were constructed using guidelines from EMRQA. EMRQA dataset splits are also provided for comparison.