SlowGuess/ACL_2020_OCR huggingface dataset:
Model hyperparameters and evaluation. We use BERT-BASE-UNCASED for Spouse and TACRED, and SCIBERT-SCIVOCAB-UNCASED for Disease from Beltagy et al. (2019). We finetune all our BERT models on MultiNLI using the Transformers library $^{2}$  using default parameters. The resulting BERT model is then frozen and used to produce features for our classifier. We use the following hyperparameters for our MLP classifier: number of feed-forward layers  $\in$  [0,1], dimension of each layer  $\in$  [64, 256], and dropout  $\in$  [0.0, 0.3]. We optionally project the 768 dimensional BERT feature vector down to 64 dimensions. To train our classifier, we use the Adam optimizer (Kingma and Ba, 2014) with default parameters, and batch size  $\in$  [32, 128].
We early stop our classifier based on the F1 score on the validation set, and choose the hyperparameters that obtain the best early-stopped F1 score on the validation set. For Spouse and Disease, we report the test F1 means and  $95\%$  confidence intervals of 5-10 runs. For TACRED, we follow Zhang et al. (2017), and report the test F1 of the median validation set F1 of 5 runs corresponding to the chosen hyperparameters.
Spouse and Disease preprocessed datasets were obtained directly from the codebase provided by Hancock et al. (2018) $^3$ . We use the train, validation, test split provided by Hancock et al. (2018) for Disease, and split the development set of Spouse randomly into a validation and test set (the split was done at a document level). To process TACRED, we use the default BERT tokenizer and indexing pipeline in the Transformers library.
A.3 Explanations
The explanations can be found in Tables 6 and 7 on the following page. We use 40 explanations for Spouse, 28 explanations for Disease, and 128 explanations for TACRED (in accompanying file). The explanations were written by the authors.
$^{2}$ https://huggingface.co/transformers/
<sup>3</sup>https://worksheets.codalab.org/worksheets/0x900e7e41deaa4ec5b2fe41dc50594548/
Table 6: Explanations for Spouse. The groups correspond to MARRIED, CHILDREN, ENGAGED, NEGATIVES and MISC.
Table 7: Explanations for Disease