SlowGuess/NAACL_2021_OCR huggingface dataset:
, which at batch size of 128 and sequence length of 128, comes out to 40 epochs if trained for 1M steps (in the ++ models). For standard models, it comes out to 29 epochs. We used cloud TPUs (v2 and v3) to train our models. All experiments are run on a combination of V100, Titan RTX and Quadro RTX 8000 GPUs.
A.2 Condition Distribution
In Appendix Figures A1 and A2, we can see the distribution of ICD-9 and MedCAT conditions across patients. With respect to the ICD-9 codes, there are only 4 conditions that are shared across  $10,000+$  patients. This number is 32 for MedCAT conditions.
A.3 Condition Given Name
In addition to the results in Table 2, we report all Spearman coefficients, relative to the frequency of conditions (in Appendix Table 10). We additionally report results for Base++, Large++, and
Figure A2: A distribution of MedCAT codes and how many patients (of the 27K) have each condition. All bin end values are not inclusive.
Model	AUC	A@10	Spearman
ICD9
Regular Base	0.614	0.056	0.177
Regular Large	0.654	0.063	0.181
Name Insertion	0.616	0.057	0.158
Template Only	0.614	0.050	0.137
Regular Base++	0.588	0.059	0.141
Regular Large++	0.535	0.046	0.107
Regular PubmedBase++	0.583	0.055	0.160
MedCAT
Regular Base	0.529	0.109	0.175
Regular Large	0.667	0.108	0.214
Name Insertion	0.541	0.112	0.161
Template Only	0.784	0.160	0.262
Regular Base++	0.511	0.109	0.124
Regular Large++	0.469	0.098	0.152
Regular PubmedBase++	0.592	0.076	0.211
Table 10: AUC, accuracy at 10 (A@10), and Spearman coefficient relative to condition frequency.
Pubmed-Base models. With respect to AUC, these models all perform worse than the Regular Large model. Additionally, in Appendix Figure A3, we can see how experiment results change with respect to the length of conditions (owing, as we mentioned in the main text, to complications in computing likelihoods of varying length sequences under MLMs).
A.4 Condition Only
In addition to the results in Table 3, we show results for Base++, Large++, and Pubmed-Base models. Interestingly, the Large and Pubmed-Base model's perform better when names are not included. We see the biggest difference between Appendix Table 10 and 11 in the Templates Only model, suggesting that this model is memorizing the relationship between patients and conditions.
(a) ICD-9 Labels
(b) MedCAT Labels
Figure A3: Per-length performance of both ICD-9 and MedCAT labels for the 'masked conditon' (only) experiments. A bin length of  $k$  contains conditions comprising  $k$  token pieces.
Model	AUC	A@10	Spearman
ICD-9
Regular Base++	0.498	0.044	0.113
Regular Large++	0.516	0.044	0.076
Regular PubmedBase++	0.544	0.043	0.123
MedCAT
Regular Base++	0.456	0.103	0.157
Regular Large++	0.454	0.113	0.122
Regular PubmedBase++	0.628	0.080	0.213
Table 11: AUC and A@10 measures with models given only a masked out condition. We calculate spearman coefficients are given relative to the frequency baseline.
A.5 MLP Probing for Names and Conditions
In this experiment, we randomly sample 10,000 patients from our 27,906 patient set (due to computational constraints), of which we keep 5,000 for training and 5,000 for testing. For each of these patient names and every condition in our universe of conditions, we construct the previously specified template and assign it a binary label indicating whether the patient have that condition or not. Since the negative class is over-represented by a large amount in this training set, we use down-sampling to balance our data. We map each of these templates to their corresponding CLS token embedding. We use the embeddings for templates associated with training set patients to train a MLP classifier implemented in Scikit-Learn (Pedregosa et al., 2011) (Note we did not use on a validation set here). We used a hidden layer size of 128 with default hyperparameters.
At test time, for each of the 5000 patients in test set and each condition, we calculate the score using this MLP probe and compute our metrics with respect to the true label associated with that patient-condition pair.
A.6 Probing for Individual Conditions
In this experiment, we samples 50 conditions from each of the 4 frequency bins. For each condition, we trained a probe to distinguish between patients that have that condition vs those that do not. This experiment differs from the preceding fill-in-the-blank and probing experiments: Here we compute an AUC for each condition (indicating whether the probe discriminates between patients that have a particular condition and those that do not), whereas in the fill-in-the-blank experiments we computed AUCs per patient.
For probing individual conditions, we used an MLP classifier implemented in Scikit-Learn (Pedregosa et al., 2011). We did not evaluate on a validation set. We used a hidden layer size of 128 with default hyperparameters. All experiments were only run once. For the Regular BERT model, we additionally experimented with backpropagating through the BERT weights, but found that this made no difference in predictive performance.
A.7 Cosine Similarities
All versions of Skipgram and CBoW (Mikolov et al., 2013) were trained for 10 epochs usinggensim library (Rehurek and Sojka, 2010), used a vector size of 200, and a window size of 6. We only trained one variant of each W2V model. For BERT models, we used the last layer wordpiece embeddings. For word embedding models, we ran this experiment on whole reidentified patient set, whereas for BERT models, we sampled 10K patients. We report averages over the patients. In addition to the mean-pool collapsing of conditions, we also try 'Max-Pooling' and a variant we label as 'All Pairs Pooling'. We present results from all cosine-similarity experiments in Appendix Ta
Model	Mean	Std.
ICD9
Max Pooling
Regular Base	-0.0093	0.017
Regular Large	-0.020	0.029
SkipGram Base	-0.004	0.039
CBoW Base	-0.009	0.051
Name Insertion	-0.008	0.018
SkipGram Name Insertion	0.004	0.038
CBoW Name Insertion	-0.009	0.058
All Pairs Pooling
Regular Base	-0.006	0.014
Regular Large	-0.029	0.042
SkipGram Base	0.006	0.044
CBoW Base	0.005	0.044
Name Insertion	-0.001	0.013
SkipGram Name Insertion	0.019	0.039
CBoW Name Insertion	0.010	0.036
MedCAT
Max Pooling
Regular Base	-0.065	0.030
Regular Large	-0.092	0.033
SkipGram Base	-0.032	0.039
CBoW Base	-0.071	0.059
Name Insertion	-0.070	0.030
SkipGram Name Insertion	-0.021	0.035
CBoW Name Insertion	-0.087	0.059
All Pairs Pooling
Regular Base	-0.012	0.012
Regular Large	-0.043	0.028
SkipGram Base	-0.005	0.020
CBoW Base	-0.012	0.020
Name Insertion	-0.011	0.009
SkipGram Name Insertion	0.015	0.026
CBoW Name Insertion	0.004	0.024
ble 12. The mean pooling results in Table 6 seem to outperform the alternative pooling mechanisms presented here.
A.8 Probing for Names
To see if our BERT models are able to recognize the patient names that appear in training data, we train a linear probe on top of names encoded via BERT. We train this Linear Regression classifier using all default parameters from Scikit-Learn (10,000 max steps) (Pedregosa et al., 2011). We did not evaluate on a validation set. Each experiment was only run once.
Table 12: Similarity for Positive Conditions - Negative Conditions. All experiments are performed using ICD-9 codes. Max and Average refer to max-pooling and average-pooling over multiple embeddings, respectively. "All" entails the following: For every word piece in the name, find the cosine similarity for every word piece in the condition; then, use the largest cosine similarity. All word embedding models are trained for 10 epochs, with dimensionality 200.
Model	AUC
First Name
Regular Base++	0.505
Regular Large++	0.502
Regular Pubmed-base	0.501
Last Name
Regular Base++	0.504
Regular Large++	0.502
Regular Pubmed-base	0.504
Table 13: We compute the perplexity of the masked parts of names for all 46,520 patients and measure whether the (27,906) reidentified patients receive lower perplexity, compared to remaining patients.
A.9 Does observing part of a name reveal more information?
Similar to the results in Table 8, we report results on the Base++, Large++, and Pubmed-Base models (Appendix Table 13). We find no significant difference between these results and the ones reported in Table 8.