# Generating Synthetic Training Data for Supervised De-Identification of Electronic Health Records
## 4 Results
### Extrinsic Evaluation: Is the Utility of Synthetic Data Sufficient for Downstream Use?
#### 4.2.1 Using Synthetic Data as a Replacement for Real Data  
We find that de-identification models trained on any of the four synthetic corpora are not as effective as the real-data baseline (Table 6). However, the results are promising. In particular, the synthetic models outperform the rule-based method DEDUCE [30] by a large margin because of a substantial increase in recall (56.4% vs. 77.3% for LSTM-temp). The rule-based method relies on domain knowledge rather than real training examples and is therefore an interesting reference when no real training data is available. Overall, we observe that the LSTM-corpora provide better utility compared to the GPT-2 corpora, both in precision and recall (Table 6). Note that this is despite our earlier finding that the LSTM-corpora are less coherent (Section 4.1.2). For a task like de-identification, it seems that syntactic correctness is more important than coherency.  
We study the influence of different PHI distributions in synthetic data by measuring precision and recall on a PHI-level (Table 7). We find that the de-identification model trained on LSTM data performs well on tags that appear frequently in the real data (e.g., Name and Date). However, the coverage of infrequent tags is insufficient (e.g., phone/fax and email). In contrast, the model trained on GPT-2 data is slightly less effective on the majority of PHI tags, but has a greater coverage of tags. We attribute this behavior to the GPT-2 p-sampling decoder, which seemingly boosted some of the rare PHI tags as discussed in Section 4.1.1. Considering the low effectiveness for identity-revealing tags, training de-identification models only on synthetic data is not yet practical. This is due to the high recall requirement for this task.  
Finally, recall from Section 3.3.3 that we set the size of the synthetic corpora to 1 million tokens for all corpora. To understand how this setting influences the effectiveness of the downstream model, we train de-identification models on subsets of the synthetic data (LSTM-p corpus). We find that the learning curve flattens when using around 70% of the training data. This indicates that generating more data will not necessarily increase effectiveness. See Appendix E for details on this experiment.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Split: Train/val/Test** & **Dataset** & **Precision** & **Recall** & **F1** \\ \hline -/-/real & NUT (rule-based) [30] & 0.807 & 0.564 & 0.664 \\ real/real/real & NUT (BiLSTM-CRF) [10] & **0.925** & 0.867 & 0.895 \\ Use case 1: synthetic data as a replacement for real data & & & \\ synth/synth/real & LSTM-p & 0.835 & 0.784 & 0.809 \\ synth/synth/real & LSTM-temp & 0.857 & 0.773 & 0.813 \\ synth/synth/real & GPT-p & 0.776 & 0.700 & 0.736 \\ synth/synth/real & GPT-beam & 0.823 & 0.688 & 0.749 \\ Use case 2: synthetic data as data augmentation method & & & \\ real+synth/real/real & NUT+LSTM-temp & 0.919\({}^{\circ}\) & **0.883\({}^{\blacktriangle}\)** & **0.901\({}^{\circ}\)** \\ real+synth/real/real & NUT+LSTM-p & 0.916\({}^{\circ}\) & 0.879\({}^{\blacktriangle}\) & 0.897\({}^{\circ}\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Summary of downstream task performance. We train on the generated synthetic data and evaluate on real data with gold-standard annotations (_NUT_ dataset [10]). Statistically significant improvements toward the _NUT_ (_BiLSTM-CRF_) baseline are marked with \({}^{\blacktriangle}\), and \({}^{\circ}\) depicts no significant difference. The test is a two-tailed approximate randomization (\(p<0.01\)).