# Generating Synthetic Training Data for Supervised De-Identification of Electronic Health Records
## 3 Materials and Methods
### Privacy Evaluation  
To gain insights into the privacy of synthetic data, we conducted a user study for a subset of synthetic documents from the corpus with highest utility in the downstream task. Our goal was to check whether any information \"leaked\" from the real data into the synthetic data, and whether this information could be used to re-identify an individual. **Finding potential worst cases for privacy. ** The assumption is that a privacy leak may have occurred when certain information of a real document reappears in a synthetic document. Similarly to the study by [PERSON] et al. [31], we have no 1-to-1 correspondence between real and synthetic records. Let \(s\in S\) be a synthetic document and \(r\in R\) be a real document. Assuming that the likelihood of a privacy leak is higher when the proximity between \(s\) and \(r\) is high, we get a set of document pairs (\(SR\)) where for each \(s\) the most similar document \(r\) is retrieved as candidate source document (cf. Figure 4). We use three measures to obtain the most similar documents to a synthetic document: ROUGE-N recall [11], with \(n=3\) and with \(n=5\), and retrieval-based BM25 scoring [12]. We use standard BM25 parameters \(b=0.75\) and \(k=1.2\)[12]. Figure 3: Overview of extrinsic evaluation procedure. We compare three settings: (1) a model trained on real data (baseline), (2) a “mixed” case, where we extend real data with synthetic data, and (3) only synthetic training data. All models were tested on real data (gold annotations). This evaluation setup extends [PERSON] et al. [15] by step (2). Instead of randomly sampling synthetic documents for manual inspection, we used several filtering steps to maximize the probability of showing pairs with high similarity and readability during evaluation: We first sorted the documents by highest ROUGE scores. Afterwards, we removed duplicates, documents longer than 1000 characters (to control the reading effort of participants), and documents that received high similarity scores mostly based on structural elements (e.g., <PAR> tokens). We took the top 122 documents with highest ROUGE score for the user study. Full details of the filtering procedure are provided in Appendix D.  
Participants were asked to answer the following questions for each pair of real/synthetic documents:  
1. \"Do you think the real doc provides enough information to identify a person?\"
2. \"Do you think the synthetic doc contains person identifying information?\"
3. \"Do you think that there is a link between the synthetic and real doc in the sense that it may identify someone in the real doc?\"
4. \"Please motivate your answer for Q3.\"  
Questions 1-3 are on a 5-point Likert scale (Yes, Probably yes, Not sure, Probably not, No), and Q4 is an open text answer. Participants received a short introduction about the task and privacy. We supplied two trial documents for participants to get used to the task. These documents were excluded from analysis. The full questionnaire and participation instructions are given in Appendix D.  
As the privacy sensitive data could not be shared with external parties, we recruited 12 participants from our institution (Nedap Healthcare). Due to the participant pool, there is a potential bias for technical and care related experts. We consider the impact for a privacy evaluation low, and indeed, because of their domain knowledge, participants have provided some helpful domain-related qualitative feedback. All participants were native Dutch speakers and each document pair was independently examined by two participants. We computed inter-participant agreement for each question with [PERSON]'s Kappa.