# Generating Synthetic Training Data for Supervised De-Identification of Electronic Health Records
## 4 Results
### Privacy Findings: Was Sensitive Information Leaked into the Synthetic Records?
#### 4.3.2 User Study  
Question 1 (Information to Re-Identify a Person in Real Document)  
There was a fair agreement between participants ([PERSON]'s Kappa \(\kappa=0.279\)). The [PERSON]'s rank-order coefficient of \(\rho=0.488\) (with \(p=1.19\times 10^{-8}\)) suggests that there is a (monotonic) positive association between the ratings of both participants. In 53 of 122 cases (Figure 7), participants agreed that the real document did not provide enough information to identify a person. In cases where participants answered with either \"Probably\" or \"Yes,\" text often contained specific diagnoses (e.g., decubitus) in conjunction with PHI. Other examples were documents with specific psychological examination results (e.g., on personality, existence of suicidal thoughts, cognition, affect) or detailed descriptions of rare events (e.g., a person leaving a care home, an individual running away, descriptions of aggressive behavior). This highlights the concern that the removal of PHI in free text may not be sufficient to make it anonymous. A reader who might have been present during a described event could potentially re-identify a person without direct identifiers, if the event was unique enough. Question 2 (Information to Re-Identify a Person in Synthetic Document)  
Similarly to the inter-participator agreement for question 1, [PERSON]'s Kappa showed a fair agreement (\(\kappa=0.215\)). [PERSON]'s rank-order coefficient was \(\rho=0.4757\) (\(p=3.07\times 10^{-8}\)). The confusion matrix of participant responses in Figure 7 reveals that also for the synthetic documents shown, the contained information was often not considered person identifying. Some comments given for question 3 indicate that part of the reason may be the general incoherence of details that shows that the text is clearly fake and not about one specific person, thereby obfuscating which information is real and which PHI is related to it. For example, a text may reference several different names that do not fit together in context. This creates a privacy-protecting effect where information cannot be linked to one specific person. Furthermore, synthetic reports were often generic descriptions of days and medications without any identifiers. In cases where participants disagreed, but at least one answered with \"Probably\" or \"Yes,\" reports were generally detailed and could contain person identifiers. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Over All Real/Synthetic Pairs**} & \multicolumn{3}{c}{**Over 122 “High-Risk” Pairs**} \\ \cline{2-9}  & **Avg.** & **Median** & **Min.** & **Max.** & **Avg.** & **Median** & **Min.** & **Max.** \\ \hline ROUGE-3 recall & 0.075 & 0.067 & 0.018 & 1.000 & 0.280 & 0.217 & 0.145 & 1.000 \\ ROUGE-5 recall & 0.031 & 0.026 & 0.000 & 1.000 & 0.207 & 0.143 & 0.025 & 1.000 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Summary statistics for ROUGE-N recall over all real/synthetic document pairs and over the filtered subset of “high-risk” documents presented to participants in the user study. Figure 7: Inter-participant agreement (count of answer given) for the user study on privacy. Question 3 (Identifying a Link between Real and Synthetic Document)  
There was a slight agreement between participants (\(\kappa=0.063\) and \(\rho=0.4104\) with \(p=3\times 10^{-6}\)). In 42% of cases (51 of 122, Figure 7) both participants agreed that there was no link between the real and synthetic document. In cases where both participants agreed on the direction, but not strength of judgment and answered \"Yes\" or \"Probably,\" the additional explanations revealed three categories of how synthetic text may identify someone from the real document:  
1. **Contextual information was copied. ** For example, the synthetic and real document described similar treatment, schedule or complications, sometimes with largely identical text including medical test results. One participant pointed out that the severity of this case would depend on the uniqueness of the medical test. 2.