# Privacy and Utility of Private Synthetic Data for Medical Data Analyses
## 7 Discussion
### Evaluation Results  
At first, we would like to note at this point that we also carried out the experiments with Gretel Synthetics ([[https://github.com/gretelai/gretel-synthetics](https://github.com/gretelai/gretel-synthetics)]([https://github.com/gretelai/gretel-synthetics](https://github.com/gretelai/gretel-synthetics)) (accessed on 29 July 2022)) initially. We did not pursue this framework further after it became clear that it was not possible to choose a specific privacy budget \(\epsilon\). Instead, Gretel outputs the \(\epsilon\) value only after a model with certain hyperparameters was trained. Furthermore, the \(\epsilon\) values were in the three-digit range, which would not have permitted a meaningful comparison with SmartNoise. One reason that the MWEM approach performs worse than the other technologies on both the TU and TP metrics could be that the concept of MWEM was not designed for time series or continuous values. MWEM should rather be used for categorical data [36]. For our experiments, we increased the default max_bin_count of SmartNoise to 1440 (number of minutes in a day), otherwise the results were even worse. In terms of privacy, the influence of the increasing \(\epsilon\) on our TP metric seems to be limited to a specific range. This is consistent with the observations from our plain DP experiments in Section 5.2. There could be several explanations for this. One likely explanation is that noise added to the models by DP does not change the characteristics of the model (e.g., the structure of the gradient remains the same). Additionally, the results are heavily dependent on the implementation. While we believe that SmartNoise offers a well-tested implementation, implementation details could affect the quality of the results with respect to our domain-specific metrics. Another noticeable detail is that the individual with ID 4 has the largest dispersion on TP regardless of the choice of \(\epsilon\). The reason for this could be that this individual did not provide data for 24 h, which entails data with more  
Figure 7: Runtime measurements for MWEM, DP-CTGAN and PATE-CTGAN. measurement gaps than the other participants in the dataset. Furthermore, ID 4 has more tachycardia minutes compared to the other participants presented here. Thus, it could be concluded that the synthetic data generators perform worse on outlier data. When looking at the privacy values of PATE-CTGAN, there seems to be a downward trend for TP with increasing \(\epsilon\). In addition, the dispersion is also smaller for PATE-CTGAN compared to the other methods (see Figure 3). Looking at the mean TP values for small \(\epsilon\), it can be seen that PATE-CTGAN has the highest values for \(\epsilon=0.5\) compared with the other techniques. However, overall DP-CTGAN has a better TP for \(\epsilon=1\) and scores the highest TP values there compared to the other techniques. While one might conclude that DP-CTGAN performs best in terms of our TP score, it should be noted that DP-CTGAN has a wider dispersion and in some cases PATE-CTGAN achieves better privacy, particularly with \(\epsilon=0.5\). Additionally, it should be noted that MWEM shows relatively stable TP values and seems to be independent from \(\epsilon\). While this is not necessarily the preferred behavior, MWEM is also out of competition because of the reasons mentioned in the beginning of this section. In terms of the utility metric TU, DP-CTGAN performs better than the other methods up to an epsilon of 5 and can compete thereafter. For smaller values, it can be also seen that PATE-CTGAN cannot keep up and only starts to produce usable results for an \(\epsilon\) of around 6. This changes for larger \(\epsilon\) values, which also seems to be a literature consensus that PATE-CTGAN performs better for larger \(\epsilon\)[11]. The explanation for this phenomenon could be how DP is incorporated in the respective mechanism. While for DP-CTGAN noise is only applied to the weights between training epochs, PATE-CTGAN applies noise to the weights of every teacher network.