SlowGuess/NeurIPS_2023_OCR huggingface dataset:
their suggested parameters, with most models trained for 10,000 epochs. We note that we trained for more than the suggested epochs (50,000 instead of 10,000) and tried many additional hyperparameter settings for RGAN to attempt to improve its performance and provide the fairest comparison possible.
Figure 8: Heatmaps for GlucoSynth Across Different Privacy Budgets
Figure 9: Heatmaps for RGAN Across Different Privacy Budgets
Figure 10: Heatmaps for dpGAN Across Different Privacy Budgets
C Additional Evaluation: Fidelity
C.1 Visualizations
We provide heatmap visualizations of sample real and synthetic glucose traces from all the models. Although this is not a comprehensive way to evaluate trace quality, it does give a snapshot view about how the synthetic traces compare to the real ones. Each heatmap contains 100 randomly sampled
glucose traces. Each row is a single trace from timestep 0 to 288. The values (coloring) in each row indicate the glucose value (between  $40\mathrm{mg / dL}$  and  $400\mathrm{mg / dL}$ ). Figure 5 shows the nonprivate models, and Figures 8, 9, 10 show the private models with different privacy budgets. Upon examining the heatmaps, we notice that GlucoSynth consistently generates realistic looking glucose traces, even at very small privacy budgets.
C.2 Population Statistics
In order to evaluate fidelity on a population scale, we compute a common set of glucose metrics used to evaluate patient glycemic control on the real and synthetic data, including average trace variability (VAR), Time in Range (TIR), the percentage of time glucose is within the clinical guided range of  $70 - 180\mathrm{mg / dL}$ ; and time hypo- and hyper- glycemic (time below and above range, respectively) in Table 4. More details on each of the metrics are included in Table 3. We test if the difference in metrics between the synthetic and real data is statistically significant, using a p-value of 0.05. A p-value  $< 0.05$  indicates the difference is statistically significant. We want synthetic data that has similar population statistics to the real data: p-values  $> 0.05$  such that the differences in statistics between real and synthetic data are not significant. GlucoSynth outperforms all other models, with no statistically significant difference in all metrics for privacy budgets of  $\epsilon \geq 100$  and only one metric with a statistically significant difference for budgets  $\epsilon = 1 - 10$ .
Table 3: Glycemic Metric Explanations
Metric	Name	Explanation
VAR	Signal Variance	average trace variability
TIR	Time in Range	% of time glucose ≥ 70 & ≤ 180
Hypo	Time Hypoglycemic	% of time glucose < 70
Hyper	Time Hyperglycemic	% of time glucose > 180
GVI	Glycaemic Variability Index	more detailed measure of glucose variability
PGS	Patient Glycaemic Status	metric combining GVI and TIR
Table 4: Population Data Statistics. Each cell value for the synthetic data shows the (metric, p-value) using a 0.05 testing threshold. Bolded values do not have a statistically significant difference from the real data (what we want).
Model	ε	VAR	TIR	Hypo	Hyper	GVI	PGS
Real Data	N/A	2832.76	60.31	1.58	38.11	4.03	349.23
GlucoSynth	0.01	2575.501, 0.0	61.759, 2.0e-5	1.331, 0.0	36.91, 5.66e-4	4.002, 0.085	323.056, 0.0
0.1	2803.513, 0.356	60.088, 0.532	1.264, 0.0	38.648, 0.137	3.969, 2.74e-4	347.562, 0.712
1	2760.853, 0.022	60.597, 0.41	1.512, 0.163	37.892, 0.537	4.019, 0.577	345.159, 0.368
10	2800.805, 0.316	60.24, 0.845	1.538, 0.395	38.222, 0.76	3.963, 6.7e-5	344.376, 0.28
100	2796.424, 0.244	60.138, 0.625	1.567, 0.808	38.295, 0.609	4.044, 0.32	352.679, 0.449
∞	2811.622, 0.503	60.165, 0.682	1.54, 0.416	38.295, 0.61	4.056, 0.083	353.584, 0.339
TimeGAN	∞	2234.576, 8.08e-3	62.315, 0.42	0.657, 8.233e-3	37.028, 0.669	5.482, 0.0	503.148, 0.2e-5
FF	∞	2836.067, 0.902	46.578, 0.0	5.627, 0.0	47.795, 0.0	4.931, 0.0	528.773, 0.0
NVP	∞	1789.430, 0.0	65.499, 0.0	1.507, 0.154	32.994, 0.0	6.607, 0.0	589.473, 0.0
RGAN	0.01	56.96, 0.0	78.756, 0.0	0.0, 1.78e-4	21.244, 0.0	2.52, 0.0	93.409, 0.0
0.1	52.553, 0.0	71.617, 3.7e-5	0.0, 1.78e-4	25.715, 0.0	2.208, 0.0	98.944, 0.0
1	67.346, 0.0	78.154, 0.0	0.0, 1.78e-4	21.846, 0.0	2.251, 0.0	85.417, 0.0
10	76.632, 0.0	83.681, 0.0	0.0, 1.78e-4	16.319, 0.0	2.23, 0.0	64.562, 0.0
100	84.918, 0.0	74.285, 0.0	0.0, 1.78e-4	25.715, 0.6e-5	2.208, 0.0	98.944, 0.0
∞	89.702, 0.0	78.044, 0.0	0.0, 1.78e-4	21.956, 0.0	2.184, 0.0	82.923, 0.0
dpGAN	0.01	451.098, 0.0	95.275, 0.0	4.60, 0.0	0.124, 0.0	7.718, 0.0	41.549, 0.0
0.1	1057.205, 0.0	86.43, 0.0	0.837, 0.0	12.732, 0.0	6.349, 0.0	148.412, 0.0
1	874.663, 0.0	86.631, 0.0	1.135, 0.0	12.234, 0.0	4.794, 0.0	118.286, 0.0
10	1029.971, 0.0	88.122, 0.0	2.002, 0.0	9.876, 0.0	4.759, 0.0	93.632, 0.0
100	821.636, 0.0	89.354, 0.0	0.664, 0.0	9.982, 0.0	4.613, 0.0	82.561, 0.0
∞	1120.553, 0.0	81.773, 0.0	1.359, 0.3e-5	16.868, 0.0	6.248, 0.0	188.991, 0.0
(c) FF
Figure 11: PCA Comparison for Nonprivate Models
(d) NPV
C.3 Distributional Comparisons
We visualize differences in distributions between the real and synthetic data by plotting the distribution of variances and using PCA [27]. Figure 6 and Figure 11 show the variance distribution and PCA plots, respectively for the nonprivate models. We also compare distributional changes across privacy budgets: Figures 12 and 13 show GlucoSynth, Figures 14 and 15 show RGAN and Figures 16 and 17 show dpGAN.
Looking at the figures, GlucoSynth better captures the distribution of the real data compared to all of the nonprivate models. As evidenced in the PCA plot, (Fig. 11), FF comes the closest to capturing the real distribution in its synthetic data, but ours does a better job of representing the more rare types of traces. GlucoSynth also outperforms all of the private models across all privacy budgets. Even at small budgets  $(\epsilon < 1)$ , the general shape of the overall distribution is conserved (e.g., see Figure 12).
D Additional Evaluation: Breadth
Compared to all other models across all privacy budgets, our model has the best ratio of found validation motifs, with close to 1.0 for VM and the lowest MSEs. It also has the best coverage for nonprivate settings and an  $\epsilon$  of 100. Interestingly, dpGAN has the best coverage compared to all other models for privacy budgets  $\epsilon \leq 10$  but worse MSEs across all budgets than GlucoSynth. This means that although it finds a broader number of motifs contained in the real data, the overall distributions of motifs it creates in the synthetic data have much higher error rates. We argue that the tradeoff found by our model is better because although it does miss some of the types of motifs from the real data (misses some breadth), from the ones it does find it constructs realistic distributions of the motifs and generates very few anomalous ones.
Figure 12: GlucoSynth Distributional Variance Comparison Across Privacy Budgets
Figure 13: GlucoSynth PCA Comparison Across Privacy Budgets
Figure 14: RGAN distributional Variance Comparison Across Privacy Budgets
Figure 15: RGAN PCA Comparison Across Privacy Budgets
Figure 16: dpGAN distributional Variance Comparison Across Privacy Budgets
Figure 17: dpGAN PCA Comparison Across Privacy Budgets
(e) RGAN
(f) dpGAN
Figure 18: Clarke Error Zone Figures for All Models
E Additional Evaluation: Utility
Since RMSE may provide a limited view about the predictions from the glucose forecasting model, we also plot the Clarke Error Grid [29], which visualizes the differences between a predictive measurement and a reference measurement, and is the basis used for evaluation of the safety of diabetes-related medical devices (for example, used for evaluating glucose outputs from predictive models integrated into artificial insulin delivery systems). The Clarke Error Grid is implemented using www.github.com/suetAndTie/ClarkeErrorGrid. The grids are shown in Figure 18.
In the figures, the x-axis is the reference value and the y-axis is the prediction. A diagonal line means the predicted value is exactly the same as the reference value (the best case). There are 5 total zones that make up the grid, listed in order from best to worst:
- Zone A - Clinically Accurate: Predictions differ from actual values by no more than  $20\%$  and lead to clinically correct treatment decisions.
- Zone B – Clinically Acceptable: Predictions differ from actual values by more than  $20\%$  but would not lead to any treatment decisions.
- Zone C – Overcorrections: Acceptable glucose levels would be corrected (overcorrection).
- Zone D - Failure to Detect: Predictions lie within the acceptable range but the actual values are outside the acceptable range, resulting in a failure to detect and treat errors in glucose.
- Zone E - Erroneous Treatment: Predictions are opposite the actual values, resulting in erroneous treatment, opposite of what is clinically recommended.
We show Clarke Error grids for all models (and the private models with no privacy included,  $\epsilon = \infty$ ). This is because comparing the models at different privacy budgets is not very informative - it can be hard to tell exactly where changes between different budgets may occur. We also present a table with the percentages of predicted datapoints in each category in Table 5. This table includes a comparison among different privacy budgets for the private models (much more effective than the figures by themselves.)
Table 5: Clarke Error Grid Zones. Value is the percentage of predicted datapoints. Categories go from A to E, best to worst. Bolded rows indicate the best results on the synthetic data at each privacy budget (nonprivate models compared with private models when  $\epsilon = \infty$ )
Model	ε	A: Accurate	B: Acceptable	C: Overcorrection	D: Failure to Detect	E: Error
GlucoSynth	0.01	0.858 ± 1.057e-3	0.131 ± 1.172e-3	3.271e-3 ± 0.0	0.017 ± 1.158e-4	5.79e-6 ± 1.2e-6
0.1	0.863 ± 6.947e-3	0.126 ± 7.526e-4	3.054e-3 ± 1.45e-5	0.018 ± 4.34e-5	5.79e-6 ± 0.0
1	0.862 ± 1.578e-3	0.128 ± 1.259e-3	3.343e-3 ± 1.45e-5	0.016 ± 3.329e-4	5.79e-6 ± 0.0
10	0.864 ± 6.947e-3	0.125 ± 6.513e-4	3.039e-3 ± 5.79e-5	0.017 ± 4.34e-5	8.68e-6 ± 2.89e-5
100	0.864 ± 1.74e-3	0.126 ± 1.447e-3	3.387e-3 ± 0.0	0.017 ± 2.895e-4	5.79e-6 ± 0.0
∞	0.964 ± 1.201e-3	0.035 ± 1.158e-3	3.039e-4 ± 2.89e-5	1.732e-4 ± 1.158e-4	8.68e-6 ± 1.45e-5
TimeGAN	∞	0.741 ± 0.012	0.233 ± 0.012	2.240e-3 ± 9.8e-5	0.024 ± 8.44e-4	2.19e-4 ± 1.9e-5
FF	∞	0.824 ± 6.624e-3	0.156 ± 6.148e-3	3.547e-3 ± 9.0e-5	0.017 ± 3.940e-4	3.57e-4 ± 8.0e-6
NVP	∞	0.79 ± 3.03e-4	0.186 ± 3.87e-4	3.49e-3 ± 1.5e-5	0.02 ± 1.04e-4	3.58e-4 ± 5.0e-6
RGAN	0.01	0.54 ± 0.014	0.435 ± 0.014	3.389e-4 ± 1.197e-4	0.024 ± 2.71e-4	2.429e-4 ± 3.43e-5
0.1	0.594 ± 1.998e-3	0.38 ± 1.74e-3	1.326e-3 ± 1.429e-4	0.025 ± 1.069e-4	2.873e-4 ± 8.68e-6
1	0.637 ± 6.785e-3	0.336 ± 6.128e-3	2.661e-3 ± 1.87e-5	0.024 ± 6.464e-4	2.792e-4 ± 2.95e-5
10	0.634 ± 3.452e-3	0.338 ± 3.247e-3	2.253e-3 ± 1.004e-4	0.025 ± 2.894e-4	3.027e-4 ± 1.71e-5
100	0.638 ± 4.709e-3	0.335 ± 4.219e-3	1.991e-3 ± 2.17e-5	0.025 ± 4.884e-4	2.949e-4 ± 2.26e-5
∞	0.646 ± 6.89e-4	0.326 ± 7.19e-4	2.613e-3 ± 2.852e-4	0.024 ± 3.006e-4	2.859e-4 ± 1.5e-5
dpGAN	0.01	0.308 ± 3.482e-3	0.509 ± 3.71e-3	2.894e-7 ± 0.0	0.183 ± 2.33e-4	1.114e-5 ± 4.196e-6
0.1	0.781 ± 6.35e-4	0.191 ± 5.37e-4	3.226e-3 ± 5.715e-5	0.024 ± 3.8e-5	2.533e-4 ± 1.881e-6
1	0.786 ± 5.44e-4	0.187 ± 5.81e-4	2.409e-3 ± 2.894e-7	0.024 ± 3.6e-5	2.078e-4 ± 5.787e-7
10	0.806 ± 7.34e-4	0.169 ± 6.09e-4	2.386e-3 ± 1.476e-5	0.023 ± 1.113e-4	2.146e-4 ± 2.749e-6
100	0.813 ± 3.18e-4	0.161 ± 2.86e-4	2.266e-3 ± 2.083e-5	0.023 ± 5.4e-5	1.889e-4 ± 1.013e-6
∞	0.819 ± 1.487e-3	0.16 ± 1.306e-3	3.193e-3 ± 2.677e-5	0.018 ± 1.60e-4	3.166e-4 ± 5.208e-6
Looking at the grids, we can see that GlucoSynth performs the best, with most of the values along the diagonal axis (Zone A and B) and less around the other zones (Zones C-E) as compared to the other models. This means that most of the predicted glucose values from the model trained on our synthetic data are in the Clinically Accurate and Acceptable ranges, with less in the erroneous zones. Moreover, by examining the table we see that GlucoSynth outperforms all other models across all privacy budgets as well.