SlowGuess/NeurIPS_2023_OCR huggingface dataset:
causality matrix was not trained on data in the embedded space, we first run the set of embedded synthetic data through the recovery network  $\hat{x}_e\rightarrow \hat{x}$ . From there, the Motif Causality Loss,  $\mathcal{L}_M$ , is computed as the MSE error between the two matrices:  $\mathrm{MSE}(M,M_{\hat{x}})$ . These matrices give a causal value of seeing a motif  $\mu_{i}$  in the future after some motif  $\mu_{j}$  — unrealistic motif sequences will have causal values close to 0. As the generator learns to generate synthetic data that yields a realistic causal matrix (thereby identifying appropriate causal relationships from the motifs), it implicitly learns to not generate unrealistic motif sequences.
3 - Distributional. To guide the generator to produce a diverse set of traces, the generator computes a Distributional Loss,  $\mathcal{L}_D$ , the moments loss (MML), between the overall distribution of the real data  $x_{e}$  and the distribution of the synthetic data  $\hat{x}_{e}$ :  $\mathrm{MML}(x_e,\hat{x}_e)$ . The MML is the difference in the mean and variance of two matrices.
Discriminator. The discriminator is a traditional discriminator model using an RNN, the only change being it also works in the embedded space. The discriminator yields the Adversarial Loss Real,  $\mathcal{L}_{Ar}$ , the Binary Cross Entropy (BCE) between the discriminator guesses on the real data  $y_{x_e}$  and the ground truth  $y$ , a vector of 0's,  $\mathrm{BCE}(y_{x_e}, y)$  and the Adversarial Loss Fake,  $\mathcal{L}_{Af}$ , the BCE between the discriminator guesses on the fake data  $y_{\hat{x}_e}$  and the ground truth  $y$ , a vector of 1's,  $\mathrm{BCE}(y_{\hat{x}_e}, y)$ .
5.2 Training Procedure
First, the motif causality block is trained following the procedure described in Section 4.2, and then the rest of the GAN is trained. The autoencoder is optimized to minimize  $\mathcal{L}_R + \alpha \mathcal{L}_S$ , where  $\alpha$  is a hyperparameter that balances the two loss functions. If the AE only receives  $\mathcal{L}_R$  (as is typically done), it becomes overspecialized, i.e., it becomes too good at learning the best lower dimensional representation of the data such that the embedded data are no longer helpful to the generator. For this reason, the AE also receives  $\mathcal{L}_S$ , enabling the dual training of the generator and embedder. The generator is optimized using  $\min(1 - \mathcal{L}_{Af}) + \eta(\mathcal{L}_S + \mathcal{L}_D) + \mathcal{L}_M$ , where  $\eta$  is a hyperparameter that balances the effect of the stepwise and distributional loss. Finally the discriminator is optimized using the traditional adversarial feedback  $\min(\mathcal{L}_{Af} + \mathcal{L}_{Ar}$ . The networks are trained in sequence (within each epoch) in the following order: autoencoder, generator, then discriminator. In our experiments we set  $\alpha = 0.1$  and  $\eta = 10$  as they enable GlucoSynth to converge fastest, i.e., in the fewest epochs.
6 Providing Differential Privacy
There are two components to our privacy architecture, described in the following two subsections: (1) each network in the GAN (Embedder, Recovery, Generator and Discriminator networks) is trained in a differentially private manner using the Differentially-Private Stochastic Gradient Descent (DP-SGD) algorithm from Abadi et al. [21]; and (2) the motif causality block is trained using the PATE framework from Papernot et al. [22]. Importantly, two completely separate datasets are used for the training of the motif causality block (dataset B in Figure 4) and the GAN (dataset A in Figure 4). We structure the privacy integration in this way to allow for better privacy-utility trade-offs. Our design satisfies the formal differential privacy notion introduced by Dwork et al. [23]. Differential Privacy (DP) provides an intuitive bound on the amount of information that can be learned about any individual in a dataset. A randomized algorithm  $\mathcal{M}$  satisfies  $(\epsilon, \delta)$ -differential privacy if, for all datasets  $D_{1}$  and  $D_{2}$  differing by at most a single unit, and all  $S \subseteq \operatorname{Range}(\mathcal{M})$ ,  $Pr[\mathcal{M}(D_1) \in S] \leq e^\epsilon Pr[\mathcal{M}(D_2) \in S] + \delta$ . The parameters  $\epsilon$  and  $\delta$  determine the privacy loss budget, which provide a way to tradeoff privacy and utility; smaller values have stronger privacy. Importantly, privacy is provisioned at the trace level, and we assume each individual has only one trace in the dataset.
6.1 Training the GAN Networks with DP
To add privacy to the GAN components, each of the networks (Embedder, Recovery, Generator and Discriminator) is trained in a differentially private manner using DP-SGD [21]. Although the overall GAN framework is complicated, the individual networks all use simple RNN or LSTM architectures with Adam optimizers. As such, adding DP noise to their network weights is straightforward. We employ the following procedure using Tensorflow Privacy functions [24]. Since there are four
networks being trained with DP, we divide the privacy loss budget evenly to get the budget per network,  $\epsilon_{net} = \epsilon /4$ . Then, we use Tensorflow's built-in DP accountant to determine how much noise must be added to the weights of each network based on the number of epochs, batch size, number of traces and  $\epsilon_{net}$ . This function returns a noise multiplier, which we use when we instantiate a Tensorflow DP Keras Adam Optimizer for each network. Finally, we train each of the networks using their respective DP Keras Adam Optimizer, which automatically trains the network using DP-SGD.
6.2 Training the Motif Causality Block with DP
We train the motif causality block using the PATE framework [22]. PATE provides a way to return aggregated votes about the class a data point belongs to. First, the data is partitioned into  $r$  partitions, where  $r$  is determined based on the size of the dataset and the privacy loss budget. Then, a class membership model is trained independently for each partition. The class membership votes from each partition are aggregated by adding noise to the vote matrix and the noisiest votes are returned using the max-of-Laplacian mechanism (LNMax), tuned based on the privacy budget and  $r$ .
We use PATE to train the motif causality block: instead of predicting the degree of class membership we predict causal membership, e.g., does motif  $\mu_{i}$  have a causal relationship to  $\mu_{j}$ . The motif causality block is trained in the same procedure described in Section 4.2 with two changes: (1) the number of data partitions,  $r$ , is determined based on the privacy budget, instead of a user-specified value; (2) the final causality matrix  $M$  is aggregated using DP across the partitions. In normal PATE, carefully calibrated noise is added to a matrix of votes for each class, such that the classes with the noisiest votes are outputted. In our use, each value in a motif causality matrix may be likened to a class (i.e., causal "class" prediction between motif  $\mu_{i}$  and  $\mu_{j}$ ). Thus, we use the LNMax mechanism (from predefined Tensorflow Privacy functions [24]) to aggregate the matrices weights and return  $M$ .
We use PATE instead of training each motif network using DP-SGD for better privacy-utility trade-offs. With DP-SGD, we would need to add noise to every motif net, eating up our privacy budget quickly and severely impacting the quality of the returned casualty matrices. PATE allows us to train each of the motif networks without any noise on the gradients, but then aggregates their returned causality matrices in a privacy-preserving manner, resulting in a better privacy-utility trade-off.
7 Evaluation
Evaluating synthetic data is notoriously difficult [25], so we provide an extensive evaluation across three criteria. Synthetic data should: 1) conserve characteristics of the real data (fidelity, Section 7.1); 2) contain diverse patterns from the real data without the introduction of anomalous patterns (breadth, Section 7.2); and 3) be usable in place of the original for real-world use cases (utility, Section 7.3).
Data and Benchmarks. We use 100,000 single-day glucose traces randomly sampled across each month from January to December 2022, for a total of 1.2 million traces, collected from Dexcom's G6 Continuous Glucose Monitors (CGMs) [26]. Data was recorded every 5 minutes  $(T = 288)$  and each trace was aligned temporally from 00:00 to 23:59. We restrict our comparison to the five most closely related state-of-the-art models for generating synthetic univariate time series with no labels or auxiliary data: Three nonprivate—TimeGAN [4], Fourier Flows (FF) [12], non-volume preserving transformations (NVP) [9]; and two private—RGAN [11] and dpGAN [14]. We refer the reader to Appendix B for additional experimental details and all hyperparameter settings, including reasoning behind the choice of motif size  $\tau = 48$ .
7.1 Fidelity
Visualization. We provide visualizations of sample real and synthetic glucose traces from all models. Although this is not a comprehensive way to evaluate trace quality, it does give a snapshot view about what synthetic traces may look like. We provide heatmap visualizations, where each heatmap contains 100 randomly sampled glucose traces. Each row is a single trace from timestep 0 to 288. The values in each row indicate the glucose value (between  $40\mathrm{mg / dL}$  and  $400\mathrm{mg / dL}$ ). Figure 5 shows the nonprivate models, and Figures 8, 9, 10 in Appendix C.1 show the private models with different privacy budgets. Upon examining the heatmaps, we notice that GlucoSynth consistently generates realistic looking glucose traces, even at very small privacy budgets.
Figure 5: Heatmaps for Nonprivate Models
Population Statistics. To evaluate fidelity on a population scale, we compute a common set of glucose metrics and test if the difference between the synthetic and real data is statistically significant. Table 1 provides an abbreviated summary of the results; Appendix C.2 has complete results. GlucoSynth performs the best, with few statistical differences between the real and synthetic data for  $\epsilon \geq 0.1$ .
Distributional Comparisons. We visualize differences in distributions between the real and synthetic data by plotting the distribution of variances and using PCA [27]. Figure 6 shows the variance distribution for the nonprivate models. Additional comparisons across privacy budgets are available in Appendix C.3. In both nonprivate and private settings, GlucoSynth produces synthetic distributions closest to the real ones, better than all other models.
7.2 Breadth
We quantify breadth in terms of glucose motifs. For each model's synthetic traces, we build a motif set (see Section 3.1). Given a real motif set from the validation traces  $S_{x}$ , for each synthetic motif set  $S_{\hat{x}}$ , we compute "Validation Motifs", (VM), the fraction of motifs found in the validation motif set that are present in the synthetic motif set,  $\mathrm{VM} / |S_{\hat{x}}|$ . This metric quantifies how good our synthetic motif set is (e.g., are its motifs mostly similar to motifs found in real traces). We also compute metrics related to coverage, the fraction of motifs in the validation motif set that are found in our synthetic data, defined as  $\mathrm{VM} / |S_{x}|$ . This gives a sense of the breadth in a more traditional manner. To compare actual distributions of motifs (not just counts), we compute the MSE between the distribution of real motifs  $S_{x}$  and the distribution of synthetic motifs  $S_{\hat{x}}$ . This gives a measure about how close the synthetic motif distribution is to the real one. We want high VM and coverage, and low MSE. Results are in Table 1 with additional analysis in Appendix D; overall our model provides the best breadth.
7.3 Utility
We evaluate our synthetic glucose traces for use in a glucose forecasting task using the common paradigm TSTR (Train on Synthetic, Test on Real), in which the synthetic data is used to train the model and then tested on the real validation data. We train an LSTM network optimized for glucose forecasting tasks [28] and report the Root Mean Square Error (RMSE) in Table 1. We run the experiment 10 times and train the LSTM for 10,000 epochs. We have also tested with other models including RNNs, attention-based models and other LSTM architectures (such as bidirectional LSTMs) but show the results for the best performing model, the LSTM optimized for glucose forecasting. Since RMSE provides a limited view about the model's predictions, we also plot the Clarke Error Grid [29], which visualizes the differences between a predictive and reference measurement, and is a basis for evaluating the safety of diabetes-related medical devices. More details are in Appendix E. GlucoSynth provides the best forecasting results compared to all other models across all privacy budgets.
Table 1: Fidelity, Breadth and Utility Evaluation. Fidelity: bolded values do not have a statistically significant difference from the real data (what we want). Breadth and Utility: VM = fraction found validation motifs; We want high VM, Coverage and low MSE, RMSE; Bolded values indicate the best ones at each privacy budget (nonprivate compared with private models when  $\epsilon = \infty$ ).
Model	ε	Fidelity (metric, p-val)	VM	Breadth Coverage	MSE	Utility RMSE
Variance	Time-in-Range
GlucoSynth	0.01	2576, <1e-5	61.8, 2e-5	1.000	0.010	99.0	0.038 ± 3e-4
0.1	2809, 0.356	60.1, 0.532	1.000	0.083	11.2	0.036 ± 3e-4
1	2761, 0.022	60.6, 0.410	0.992	0.145	6.7	0.030 ± 1e-4
10	2801, 0.316	60.2, 0.845	1.000	0.167	5.0	0.029 ± 1e-4
∞	2812, 0.503	60.2, 0.682	0.987	0.534	1.6	7e-3 ± 2e-4
TimeGAN	∞	2235, 8e-3	62.3, 0.420	0.625	6e-3	107.7	0.061 ± 3e-4
FF	∞	2836, 0.902	46.6, <1e-5	0.642	0.405	2.0	0.038 ± 3e-4
NVP	∞	1789, <1e-5	65.5, <1e-5	0.482	0.328	1.9	0.029 ± 3e-5
RGAN	0.01	57, <1e-5	78.8, <1e-5	0.013	1e-3	108.6	0.819 ± 0.010
0.1	53, <1e-5	71.6, 3e-5	0.015	0.031	107.3	0.688 ± 6e-3
1	67, <1e-5	78.2, <1e-5	0.015	0.033	103.3	0.651 ± 0.018
10	77, <1e-5	83.7, <1e-5	0.017	0.053	100.3	0.619 ± 0.016
∞	90, <1e-5	78.0, <1e-5	0.026	0.091	79.6	0.460 ± 0.013
dpGAN	0.01	451, <1e-5	95.3, <1e-5	0.094	0.054	180.1	0.205 ± 5e-3
0.1	1057, <1e-5	86.4, <1e-5	0.390	0.195	28.9	0.045 ± 2e-4
1	875, <1e-5	86.6, <1e-5	0.480	0.239	23.2	0.030 ± 2e-5
10	1030, <1e-5	88.1, <1e-5	0.743	0.251	16.1	0.035 ± 8e-5
∞	1121, <1e-5	81.8, <1e-5	0.855	0.293	10.9	0.028 ± 5e-5
(a) GlucoSynth
(b) TimeGAN
(c) FF
(d) NPV
Figure 6: Distributional Variance for Nonprivate Models
8 Limitations & Conclusion
Limitations. In order to train on a huge set of glucose traces, we used a private dataset, not publicly available (one of the motivations for this project was actually to share a synthetic version of these traces). That being said, smaller samples of glucose traces with similar patient populations are available at OpenHumans [30] and T1D Exchange Registry [31]. In addition, one of the reasons our privacy results perform well is because we use two separate datasets for the training of the motif causality block and the GAN. However, this may be a limiting factor for others that do not have a large enough set of traces available to be able to train adequately on partitioned data.
Conclusion. In this paper we have presented GlucoSynth, a novel GAN framework with integrated differential privacy to generate synthetic glucose traces. GlucoSynth conserves motif relationships within the traces, in addition to the typical temporal dynamics contained within time series. We presented a comprehensive evaluation using 1.2 million glucose traces wherein our model outperformed all previous models across three criteria of fidelity, breadth and utility.
This work was supported in part by the U.S. National Science Foundation under Grant CCF-1942836, CNS-2213700, CNS-2220433, OAC-2319988 and by a Graduate Research Fellowship under Grant No. 1842490. This work was also supported by a Dexcom Graduate Fellowship.
[37] Debapriya Hazra and Yung-Cheol Byun. SynSigGAN: Generative Adversarial Networks for synthetic biomedical signal generation. Biology, 9(12):441, 2020.
[38] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In International Conference on Learning Representations, 2018.
We overview related work in three lines of research: time series, conditional time series, and time series methods that employ differential privacy. Table 2 summarizes previous time series synthesis methods. We note that there have been exciting developments for adjacent research tasks (data augmentation, forecasting) such as diffusion models [32], but there are not yet any publicly available models specifically for the generation of complete synthetic time series datasets. As such, we focus the scope of our comparison on the current state-of-the-art methods for synthetic time series which all build upon Generative Adversarial Networks (GANs) [3] and transformation-based approaches [9]. In particular TimeGAN [4], RGAN [11] and dpGAN [14] are most similar to ours and used as benchmarks in the evaluation in Section 7.
Table 2: Summary of Previous Methods for Time Series Synthesis. *CI = conditional information or extra features
Name	Private?	No Labels Required?	No CI*?	Length
TimeGAN [4]	X	✓	✓	24 - 58
TTS-GAN [5]	X	X	✓	24 - 150
SigCWGAN [33]	X	✓	X	80,000
RGAN [11]	✓	✓	✓	16 - 30
RCGAN [11]	✓	✓	X	16 - 30
dpGAN [14]	✓	✓	✓	96
RDP-CGAN [6]	✓	✓	X	2 - 4097
DoppelGANger [34]	✓	✓	X	50 - 600
GlucoSynth (Ours)	✓	✓	✓	288
Time Series. There have been promising models to generate synthetic time series across a variety of domains such as financial data [35], cyber-physical systems (e.g., smart homes [36]), and medical signals [37]. Brophy et al. [10] provides a survey of GANs for time series synthesis. TimeGan [4] is a popular benchmark that jointly learns an embedding space using supervised and adversarial objectives in order to capture the temporal dynamics amongst traces. TTS-GAN [5], trains a GAN model that uses a transformer encoding architecture in order to best preserve temporal dynamics. Transformation-based approaches have also had success for time series data. Real-valued non-volume preserving transformations (NVP) [9] model the underlying distribution of the real data using generative probabilistic modeling and use this model to output a set of synthetic data. Similarly, Fourier Flows (FF) [12] transform input traces into the frequency domain and output a set of synthetic data from the learned spectral representation of the original data. Methods that only focus on learning the temporal or distributional dynamics in time series are not sufficient for generating realistic synthetic glucose traces due to the lack of temporal dependence within sequences of glucose motifs.
Conditional Time Series. Many works have developed time series models that supplement their training using extra features or conditional data. Esteban et al. [11] develops two GAN models (RGAN/RCGAN) with RNN architectures, conditioned on auxiliary information provided at each timestep during training. SigCWGAN [33] uses a mathematical conditional metric  $(Sig - W_{1})$  characterizing the signature of a path to capture temporal dependence of joint probability distributions in long time series data. However, our glucose traces do not have any additional information available so these methods cannot be used<sup>1</sup>.
Differentially-Private GANs. To protect sensitive data, several GAN architectures (DP GANs) have been designed to incorporate privacy-preserving noise needed to satisfy differential privacy guarantees [13]. Although DP GANs such as PateGAN [38] have had great success for other data types and learning tasks (e.g., tabular data, supervised classification tasks), results have been less satisfactory in DP GANs developed for time series.
RGAN/RCGAN [11] also includes a DP implementation, but the authors find large gaps in performance between the nonprivate and private models. Frigerio et al. [14] extends a simple DP GAN architecture (denoted dpGAN) to time-series data. The synthetic data from their private model
There is a caveat here that RGAN does not use auxiliary information, hence why we compare with it in our benchmarks.
Figure 7: Example motif causality matrix for a small motif set  $(m = 10)$ . Each value in the grid is between 0 and 1. 0 indicates no motif-causal relationship, and 1 indicates the strongest motif causal relationship.
conserves the distribution of the real data but loses some of the variability (diversity) from the original samples. RDP-CGAN [6] develops a convolutional GAN architecture that uses Rényi differential privacy specifically for medical data. Across different datasets, they find that reasonable privacy budgets result in major drops in the performance of the synthetic data. Finally, DoppelGAN [34] develops a temporal GAN framework for time series with metadata and perform an in-depth privacy evaluation. Notably, they find that providing strong theoretical DP guarantees results in destroying the fidelity of the synthetic data, beyond anything feasible for use in real-world scenarios. Each of these methods touches on the innate challenge of generating DP synthetic time series due to very high tradeoffs between utility and privacy. Our DP framework uses two different methods to integrate privacy into our GAN architecture, resulting in a better utility-privacy trade-off than previous methods.
Note on Data Use. As explained in the approach (Section 5), our model uses two separate datasets for the training of the motif causality block and the rest of the GAN. As such, we used two different samples of glucose traces with no overlap between patients for the training of each section (meaning we actually used a total of 2.4 million traces across the entire model). We also note that we have received the proper ethical and legal consent from the individuals to use their data in this way (and for this purpose).
Hyperparameters. Our experiments were completed in the Google Cloud platform on an Intel Skylake 96-core cpu with 360 GB of memory. We use a separate validation dataset (not the set of original training traces) for all experimental results. Throughout all our experiments we use GlucoSynth model parameters of  $\alpha = 0.1$  and  $\eta = 10$  and a motif tolerance of  $\sigma = 2\mathrm{mg / dL}$  and motif length  $\tau = 48$ . Motif length of 48 timesteps is equivalent to 4 hours of time and represents a clinically significant threshold. This threshold was chosen because the effect of any behaviors on glucose occur within 4 hours of the event (e.g., the effect from eating a meal - a rise in glucose - will occur within 4 hours after eating.) We note that other choices for  $\tau$  could be used, based on what types of phenomenon the users wish to replicate; for example, to capture day/night glucose rhythm effects, we suggest a  $\tau$  of 144, corresponding to 12 hours of time.
We vary  $\epsilon$  in our privacy experiments, but keep  $\delta$  the same at  $5e - 4$ . Importantly, in order to meet our privacy guarantees, we assume that privacy is provisioned at the trace level and each individual has only one trace in the dataset. The motif set is derived separately from the training data (either from a public dataset or generated based on knowledge about the underlying data, e.g., the possible glucose motif combinations), so as not to effect the differential privacy guarantees or use up any privacy budget. In our case, we assume the motif set is all-encompassing and generated from the universe of possible motifs, resulting in  $m = 5,977,610$  total motifs in the motif set.
Benchmark Details. TimeGAN [4] is implemented from www.github.com/jsyoon0823/TimeGAN; Fourier Flows (FF) [12] are implemented from www.github.com/ahmedmalaa/Fourier-flows; RGAN [11] is implemented from www.github.com/ratschlab/RGAN; and DPGAN [14] is adapted from www.github.com/SAP-samples/security-research-differentially-private-generative-models. All the benchmarks were trained according to