SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Josephine Lamp<sup>1,2</sup> Mark Derdzinski<sup>2</sup> Christopher Hannemann<sup>2</sup> Joost van der Linden<sup>2</sup> Lu Feng<sup>1</sup> Tianhao Wang<sup>1</sup> David Evans<sup>1</sup>
<sup>1</sup>University of Virginia, Charlottesville, VA, USA; <sup>2</sup>Dexcom, USA
jl4rj@virginia.edu; {mark.derdzinski; christopher.hannemann;
joost.vanderlinden}@dexcom.com; {lu.feng; tianhao; evans}@virginia.edu
We focus on the problem of generating high-quality, private synthetic glucose traces, a task generalizable to many other time series sources. Existing methods for time series data synthesis, such as those using Generative Adversarial Networks (GANs), are not able to capture the innate characteristics of glucose data and cannot provide any formal privacy guarantees without severely degrading the utility of the synthetic data. In this paper we present GlucoSynth, a novel privacy-preserving GAN framework to generate synthetic glucose traces. The core intuition behind our approach is to conserve relationships amongst motifs (glucose events) within the traces, in addition to temporal dynamics. Our framework incorporates differential privacy mechanisms to provide strong formal privacy guarantees. We provide a comprehensive evaluation on the real-world utility of the data using 1.2 million glucose traces; GlucoSynth outperforms all previous methods in its ability to generate high-quality synthetic glucose traces with strong privacy guarantees.
The sharing of medical time series data can facilitate therapy development. As a motivating example, sharing glucose traces can contribute to the understanding of diabetes disease mechanisms and the development of artificial insulin delivery systems that improve people with diabetes' quality of life. Unsurprisingly, there are serious legal and privacy concerns (e.g., HIPAA, GDPR) with the sharing of such granular, longitudinal time series data in a medical context [1]. One solution is to generate a set of synthetic traces from the original traces. In this way, the synthetic data may be shared publicly in place of the real ones with significantly reduced privacy and legal concerns.
This paper focuses on the problem of generating high-quality, privacy-preserving synthetic glucose traces, a task which generalizes to other time series sources and application domains, including activity sequences, inpatient events, hormone traces and cyber-physical systems. Specifically, we focus on long (over 200 timesteps), bounded, univariate time series glucose traces. We assume that available data does not have any labels or extra information including features or metadata, which is quite common, especially in diabetes. Continuous Glucose Monitors (CGMs) easily and automatically send glucose measurements taken subcutaneously at fixed intervals (e.g., every 5 minutes) to data storage facilities, but tracking other sources of diabetes-related data is challenging [2]. We characterize the quality of the generated traces based on three criteria—synthetic traces should (1) conserve characteristics of the real data, i.e., glucose dynamics and control-related metrics (fidelity); (2) contain representation of diverse types of realistic traces, without the introduction of anomalous patterns that do not occur in real traces (breadth); and (3) be usable in place of the original data for real-world use cases (utility).
Figure 1: Example Real Glucose Traces and Glucose Motifs from our Dataset.
Generative Adversarial Networks (GANs) [3] have shown promise in the generation of time series data. However, previous methods for time series synthesis, e.g., [4, 5, 6], suffer from one or more of the following issues when applied to glucose traces: 1) surprisingly, they do not generate realistic synthetic glucose traces – in particular, they produce human physiologically impossible phenomenon in the traces; 2) they require additional information (features, metadata or labels) to guide the model learning which are not available for our traces; 3) they do not include any privacy guarantees, or, in order to uphold a strong formal privacy guarantee, severely degrade the utility of the synthetic data.
Generating high-quality synthetic glucose traces is a difficult task due to the innate characteristics of glucose data. Glucose traces can be best understood as sequences of events, which we call motifs, shown in Figure 1, and they are more event-driven than many other types of time series. As such, a current glucose value may be more influenced by an event that occurred in the far past compared to values from immediate previous timesteps. For example, a large meal eaten earlier in the day (30-90 minutes ago) may influence a patient's glucose more than the glucose values from the past 15 minutes. As a result, although there is some degree of temporal dependence within the traces, only conserving the immediate temporal relationships amongst values at previous timesteps does not adequately capture the dynamics of this type of data. In particular, we find that the main reason previous methods fail is because they may not sufficiently learn event-related characteristics of glucose traces.
Contributions. We present GlucoSynth, a privacy-preserving GAN framework to generate synthetic glucose traces. The core intuition behind our approach is to conserve relationships amongst motifs (events) within the traces, in addition to the typical temporal dynamics contained within time series. We formalize the concept of motifs and define a notion of motif causality, inspired from Granger causality [7], which characterizes relationships amongst sequences of motifs within time series traces (Section 4). We define a local motif loss to first train a motif causality block that learns the motif causal relationships amongst the sequences of motifs in the real traces. The block outputs a motif causality matrix, that quantifies the causal value of seeing one particular motif after some other motif. Unrealistic motif sequences (such as a peak to an immediate drop in glucose values) will have causal relationships close to 0 in the causality matrix. We build a novel GAN framework that is trained to optimize motif causality within the traces in addition to temporal dynamics and distributional characteristics of the data (Section 5). Explicitly, the generator computes a motif causality matrix from each batch of synthetic data it generates, and compares it with the real causality matrix. As such, as the generator learns to generate synthetic data that yields a realistic causal matrix (thereby identifying appropriate causal relationships from the motifs), it implicitly learns not to generate unrealistic motif sequences. We also integrate differential privacy (DP) [8] into the framework (Section 6), which provides an intuitive bound on how much information may be disclosed about any individual in the dataset, allowing the GlucoSynth model to be trained with privacy guarantees. Finally, in Section 7, we present a comprehensive evaluation using 1.2 million glucose traces from individuals with diabetes collected across 2022, showcasing the suitability of our model to outperform all previous models and generate high-quality synthetic glucose traces with strong privacy guarantees.
We focus the scope of our comparison on current state-of-the-art methods for synthetic time series which all build upon Generative Adversarial Networks (GANs) [3] and transformation-based approaches [9]. An extended related work is in Appendix A.
(a) Glucose Motif 1
(b) Glucose Motif 2
(c) Temporal Motif 1
Figure 2: Temporal Distributions of Sample Motifs. Each radial graph displays the temporal distribution of a motif; there are 24 radial bars from 00:00 to 23:00, and each segment displays the  $\%$  of motif occurrences by each hour. Glucose motifs 1 and 2 are from Fig. 1; they are not temporally-dependent and show up across the day. Temporal motifs 1 and 2 are from a cardiology dataset [15].
(d) Temporal Motif 2
Time Series. Brophy et al. [10] provides a survey of GANs for time series synthesis. TimeGan [4] is a popular benchmark that jointly learns an embedding space using supervised and adversarial objectives in order to capture the temporal dynamics amongst traces. Esteban et al. [11] develops two time series GAN models (RGAN/RCGAN) with RNN architectures, conditioned on auxiliary information provided at each timestep during training. TTS-GAN [5] trains a GAN model that uses a transformer encoding architecture in order to best preserve temporal dynamics. Transformation-based approaches such as real-valued non-volume preserving transformations (NVP) [9] and Fourier Flows (FF) [12], have also had success for time series data. These methods model the underlying distribution of the real data to transform the input traces into a synthetic data set. Methods that only focus on learning the temporal or distributional dynamics in time series are not sufficient for generating realistic synthetic glucose traces due to the lack of temporal dependence within sequences of glucose motifs.
Differentially-Private GANs. To protect sensitive data, several GAN architectures have been designed to incorporate privacy-preserving noise needed to satisfy differential privacy guarantees [13]. Frigerio et al. [14] extends a simple differentially-private architecture (dpGAN) to time-series data and RDP-CGAN [6] develops a convolutional GAN architecture specifically for medical data. These methods find large gaps in performance between the non-private and private models. Providing strong theoretical DP guarantees using these methods often results in synthetic data with too little fidelity for use in real-world scenarios. Our framework carefully integrates DP into the motif causality block and each network of the GAN, resulting in a better utility-privacy tradeoff than previous methods.
3.1 Motifs
Glucose (and many other) traces can be best understood as sequences of events or motifs. Motifs characterize phenomenon in the traces, such as peaks or troughs. We define a motif,  $\mu$ , as a short, ordered sequence of values  $(v)$  of specified length  $\tau$ ,  $\mu = [v_{i}, v_{i+1}, \ldots, v_{i+\tau}]$  and  $\sigma$  is a tolerance value to allow approximate matching (within  $\sigma$  for each value). Some examples of glucose traces and motifs are shown in Figure 1. We denote a set of  $n$  time series traces as  $X = [x_{1}, \ldots, x_{n}]$ . Each time series may be represented as a sequence of motifs:  $x_{i} = [\mu_{i_{1}}, \mu_{i_{2}} \ldots]$  where each  $i_{j}$  gives the index of the motif in the set that matches  $x_{i_{j-\tau}}, \ldots, x_{i_{(j+1)\cdot\tau-1}}$ . Given the motif length  $\tau$ , the motif set is the union of all size- $\tau$  chunks in the traces. This definition is chosen for a straightforward implementation but motifs can be generated in other ways, such as through the use of rolling windows or signal processing techniques [16, 17]. Motifs are pulled from the data such that there is always a match from a trace motif to a motif from the set (if multiple matches, the closest one is chosen).
3.2 Glucose Dynamics (Why Standard Approaches Fail)
We first present a study of the characteristics of glucose data in order to motivate the development of our framework. Although there are general patterns in sequences of glucose motifs (e.g., motif patterns corresponding to patients that eat 2x vs. 3x a day), individual glucose motifs are typically not time-dependent, as illustrated in Figure 2. The radial graphs display the temporal distribution of
the first two glucose motifs from Figure 1 and two temporally-dependent motifs from a cardiology dataset [15]. There are 24 radial bars from 00:00 to 23:00 for each hour of the day, and the bar value is the percentage of total motif occurrences at that hour across the entire dataset (i.e., value of 10 would indicate that  $10\%$  of the time that motif occurs during that hour). Note that the glucose motifs show up fairly evenly across all hours of the day whereas the motifs from the cardiology dataset have shifts in their distribution and show up frequently at specific hours of the day. The lack of temporal dependence in glucose motifs is likely due to the diverse patient behaviors within a patient population. Glucose in particular is highly variable and influenced by many factors including eating, exercise, stress levels, and sleep patterns. Moreover, due to innate variability within human physiology, motif occurrences can differ even for the same patient across weeks or months. These findings indicate that only conserving the temporal relationships within glucose traces (as many previous methods do) may not be sufficient to properly learn glucose dynamics and output realistic synthetic traces.
3.3 Granger Causality
Granger causality [7] is commonly used to quantify relationships amongst time series without limiting the degree to which temporal relationships may be understood as done in other time series models, e.g., pure autoregressive ones. In this framework, an entire system (set of traces) is studied together, allowing for a broader characterization of their relationships, which may be advantageous, especially for long time series. We define  $x_{t} \in \mathbb{R}^{n}$  as an  $n$ -dimensional vector of time series observed across  $n$  traces and  $T$  timesteps. To study causality, a vector autoregressive model (VAR) [18] may be used. A set of traces at time  $t$  is represented as a linear combination of the previous  $K$  lags in the series:  $x_{t} = \sum_{k=1}^{K} A^{(k)} x_{t-k} + e_{t}$  where each  $A^{(k)}$  is a  $n \times n$  dimensional matrix that describes how lag  $k$  affects the future timepoints in the series' and  $e_{t}$  is a zero mean noise. Given this framework, we state that time series  $q$  does not Granger-cause time series  $p$ , if and only if for all  $k$ ,  $A_{p,q}^{(k)} = 0$ . To better represent nonlinear dynamics amongst traces, a nonlinear autoregressive model (NAR) [19],  $g$ , may be defined, in which  $x_{t} = g(x_{1:t}, \dots, x_{n:t}) + e_{t}$  where  $x_{p:t} = (x_{p_1}, \dots, x_{p_{t-1}}, x_{p_t})$  describes the past of series  $p$ . The NAR nonlinear functions are commonly modeled jointly using neural networks.
4 Motif Causality
Using Granger causality as defined would overwhelm the generator with too much information, resulting in convergence issues for the GAN. Instead of looking at traces comprehensively, we need a way to scope how the generator understands relationships between time series. To this end, we aim to use the same intuition developed from Granger causality, namely developing an understanding of relationships comprehensively using less stringent temporal constraints, but scope these relationships specifically in terms of motifs. Therefore, we develop a concept of motif causality which, by learning causal relationships amongst sequences of motifs, allows the generator to learn realistic motif sequences and produce high quality synthetic traces as a result.
4.1 Extending Granger Causality to Motifs
In order to quantify the relationships amongst sequences of motifs to best capture glucose dynamics, we extend the idea of Granger causality to work with motifs. Given a motif set with  $m$  motifs, we build a separate (component) model, called a motif network in our method, for each motif, resulting in  $m$  motif networks. For a single motif  $\mu_{i}$  at time  $t$ ,  $\mu_{i_t}$ , we define a function  $g_{i}$  specifying how motifs in previous timesteps are mapped to that motif:  $\mu_{i_t} = g_i(\mu_{1 < t},\dots,\mu_{m < t}) + e_{i_t}$  where  $\mu_{j_{<t}} = (\mu_{j_1},\dots,\mu_{j_{t-1}},\mu_{j_t})$  describes the past of motif  $\mu_j$ . The output of  $g_{i}$  is a vector, which is added to the noise vector  $e_{i_t}$ . Essentially, we define motif  $\mu_i$  in terms of its relationship to past motifs. The  $g_{i}$  function takes in some mapping that describes how motifs in previous timesteps are mapped to the current motif  $\mu_{i_t}$ . The mapping is not specified in this notation, and could be defined in many different ways. In our case, we instantiate  $g_{i}$  using a single-layer LSTM, described next.
A  $g_{i}$  function for each motif  $\mu_{i}$  in the motif set is modeled using a motif network with a single-layer RNN architecture. For a RNN predicting a single component motif, let  $h_{t} \in \mathbb{R}^{m}$  represent the  $m$ -dimensional hidden state at time  $t$ . This represents the historical context of the motifs in the series for predicting a component motif at time  $t$ ,  $\mu_{i_t}$ . At time  $t$ , the hidden state is updated:  $h_{t} = g_{i}(h_{t - 1}) + e_{i_{t}}$ .  $g_{i}$  here is the function describing how motifs in previous timesteps are mapped
Figure 3: Motif Causality Block.
to the current motif, and is modeled (instantiated) as a single-layer LSTM as they are good at modeling long, nonlinear dependencies amongst traces [20]. The output for a motif  $\mu_{i}$  at time  $t$ ,  $\mu_{i_t}$  can be obtained by a linear decoding of the hidden state,  $\mu_{i_t} = W^o h_t + e_{i_t}$ , where  $W^o$  is a matrix of the output weights. These weights control the update of the hidden state and thereby control the influence of past motifs on this component motif. Essentially, this function learns a weighting that quantifies how helpful motifs in previous timesteps are for predicting the specified motif  $\mu_{i}$  at time  $t$ . We note that we define causality in this way based on how Granger causality models such relationships, which is different from traditional causality models.
If all elements in the  $j$ th column of  $W^o$  are zero ( $W_{:j}^o = 0$ ), this is a sufficient condition for an input motif  $\mu_j$  being motif non-causal on an output  $\mu_i$ . Therefore, we can find the motifs that are motif-causal for motif  $\mu_i$  using a group lasso penalty optimization across the columns of  $W^o$ :
$$
\min _ {W} \sum_ {t = 2} ^ {T} (\mu_ {i _ {t}} - g _ {i} (\mu_ {0 _ {<   t}}, \dots , \mu_ {m _ {<   t}})) ^ {2} + \sum_ {j = 1} ^ {m} | | W _ {: j} ^ {o} | | _ {2}
$$
We define this as the local motif loss,  $\mathcal{L}_{ml}$ , which is optimized in each motif network using proximal gradient descent.
4.2 Training the Motif Causality Block
We next describe how the motif causality block is trained to learn motif causal relationships amongst traces, displayed in Figure 3. The block is structured in this way to accommodate the privacy integration (Section 6.2); here, we present its implementation without any privacy noise.
Partition data. First, the data is partitioned into  $r$  partitions (Step 1, Figure 3) such that no models are trained on overlapping data. The number of partitions,  $r$ , is a user-specified hyperparameter.
Build motif network for each motif. Next, within each data partition a set of motif networks is trained. As a pre-processing step, we assume each trace has been chunked into a sequence of motifs of size  $\tau$  (Section 3.1).  $\tau$  is a hyperparameter, which we suggest chosen based on the longest effect time of a trace event. We use  $\tau = 48$ , corresponding to 4 hours of time, because large glucose events (from behaviors like eating) are encompassed within that time frame; see Appendix B for more details. We assume a tolerance of  $\sigma = 2\mathrm{mg / dL}$ , chosen to allow for reasonable variations in glucose. To model motif causality for an entire set of data, a  $g_{i}$  function is implemented for each motif via a separate RNN motif net following the description provided previously, resulting in  $m$  total networks (Step 2a, Figure 3). If all the motifs were trained together using a single motif network, it would not be possible to quantify the exact causal effects between each individual motif as we would not know which exact motifs contributed to a prediction (only that there is some combination of unknown motifs that contribute to an accurate prediction for a particular motif). By training each motif network separately, we are able to quantify the exact effect each motif has on each other, without any confounding effects from other motifs.
Combine outputs of individual motif networks. Each motif network outputs a vector of weights  $W^{o}$  of dimensionality  $1 \times m$ , corresponding to the learned causal relationships (Step 2b, Figure 3). Values in the vector are between 0 (no causal relationship) and 1 (strongest causal relationship) and
Figure 4: Overview of GlucoSynth Architecture.
give the degree to which every other motif is motif causal of the particular motif  $\mu_{i}$  the RNN was specialized for. To return a complete matrix that summarizes causal relationships amongst all motifs, we stack the weights (Step 2c). The output of each data partition is a complete motif causality matrix, resulting in  $r$  total matrices, each of dimensionality  $m\times m$ .
Aggregate matrices and integrate with GAN. After motif causality matrices have been outputted from each data partition, the weights in the matrices are aggregated (Step 3, Figure 3) to return the final aggregate causality matrix,  $M$  (Step 4). In the nonprivate version, the weights are averaged. Finally,  $M$  is sent to the generator to help it learn how to conserve motif relationships within sequences of motifs in the synthetically generated data. Details are described next in the subsequent section.
5 GlucoSynth
The complete GlucoSynth framework, shown in Figure 4, comprises four key blocks: the motif causality block (explained previously in Section 4), an autoencoder, a generator and a discriminator. We walk through the remaining components of the framework surrounding the GAN next.
5.1 GAN Architecture Components
Autencoder. We use an autoencoder (AE) with an RNN architecture to learn a lower dimensional representation of the traces, allowing the generator to better preserve underlying temporal dynamics of the traces. The autoencoder consists of two networks: an embedder and a recovery network. The embedder uses an encoding function to map the real data into a lower dimensional space:  $Enc(x): x \in \mathbb{R}^n \to x_e \in \mathbb{R}^e$  while the recovery network reverses this process, mapping the embedded data back to the original dimensional space:  $Dec(x_e): x_e \in \mathbb{R}^e \to \tilde{x} \in \mathbb{R}^n$ . A foolproof autoencoder perfectly reconstructs the original input data, such that  $x = \tilde{x} \equiv Dec(Enc(x))$ . This process yields the Reconstruction Loss,  $\mathcal{L}_R$ , the Mean Square Error (MSE) between the original data  $x$  and the recovered data,  $\tilde{x}: \mathrm{MSE}(x, \tilde{x})$ .
Generator. We implement the generator via an RNN or LSTM. Importantly, the generator works in the embedded space, by receiving the input traces passed through the embedder  $(x_{e})$ . To generate synthetic data, a random vector of noise,  $z$  is passed through the generator and then the recovery network to return the synthetic traces in the original dimensional space. To learn how to produce high-quality synthetic data, the generator receives three key pieces of information:
1 - Stepwise. The generator receives batches of real data to guide the generation of realistic next step vectors. To do this, a Stepwise Loss,  $\mathcal{L}_S$ , is computed at time  $t$  using the MSE between the batch of embedded real data,  $x_{et}$ , and the batch of embedded synthetic data,  $\hat{x}_{et}$ :  $\mathrm{MSE}(x_{et},\hat{x}_{et})$ . This allows the generator to compare (and learn to correct) the discrepancies in stepwise data distributions.
2 - Motif Causality. The generator needs to preserve sequences of motifs in addition to temporal dynamics. Using the aggregate causality matrix  $M$  returned from the Motif Causality Block, the generator computes a motif causality matrix,  $M_{\hat{x}}$ , on the set of synthetic data  $\hat{x}$ . Because the original