GlucoBench: Curated List of Continuous Glucose Monitoring Datasets with Prediction Benchmarks
Authors: Renat Sergazinov
Elizabeth Chun
Valeriya Rogovchenko
Nathaniel J Fernandes
Nicholas Kasman
Irina Gaynanova
Venue: ICLR 2024 poster (2024)
Keywords: diabetes management
continuous glucose monitors (CGM)
glucose trajectory prediction
artificial pancreas systems
public datasets
standardized tasks
benchmark models
glycemic control
Primary area: datasets and benchmarks
TL;DR: The paper introduces a comprehensive resource for CGM-based glucose forecasting, including a curated list of public datasets, a standardized task list, and benchmark models.

Abstract: The rising rates of diabetes necessitate innovative methods for its management. Continuous glucose monitors (CGM) are small medical devices that measure blood glucose levels at regular intervals providing insights into daily patterns of glucose variation. Forecasting of glucose trajectories based on CGM data  holds the potential to substantially improve diabetes management, by both refining artificial pancreas systems and enabling individuals to make adjustments based on  predictions to maintain optimal glycemic range. Despite numerous methods proposed for CGM-based glucose trajectory prediction, these methods are typically evaluated on small, private datasets, impeding reproducibility, further research, and practical adoption. The absence of standardized prediction tasks and systematic comparisons between methods has led to uncoordinated research efforts, obstructing the identification of optimal tools for tackling specific challenges. As a result, only a limited number of prediction methods have been implemented in clinical practice.  

To address these challenges, we present a comprehensive resource that provides (1) a consolidated repository of curated publicly available CGM datasets to foster reproducibility and accessibility; (2) a standardized task list to unify research objectives and facilitate coordinated efforts; (3) a set of benchmark models with established baseline performance, enabling the research community to objectively gauge new methods' efficacy; and (4) a detailed analysis of performance-influencing factors for model development. We anticipate these resources to propel collaborative research endeavors in the critical domain of CGM-based glucose predictions. Our code is available online at github.com/IrinaStatsLab/GlucoBench.

Published as a conference paper at ICLR 2024
GLUCOBENCH: CURATED LIST OF CONTINUOUS GLU- COSE MONITORING DATASETS WITH PREDICTION BENCHMARKS
Renat Sergazinov-; Elizabeth Chun', Valeriya Rogovchenko', Nathaniel Fernandes", Nicholas Kasman², Irina Gaynanova DDpaarmment of Statistics, Texas A&M University "Department of Electrical and Computer Engineering, Texas A&M University
ABSTRACT
The rising rates of diabetes necessitate innovative methods for its management. Continuous glucose monitors (CGM) small medical devices that measure blood glucose levels at regular intervals providing insights into daily patterns of glucose variation. Forecasting glucose trajectories based on CGM data holds the potential to substantially improve diabetes management, by both refining artificial pancreas systems and enabling individuals make adjustments based on predictions maintain optimal glycemic range. Despite numerous methods proposed for CGM- based glucose trajectory prediction, these methods are typically evaluated on small, private datasets, impeding reproducibility, further research, and practical adoption. The absence of standardized prediction tasks and systematic comparisons between methods has led uncoordinated research efforts, obstructing the identification of optimal tools for tackling specific challenges. As result, only limited number prediction methods have been implemented in clinical practice.
To address these challenges, present comprehensive resource that provides (1) a consolidated repository of curated publicly available CGM datasets to foster repro- ducibility and uccessibility; (2) a standardized task list to unify research objectives and facilitate coordinated efforts; (3) a set of benchmark models with established baseline performance, enabling the research community to objectively gauge new methods' efficacy; and (4) detailed analysis of irfformannc-influenncd for model development. We anticipate these resources to propel collaborative research endeavors in the critical domain of CGM-based glucose predictions. Our code is available online at github.
factors
INTRODUCTION
According to the International Diabetes Federation, 463 million adults worldwide have diabetes with 34.2 million people affected in the United States alone (ID, 202T). Diabetes is a leading cause of heart disease (Nanayakkara al. 202I). blindness (Wykoff et al. [021). and kidney discase (Alicia et al. 2017). Glucose management is a critical component of diabetes care, however achieving target glucose levels difficult due to multiple factors that affect glucose fluctuations, e.g., diet, exercise, stress, medications, and individual physiological variations.
Continuous glucose monitors (CGM) are medical devices that measure blood glucose levels frequent intervals, often with a granularity of approximately one minute. CGMs have great potential to improve diabetes management by furnishing real-time feedback to patients and by enabling an autonomous artificial pancreas (AP) system when paired with an insulin pump (Contreras & Vchi 2018, Kim & Yoon, 2020). Figure Iillsttrates an example of a CGM-human feedback loop recommender setting. The full realization of CGM potential, however, requires accurate glucose prediction models. Although numerous prediction models (Fox et al.) 2018) |Armandpour al. 2021 Sergazinov et al. 2023) have been proposed, only simple physiological (Bergman et al. 1979
Hovorka et al. 2004) or statistical (Oviedo et al., 2017 Mirshekarian et al. 2019 Xie & Wang
Address correspondence misergazinov @tamu.edu, irinag@tami

---

Published as a conference paper at ICLR 2024
Data Collection
223
Aggregation
18ce-3-02
1422-33-00
1000-41-55
1460-22-05 192301-47
1000440-44
53350
E250
100
1868-22.0 1900
1000
1000
3.00
1440
150
1270
Decision-making
Learning
azo
225
2200
150
130
210
200
100
Figure 1: Sample of glucose curves captured by the Dexcom G4 Continuous Glucose Monitoring (CGM) system, with dates de-identified for privacy (Wennstok et al. 2016).
2020 models are utilized within current CGM and AP software. The absence of systematic model evaluation protocols and established benchmark datasets hinder the analysis of more complex models' risks and benefits, leading to their limited practical adoption (Mirshekarian et al. 2019).
In response, present curated list of five public CGM datasets and systematic protocol for models' evaluation and benchmarking. The selected datasets have varying sizes and demographic characteristics, while the developed systematic petta-prprocessing pipeline facilitates the inclusion of additional datasets. We propose two tasks: (1) enhancing the predictive accuracy; (2) improving the uncertainty quantification (distributional fit) associated with predictions. In line with previous works (Mirshekarian et al., 2019 Xie & Wang 2020), we measure the performance on the first task with mean squared error (MSE) and mean absolute error (MAE), and on the second task with likelihood and expected calibration error (ECE) (Kuleshov et al. 2018). For each task, we train and evaluate set of baseline models. From data-driven models, we select linear regression and ARIMA to represent shallow baselines, and Transformer (Vaswani et al.] 2017). NHiTS (Challu et al. (023), TFT (Lim) jet al. ()2T), and Gluformer (Sergazinov et al.) 2023) to represent deep learning baselines. We select the Latent ODE (Rubanova al. 2019) to represent hybrid data-driven/ / physiological model.
Our work contributes a curated collection of diverse CGM datasets, formulation of two tasks focused on model accuracy and uncertainty quantification, efficient benchmarking protocol, evaluation of range of baseline models including shallow, deep and hybrid methods, and a detailed analysis of performance-infuueciing factors for model optimization.
RELATED WORKS
An extensive review of glucose prediction models and their utility is provided by Oviedo al. (2017)
Contreras & Vchil y018); Kim & Yoon (2020); Kavakiotis et al. (2017). Following Oviedo et al. (2017), we categorize prediction models as physiological, data-driven, or hybrid. Physiological models rely on the mathematical formulation of the dynamics of insulín and glucose metabolism via differential equations (Man et al. 2014 Lehmann Deutsch, 1991). A significant limitation of these models is the necessity to pre-define numerous parameters. Data-driven models rely solely on the CGM data (and additional covariates when available) to characterize glucose trajectory without incorporating physiological dynamics. These models can be further subdivided into shallow (e.g. linear regression, ARIMA, random forest, etc.) and deep learning models (c.g. recurrent neural network models, Transformer, etc.). Data-driven models, despite their capacity to capture complex
2

---

Published as a conference paper at ICLR 2024
Table 1: Summary of the glucose prediction models by dataset and model type. We indicate "open" for datasets that are publicly available online, "simulation" for the ones based on simulated data, "proprietary" for the ones that cannot be released. We indicate deep learning models by "deep". non-deep learning models by "shallow", and physiological models by "physiological." We provide full table with references all the works in Appendix A.
Type
Diabetes
# of datasets
# of deep
# of shallow
# of Physiological
Open
Type! 1
9
13
3
Simulation Proprietary
Type 1
12
3
Mixed
22
8
patterns, may suffer from overfitting and lack interpretability. Hybrid models use physiological models as pre-processing or data augmentation tool for data-driven models. Hybrid models enhance the flexibility of physiological models and facilitate the fitting process, albeit at the expense of diminished interpretability. Table summarizes existing models and datasets, indicating model type.
Limitations. The present state of the field is characterized by several key constraints, including (I) an absence of well-defined benchmark datasets and studies, (2) dearth of open-source code bases, and (3) omission of Type 2 diabetes from most open CGM studies. To address the second limitation, two benchmark studies have been undertaken to assess the predictive performance of various models MMrrheekarian al. 2019] Xie & Wang 2020). Nonetheless, these studies only evaluated the models on one dataset (Marling& & Bunescu, 2020), comprising a limited sample of 5 patients with Type diahetes, and failed provide source code. We emphasize that, among the 45 methods identified in Table I staggering 38 works do not offer publicly available implementations. For the limitation (3), is important recognize that Type is more easily managed through lifestyle change and oral medications than Type 1 which requires lifelong insulin therapy.
3 DATA
3.1 DESCRIPTION
We have selected five publicly available CGM datasets: Broll et al. (2021): Colás al. (2119): Dubosson et al. (2188); Hall et al. (2018); Weinstock et al. (2016).
To ensure data quality, we used the following set of criteria. First, we included variety of dataset sizes and verified that each dataset has measurements on at least 5 subjects and that the collection includes variety of sizes ranging from only five (Broll et al. [202]) to over 200 (Colás al. 2019 Weinstock et al. 2016) patients. On the patient level, we ensured that each subject has non-missing CGM measurements for least 16 consecutive hours. At the CGM curve level, have verified that measurements fall within clinically relevant range of 20 mg/dL to 400 mg/dL, avoiding drastic fluctuations exceeding 40 mg/dL within a 5-minute interval, and ensuring non-constant values.
Finally, we ensured that the collection covers distinct population groups representing subjects with Type diabetes (Dubosson et al.] 2018 Weinstock al. 2016). Type 2 diabetes (Broll et al.) 202T). or a mix of Type 2 and none (Colás et al. 2019 Hall et al. (0118). We expect that the difficulty of accurate predictions will depend on the population group: patients with Type have significantly larger and more frequent glucose fluctuatons. Table summarizes all five datasets with associated demographie information, where some subjects are removed due to data quality issues as a result of We describe data availability in Appendix A.
pre- ce-proceesing (Section 3.2
Covariates. In addition to CGM data, each dataset has covariates (features), which we categorize based on their temporal structure and input type. The temporal structure distinguishes covariates as static (e.g. gender), dynamic known (e.g. hour, minute), and dynamic unknown (e.g. heart beat, blood pressure). Furthermore, input types define covariates as either real-valued (e.g. age) or ordinal (e.g. education level) and categorical or unordered (c.g. gender) variables. We illustrate different types of temporal variables in Figure We summarize covariate types for each dataset in Appendix A.
3

---

Published as a conference paper at ICLR 2024
Table 2: Demographic information (average) for each dataset before (Raw) and after pre-processing (Processed). CGM indicates the device type; all devices have 5 minute measurment frequency.
Datasel
Diabetes
CGM
# of Subjects
Age
Sex (M/F)
Overall
Overall
Raw
Processed
Raw
Processed
Raw
Processed
Broll et
2021
Type
Dexcom G4
5
NA
NA
NA
NA
Colas al. 2019
Mixed
MiniMed iPro
208
201
59
59
103 / 104
100/ 100
Dubosson ct al. 2018
Type
MiniMed iPro2
9
7
NA
NA
6/3
NA
[all el al 2018
Mixed
Dexcom G4
57
48
48
25 25/32
NA
Weinstock ct al.
2016
Type
Dexcom G4
200
192
68
NA
106
101/91
(Age)
Dynamic known (Time)
Dynamic urknown lHeart Rate)
Observed
Predicted
Observed
Predicted
Predicted
24 04:00
Heart (bpm)
Age (years))
00:30-
24:00:0
2101300
Unkoww (future)
21
$1/00
Time (hours)
Time (hours)
Time (hours)
Figure 2: An illustration of static (Age), dynamic known (Date), and dynamic unknown (Heart Rate) covariate categories based on data from Hall et al. (2018) and Dubosson et al. (2018).
Table 3: Interpolation parameters for datasets.
Parameters
Broll
Colas
Dubosson
Hall
Weinstock
Gap threshold (minutes)
45
45
30
30
45
Minimum length (hours)
20
16
20
16
20
3.2 PRE-PROCESSING
We pre-process the raw CGM data via interpolation and segmentation, encoding categorical features, data partitioning, scaling, and division into input-output pairs for training a predictive model.
Interpolation and segmentation. To put glucose values on a uniform temporal grid, identify gaps in each subject's trajectory due to missing measurements. When the gap length is less than a predetermined threshold (Table we impute the missing values by linear interpolation. When the gap length exceeds the threshold, we break the trajectory into several continuous segments. Green squares in Figure 3 indicate gaps that will be interpolated, whereas red arrows indicate larger gaps where the data will be broken into separate segments. In Dubosson et al. (2018) dataset, we also interpolate missing values dynamic covariates (e.g., heart rate). Thus, for each dataset we obtain a list of CGM sequences D = +x Jag with i indexing the patients and j the continuous segments. Each segment (t) has length 50 > Limin where Lmin is the pre-specified minimal value (Table 3
(9)
Covariates Encoding. While many of the covariates are real-valued, e.g., age, some covariates are categorical, e.g., sex. particular, Weinstock al. (2016) dataset has 36 categorical covariates with an average of 10 levels per covariate. While one-hot encoding is a popular approach for modeling categorical covariates, it will lead to 360 feature columns on Weinstock al.) (2016). making undesirable from model training time and memory perspectives. Instead, we use label encoding by converting each categorical level into numerical value. Given R covariates, we include them in the
(i) (8) dataset D (x) Ci. Rtj)is where c E IR. for static and o R R for dynamic.
(i)
Data splitting. Each dataset is split into train, validation, and in-distribution (ID) test sets using 90% of subjects. For each subject, the sets follow chronological time order as shown in Figure 3 with validation and ID test sets always being of a fixed length of 16 hours each (192 measurements). The data from the remaining 10% of subjects is used to form an out-of-distribution (OD) test set to assess the generalization abilities of predictive models as in Section 5.2 Thus, D D UD mat UD Dand
4

---

Published as a conference paper at ICLR 2024
Glucoss (mglaa))
800
200
100
00:00
12:00
03.00
12:00
00:30
Qlucose (mgiat)
200
150
100
50
12:00
00:00 Time (hours)
Validation
Test
Figure 3: Example data processing Weinstock al. (016). The red arrows denote segmentation, green blocks interpolate values, and red brackets indicate dropped segments due to length.
Scaling, We use min-max scaling to standardize the measurement range for both glucose values and covariates. The minimum and maximum values are computed per dataset using Dir, and then the same values are used to rescale Deal, Did, and Dod-
Input-output pairs. Let I. be a length L contiguous slice of segment from index & to k + L. We define input-output pair (x I JKtL+18+L+th where y (1) J1++++
(i)
and T is the length of prediction interval. Our choices of T, L and i: are follows. 1.4.
(i)
In line with the previous works (Oviedo et al., 2017) focus the 1-hour ahead forecasting (T = 12 for 5 minute frequency). We treat as a hyper-parameter for model optimization since different models have different capabilities in capturing long-term dependencies. We sample without replacement from among the index set of the segment during training, similar to Oreshkin al. (2020); CCallu et al. (2023), and treat the total number of samples as a model hyper-parametee We found the sampling approach to be preferable over the use of sliding window with a unit stride (Herzen al. 2022), as the latter is computationally prohibitive on larger training datasets and leads to high between-sanee sample correlation, slowing convergence in optimization. We still use the sliding window when evaluating the model on the test set.
BENCHMARKS
4.1 TASKS AND METRICS
Task 1: Predictive Accuracy. Given the model prediction L. we measure accuracy on the test set using root mean squared error (RMSE) and mean absolute error (MAE):
T' Σ T URfLL
T
RMSE.
1
0 )l242.)) MAE = T Uj.tt-I Djk+L
Σ
Since the distribution of MAE and RMSE across samples is right-skewed, we use the median of the errors as has been done in Sergazinov et (2023): Armandpour al. (202I).
Task 2: Uncertainty Quantification. To measure the quality of uncertainty quantification, we use two metrics: likeliiol on test data and calibration. For models that estimate parametric predictive
distribution over the future values, p(()
f 5 RT + [0, 1], we evaluate log-likelitee likelihokhd as
log Li,j,k k log ((0 J. (y (6) (*)+LLLER+LH))
where the parameters of the distribution are learned from training data, and the likelihood is evaluated on test data. Higher values indicate better fit the observed distribution. For both parametric
5

---

Published as a conference paper at ICLR 2024
and non non-parametric models (such as quantile-basee methods), Kuleshov et al. 2018). The original metric is limited only to the univariate distributions. To address the issue, report an average calibration across marginal estimates for each time 1, T. To compute marginal calibration at time I, we (1) pick M target confidence levels 0 < p < ...
regression calibration metric
PM < (2) estimate realized confidence level Pm using N test input-output pairs as
(2,0 F1 sk+ Lo (USAeLtte) S Pm Pm = Kusktetal
N
and (3) compute calibration across all M levels
Cal = > (pm -pm)²
The smaller the calibration value, the better the match between the estimated and true levels.
4.2
MODELS
To benchmark the performance the two tasks, compare the following models. ARIMA is classical time-series model, which has been previously used for glucose predictions (Otoom et al. 2015 Yang et al., 2019). Linear regression is simple baseline with separate model for each time step f = 1,...,7 T. Goott (Chen & Guestrin 2016) is gradient-boosted tree method, with separate model for each time step to support multi-oulput regression. Transformer represents a standard encoder-decoder auto-regressive Transformer implementation (Vaswani et al. (017). Temporal Fusion Transformer (TFT) is a quantile- based model that uses RNN with attention. TFT is the only model that offers out-of-the box support for static, dynamic known, and dynamic unknown covariates. NHiTS uses neural hierarchical interpolation for time series, focusing on the frequency domain (Challu et al. 2023). Latent ODE uses recurrent neural network (RNN) to encode the sequence to latent representation (Rubanova et al., 2019). The dynamics in the latent space are captured with another RNN with hidden state transitions modeled as ODE. Finally, generative model maps the latent space back to the original space. Gluformer is a probabilistic Transformer model that models forecasts using mixture distribution (Sergazinov al. 2023). For ARIMA, we use the code from (Federico Garza 2022) which implements the algorithm from (Hyndman & Khandakar] 2008). For linear regression, XGBoost, TFT, and NHiTS, we use the open-source DARTS library (Herzen et al. 2022). For Latent ODE and Gluformer, we use the implementation in PyTorch We report the compute resources in Appendix C.
(Rubanova al. 2019 Sergazinov al. 2023)
4.3 TESTING PROTOCOLS
In devising the experiments, pursue the principles of reproducibility and fairness all methods.
Reproducibility. As the performance results are data split dependent, we train and evaluate each model using the same two random splits. Additionally, all stochasticlllyttraiedd models (tree-based and deep learning) are initialized 10 times on each training set with different random seeds. Thus, each stochastically-trainee model is re-trained and re-evaluated 20 times, and each deterministically- trained model 2 times, with the final performance score taken as an average across evaluations. We report standard error of each metric across the re-runs in Appendix B.
Fairness. To promote fairness and limit the scope of comparisons, we focus on out-of-the-bue model performance when establishing baselines. Thus, we do not consider additional model-specific tuning that could lead to performance improvements, e.g., pre-training, additional loss functions, data augmentation, distillation, learning rate warm-up, learning rate decay, etc. However, since model hyper-parameters can significantly affect performance, we automate the selection of these parameters. selection algorithm provided in (Hyndman & Khandakar 2008). For all other models, we Optuna (Akiba al., 2019) to run Bayesian optimization with a fixed budget of 50 iterations. We provide a discussion on the selected optimal model hyperparameters for each dataset in the supplement (Appendix C).
For ARIMA, we use the native automatic

---

Published as a conference paper at ICLR 2024
2.25
otcanares
ARI
2.00
2012
XGB
UN
D:23
1.76
15.0
Density
0.06
MAE
GLU
LAT
2.00
0:75
75
0:2
8.00
Models
Accuracy (MAE)
Generalization
Error by population
(c) Daytime nighttime
Figure 4: Analysis of errors by: (a) OD versus ID, (b) population diabetic type (healthy + Type -
Type 1), (c) daytime (9:00AM to 9:00PM) versus nighttime (9:00PM to 9:00AM).
Table 4: Accuracy and uncertainty metrics for selected models based on in-distribution (ID) test set without covariates. The selected models are best at least one dataset for at least one metric. The best results on each data set are highlighted in boldface. TFT lacks likelihood information as it is
quantile-bart based model. Standard errors are reported in Appendix B.
Broll
Colas
Dubosson
Hall
Weinstock
Accuracy
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
ARIMA
10.53
8.67
5.80
4.80
13.53
11.06
8.63
7.34
13.40
11.25
Lincar
11.68
9.71
5.26
4.35
12.07
9.97
7.38
6.33
13.60
11.46
Latent ODE
14.37
12.32
6.28
5.37
20.14
17.88
7.13
6.11
13.54
11.45
Transformer
15.12
13.20
6.47
5.65
16.62
14.04
7.89
6.78
13.22
11.22
Uncertainty
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Gluformer
-2.11
0.05
-1.07
0.14
-2.15
0.06
-1.56
0.05
-2.50
0.08
TFT
0.16
0.07
0.23
0.07
0.07
4.4 RESULTS
We trained and tested each model outlined above on all five datasets using the established protocols. Table 4 present the results for the best-performing models on Task (predictive accuracy) and Task (uncertainty quantification). Appendix B includes full tables for all models together with standard errors and the visualized forecasts for the best models on /Weinstock et al. 2016) dataset.
On Task I, the simple ARIMA and linear regression models have the highest accuracy on all but two datasets. On Hall al. (2018) dataset (mixed subjects including normoglycemic, prediabetes and Type 2 diabetes), the Latent ODE model performs the best. On Weinstock et al. (2016) dataset (the largest dataset), the Transformer model performs the best.
On Task 2, Gluformer model achieves superior performance measured by model likelihood on all datasets. In regards calibration. Gluformer is best on all but datasets. On Colás et al. (2019) and Weinstock et al. (2016) datasets (the largest datasets), the best calibration is achieved by TFT.
5 ANALYSIS
5.1 WHY DOES THE PERFORMANCE OF THE MODELS DIFFER BETWEEN THE DATASETS?
Three factors consistently impact the results across the datasets and model configurations: (1) dataset size, (2) patients' composition, and (3) time of day. Below we discuss the effects of these factors accuracy (Task 1), similar observations hold for uncertainty quantification (Task 2).
Tables inndicates that the best- tist-perforing model each dataset is dependent the dataset size. For smaller datasets, such as Broll et al. (2021) and Dubosson et al.) (2018). simple models like ARIMA and linear regression yield the best results. In general, that deep learning models excel
7

---

Published as a conference paper at ICLR 2024
Table 5: Change in accuracy and uncertainty tasks between ID and OD sets. We indicate increases in performance in blue and decreases in red. TFT lacks likelihood information as it is a quantile-based
model.
Brell
Colas
Dubosson
Hall
Weinstock
Accuracy
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
ARIMA
+11.59%
+12.01%
+2.02%
+1.49%
+38.56%
+31.81%
-479%
-5.08%
+18.47%
+18.56%
Linear
+2.51%
+1.29%
+1.27%
+1.38%
+30.01%
+19.41%
+6.46%
+4.58%
+14.5%
+14.22%
Latent ODE
+4.12%
+5.93%
-00005%
-9.3%
-13.7%
-15.47%
+8.07%
+8.18%
+11.17%
+11.08%
Transformer
-7.16%
-6.96%
-7.78%
-7.23%
-5,52%
-7.52%
+3.69%
+4.15%
+7.0%
+6.16%
Uncertainty
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Gluforner
+6.7 72%
1006.76%
-50.33%
-29. 29.883%
45.64%
+83.63%
+7 .96%
+9.24%
+3.33%
46. 23%
TFT
-4.8%
+15.18%
+10.56%
+30.52%
+9.94%
Table 6: Changes in accuracy and uncertainty tasks with and without covariates on ID test set. We indicate increases in performance in blue and decreases in red. TFT lacks likelihood information as is a quantile-based model.
Broll
Colas
Dubosson
Hall
Weinstock
Accuracy
RMSE
MAB
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
Linear
-14.85%
-13.34%
+5.54%
+5.75%
8.84%%
+0.61%
46.17%
+5.09%
-1.54%
-1.08%
Transforner
-15.14%
-14.64%
+30.31%
+37.56%
+64.99%
+73.82%
-5066%
-5.4%
+9.33%
+12.41%
Uncertainty
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Lik.
Cal.
Lik.
Cal
IFT
+94.6%
+114.61%
+7.57%
+16.84%
-21.55%
larger datasets: Hall et al. (0188) (best model is Latent ODE) and Weinstock et al. (0116) (best model is Transformer) are 2 of the largest datasets. The only exception is Colás et al. (2019). on which the best model is linear regression. We suggest that this could be explained by the fact that despite being large. Colás et al. (019)) dataset has low number observations per patient: 100,000 glucose readings across 200 patients yeilds 500 readings or days worth of data per patient. In comparison, [Hall et al. (2018) has 2,000 readings per patient or 7 days, and Weinstock et al. (2016) has approximately 3,000 readings per patient or 10 days.
Fb demonstrates that the accuracy of predictions is substantially influenced by the patients' population group. Healthy subjects demonstrate markedly smaller errors compared subjects with Type 1 or Type 2 diabetes. This discrepancy is due to healthy subjects maintaining a narrower range of relatively low glucose level, simplifying forecasting. Patients with Type exhibit larger fluctuations partly due to consistently required insulin administration in addition to lifestyle-related factors, whereas most patients with Type are not on insulin therapy.
Figure () shows the impact of time of day on accuracy, with daytime (defined as 9:00AM to 9:00PM) being compared to nighttime for Transformer model on BBoll et al. (202T) dataset. The distribution of daytime errors is more right-skewed and right-shifted compared to the distribution of nighttime errors, signifying that daytime glucose values are harder to predict. Intuitively, glucose values are less variable overnight due the absence of food intake and exercise, simplifying forecasting. We include similar plots for all models and datasets in Appendix B. This finding underscores the importance of accounting for daytime and nighttime when partitioning CGM data for model training and evaluation.
Overall, we recommend using simpler shallow models when data is limited, the population group exhibits less complex CGM profiles (such as healthy individuals or Type 2 patients), for nighttime forecasting. Conversely, when dealing with larger and more complex datasets, deep or hybrid models are the preferred choice. In clinical scenarios where data is actively collected, is advisable to deploy simpler models during the initial stages and, in later stages, maintain an ensemble of both shallow and deep models. The former can act as a guardrail or be used for nighttime predictions.
5.2 ARE THE MODELS GENERALIZABLE TO PATIENTS BEYOND THE TRAINING DATASET?
Table 5 compares accuracy and uncertainty quantification of selected models in-distribution (ID) and out-of- diitro--ditribttionn (OD) test sets, while the full table is provided in Appendix B. Here we assume that each patient is different, in that the OD set represents a distinct distribution from the ID set.
8

---

Published as a conference paper at ICLR 2024
In both tasks, most models exhibit decreased performance on the OD data, emphasizing individual- level variation between patients and the difficulty of cold starts on new patient populations. Figure a) displays OD-to-ID accuracy ratio (measured in MAE) for each model and dataset: higher ratios indicate poorer generalization, while lower ratios indicate better generalization. In general, we observe that deep learning models (Transformer, NHiTS, TFT, Gluformer, and Latent ODE) generalize considerably better than the simple baselines (ARIMA and linear regression). We attribute this to the deep learning models' ability capture and recall more patterns from the data. Notably, XGBoost also demonstrates strong generalization capabilities and, in some instances, outperforms the deep learning models in the generalization power.
5.3 How DOES ADDING THE COVARIATES AFFECT THE MODELING QUALITY?
Table demonstrates the impact of including covariates in the models on Task 1 (accuracy) and Task 2 (uncertainty quantification) compared to the same models with no covariates. As the inclusion of covariates represents providing model with more information, any changes in performance can be attributed to (1) the quality of the covariate data; (2) model's 's ability handle multiple covariates. We omit ARIMA, Gluformer, and Latent ODE models as their implementations do not support covariates.
In both tasks, the impact of covariates model performance varies depending on the dataset. For Colás et al. (2019) Dubosson et al. (2018), we observe a decrease in both accuracy and uncertainty quantification performance with the addition of covariates. Given that these are smaller datasets with a limited number of observations per patient, we suggest that the inclusion of covariates leads to model overfitting, consequently increasing test-time errors. In contrast, for Broll et al.) (202I) that is also small, unlike for all other datasets, we have covariates extracted solely from the timestamp, which appears to enhance model accuracy. This increase in performance is likely attributable to all patients within the train split exhibiting more pronounced cyclical CGM patterns, which could explain why the overfitted model performs better. This is further supported by the fact that the performance on the OD set deteriorates with the addition of covariates. Finally, in the case of Hall et al. (2018) and Weinstock et al. (2016), which are large datasets, the inclusion covariates has mixed effects, indicating that covariates do not contribute significantly to the model's performance.
DISCUSSION
Impact. We discuss potential negative socictal impact of our work. First, inaccurate glucose forecasting could lead to severe consequences for patients. This is by far the most important consideration that we discuss further in Appendix D. Second, there a potential threat from CGM device hacking that could affect model predictions. Third, the existence of pre-defined tasks and datasets may stifle research, as researchers might focus on overfitting and marginally improving upon well-known datasets and tasks. Finally, the release of health records must be treated with caution to guarantee patients' right to privacy.
Future directions. We outline several research avenues: (1) adding new public CGM datasets and tasks; (2) open-sourcing physiological and hybrid models; (3) exploring model training augmentation, such as pre-training on aggregated data followed by patient-specific fine-tuning and down-sampling night periods; (4) developing scaling laws for dataset size and model performance; and (5) examining covariate quality and principled integration within models. Related the point (5), we note that out of the 5 collected datasets, only Dubosson et al. (2018) records time-varying covariates describing patients physical activity (e.g. accelerometer readings, heart rate), blood pressure, food intake, and medication. We believe having larger datasets that comprehensively track dynamic patient behavior could lead to new insights and more accurate forecasting.
7 CONCLUSION
In this work, we have presented a comprehensive resource to address the challenges in CGMebased glucose trajectory prediction, including a curated repository of public datasets, a standardized task list, a set of benchmark models, and detailed analysis of performance-ifltueciing factors. Our analysis emphasizes the significance of dataset size, patient population, testing splits (c.g., in- and out-of-distribution. daytime, nighttime), and covariate availability.

---

Published as a conference paper at ICLR 2024
ACKNOW
The source of a subset of the data is the TID Exchange, but the analyses, content, and conclusions presented herein are solely the responsibility of the authors and have not been reviewed approved by the TID Exchange.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2623-222222222 (623-2631, 2019.
Grazia Aleppo, Katrina Ruedy, Tonya Riddlesworth, Davida F Kruger, Anne Peters, Irl Hirsch, Richard M Bergenstal, Elena Toschi, Andrew J Ahmann, Viral N Shah, et al. Replace-bg: randomized trial comparing continuous glucose monitoring with and without routine blood glucose monitoring in adults with well-controlled type diabetes. Diabetes care, 40(())538-544 2017.
Alessandro Aliberti, Irene Pupillo, Stefano Tema, Enrico Macii, Santa Di Cataldo, Edoardo Patti, and Andrea Acquaviva. A multi-patient data-driven approach to blood glucose prediction. lEEE Access, 7:6931 1 69325, 2019.
Radica Z Alicic, Michele Rooney, and Katherine R Tuttle. Diabetic kidney disease: challenges, progress, and possibilities. Clinical journal of the American Society of Nephrology: CJASN, 12 (12):2032, 2017.
Marios Anthimopoulos, Joachim Dehais, Sergey Shevchik, Botwey H Ransford, David Duke, Peter Diem, and Stavroula Mougiakakou. Computer vision-bace based carbohydrate estimation for type patients with diabetes using smartphones. Journal of diabetes science and technology, 9()): 507-515, 2015.
Mohammadreza Armandpour, Brian Kidd, Yu Du, and Jianhua Z. Huang. Deep Personalized Glucose Level Forecasting Using Attention-based Recurrent Neural Networks. In International Joint Conference Neural Networks (IJCNN,, 2021. doi: 10. 00LL0999CCNS5228820000..3333
Naviyn Prabhu Balakrishnan, Lakshminarayanann Samavedham, and Gade Pandu Rangaiah. Personal- ized Hybrid Models for Exercise, Meal, and Insulin Interventions in Type I Diabetic Children and Adolescents. Industrial Engineering Chemistry Research, 13200)::3300-1330 1ttp://1111111111222255522
2013. ISSN
0888-5885.
doi: 10.1021/ie402531k httpss//:
Jaouher Ben Ali, Takoua Hamdi, Nader Fnaiech, Véronique Di Costanzo, Farhat Fnaiech, and Jcan-Mare Ginoux. Continuous blood glucose level prediction of Type 1 Diabetes based on Artificial Neural Network. Biocybernetics and Biomedical Engineering. 38(4):828-84,, 2018. ISSN 02085216. doi: 10.1011//j 201...06...2110...0.00
http:: http:///innnnnguun..et elsevierrrreerrrrree
com/ [ctm://eeriaeee//////22222222222
Richard N Bergman, Y Ziya Ider, Charles R Bowden, and Claudio Cobelli. Quantitative estimation of insulin sensitivity. American Journal of Ehyxiology-Endocrinnoggg And Metabolism, 236(6): E667, 1979.
Alain Bock, Grégory François, and Denis Gillet. A therapy parameter-based model for predict- ing blood glucose concentrations in patients with type 1 diabetes. Computer Methods and Programs in Biomedicine, 118(2):107-223, 2015. ISSN 0169-2607. doi: 10.1016/j.cmpb. 2014. 12.022 URL https: http///wwwwcc www.wwwwwwww...eee.............e com/ wwt.://ww.c...caceeeeeeeeec /wt.eec/e//tttiee//......
S0 692607140033111
Dimitri Boiroux, Anne Katrine Duun-Henriksen, Signe Schmidt, Kirsten Nørgaard, Sten Mads- bad, Ole Skyggebjerg, Peter Ruhdal Jensen, Niels Kjølstad Poulsen, Henrik Madsen, and John Bagterp Jørgensen. Overnight Control of Blood Glucose in People with Type 1 Dia- betes. IFAC Proceedings Volumes, 45(18):73-785 78, 2012. ISSN 1474-6670. doi: 10.3182/ 20120829- HU- URL http:///wwwwwccccceeeeee...ccct. com/ wttp:ett...oooccccccce.c.ee science/
10

---

Published as a conference paper at ICLR 2024
Ransford Henry Botwey, Elena Daskalaki, Peter Diem, and Stavroula G Mougiakakou. Multi-model data fusion to improve an early warning system for hypo-hhyperlycemii events. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp.
1843-4846. IEEE, 2014.
Steven Broll, Jacek Urbanek, David Buchanan, Elizabeth Chun, John Muschelli, Naresh M Punjabi, and Irina Gaynanova. Interpreting blood glucose data with r package iglu. PloS one, 16(4): c0248560, 2021.
Remei Calm, Maira Garcia-Jaramillo, Jorge Bondia, MA Sainz, and Josep Vehí. Comparison of interval and monte carlo simulation for the prediction of postprandial glucose under uncertainty in type 1 diabetes mellitus. Computer methods and programs in biomedicine, 104(3):325- 332-332,, 2011.
Marzia Cescon. Modeling and prediction in diabetes physiology. Department of Automatic Control, Lund University, Sweden, 2013.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. 37th Conference on Artificial Intelligence (AAAI), 2023.
Cheng- Liang Chen and Hong-Woo Wen Tsai. Modeling the physiological glucose- glucose-insulin system normal and diabetic subjects. Computer methods and programs in biomedicine, 9702):330-140, 2010.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. 22nd International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD), pp. 785-794, 2016.
Ana Colás, Luis Vigil, Borja Vargas, David Cuesta-Frau, and Manuel Varela. Detrended fluctuation analysis in the prediction of type 2 diabetes mellitus in patients at risk: Model optimization and comparison with other metrics. PloS one, 14(12)xe0258177, 2019.
Ivan Contreras and Josep Vehi. Artificial Intelligence for Diabetes Management and Decision Support: Literature Review. Journal of Medical Internet Research, 20(5) 2018. ISSN 1439- 4456. doi: 10.2196/10775. URL.https.
/ /wwwwwww.. ncbi wwm.mmmmmm....mmmmmmmmm.. www. htw.ggoopp/pp/.oee/ es/
PMC60004847]
Diego De Pereda, Sergio Romero-Vivo. Beatriz Ricarte, and Jorge Bondia. On the prediction of glucose concentration under intra-patient variability in type diabetes: monotone systems approach. Computer methods and programs in biomedicine, 108(3):993-1001, 2012.
Yixiang Deng, Lu Lu, Laura Aponte, Angeliki M. Angelidi, Vera Novak, George Em Karniadakis, and Christos S. Mantzoros. Deep transfer learning and data augmentation improve glucose levels prediction in type diabetes patients. NPJ Digital Medicine, 4:109, 2021. ISSN 2398- 6352. doi: 10. 11..0008/414446-022 1000200-- httltttps:: : //wp://wwwww. ncbi mnmm n h. gww.gooovvvoooooo /mc./
articles/PMc820112
Fabien Dubosson,
Ranvier, Stefano Bromuri, Jean- Calbimonte, Juan Ruiz, and Michael Schumacher. The open namo dataset: A multi-modal dataset for research on non- invasive type 1 diabetes management. Informatics Medicine Unlocked, 13:92-100, 2018.
Anne Katrine
Signe Schmidt, Rikke Meldgaard Røge, Jonas Bech Møller, Kirsten Nørgaard, John Bagterp Jørgensen, and Henrik Madsen. Model identification using stochastic differential equation grey-box models in diabetes. Journal of diabetes science and technology, 7 (2):431-440, 2013.
Hajrudin Efendic, Harald Kirchsteiger, Guido Freckmann, and Luigi del Re. Short-term prediction of blood glucose concentration using interval probabilistic models. In 22nd Mediterranean conference on control and automation, pp. 1499. IEFF, 2014.
Meriyan Pren-Oruklu, Ali Cinar, and Lauretta Quinn. Hypoglycemia Prediction with Subject-Specific Recursive Sime-S Models. Journal of Diabetes Science and Technology, 4(1):25-333 33, 2010. ISSN 1932-2968. doi: /0.1177/19322968 00022299680000001000 1004400104. URL 19322998 0004001
11

---

Published as a conference paper at ICLR 2024
Qiang Fang, Lei Yu, and Peng Li. A new insulin-glucon metabolic model of type diabetes mellitus: An silico study. Computer methods and programs in Biomedicine, 120(1):16 2611662,2220 2015.
International Diabetes Federation. International Diabetes Federation Diabetes Atlas. International Diabetes Federation, 2021.
Cristian Challú Kin G. Olivares Federico Garza, Max Mergenthaler Canseco. StatsForecast: Lightning fast forecasting with statistical and econometric models. PyCon Salt Lake City, Utah, US 2022, www.
2022. URL http:: // httpp//.www....aaaaattaatatrerr.
Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, and Jenna Wiens. Deep Multi-Output Forecasting: Learning to Accurately Predict Blood Glucose Trajectories. 24th International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD), pp. 1387-1395, 2018.
doi: 10. 10.114//319819.3222 3220102
URL https: //dl wam. www.ooo./o...ooooooo////oooo.o.a 1111//11.1111111111
Eleni Georga, Vasilios Protopappas, Alejandra Guillen, Giuseppe Fico, Diego Ardigo, Maria Teresa Arredondo, Themis P Exarchos, Demosthenes Polyzos, and Dimitrios Fotiadis. Data mining for blood glucose prediction and knowledge discovery in diabetic patients: The metabo diabetes mod- eling and management system. In 2009 annual international conference of the IEEE engineering in medicine and biology society, pp. 5633-5636. IEFF, 2009.
Eleni I. Georga, Vasilios C. Protopappas, Demosthenes Polyzos, and Dimitrios I. Fotiadis. Evaluation of short-term predictors of glucose concentration in type 1 diabetes combining feature ranking with regression models. Medical & Biological Engineering & Computing, 53 105-1318, 2015. doi: 10. 10.1007/s11517-005 1263-111--11611111 12 URL atthttps: http:////o/o...//.111000000
ISSN 1741-0444.
611517-015-1263 1263-1
Péter Gyuk, István Vassányi, and István Kósa. Blood Glucose Level Prediction for Diabetics Based on Nutrition and Insulin Administration Logs Using Personalized Mathematical Models. Journal of Healthcare Engineering, pp. 8605206, 2019. ISSN 2040-2295, doi: 10.1 /055/019//6052000 URL httph::/ //ww: wwb.ncciin.... nmmm nih
Heather Hall, Dalia Perelman, Alessandra Breschi, Patricia Limcaoco, Ryan Kellogg, Tracey McLaughlin, and Michael Snyder. Glucotypes reveal new patterns of glucose dysregulation. PLoS biology, (6(7):e2005143 2018.
Jinli He and Youqing Wang. Blood glucose concentration prediction based on kernel canonical correlation analysis with particle swarm optimization and emror compensation. Computer Methods and Programs in Biomedicine, 196: 105574, 2020. ISSN 0169-2607. doi: 10. 10.1016/j.cppb
2020. 105574. URL https:://// /wtp:://wwwww.. www.ssccieeeeiiieeoooooeoe com/ wwwiiinnee/tiice//pll/ee 5016926072030555
Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasicka, Andrzej Skrodzki, Nicolas Huguenin, et al. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research, 23 (124):1-6, 2022. http::/jmmmt...... wwwp:/pppppppp.oopoo.oo.p..eeeep w233 1177 html
J. Ignacio Hidalgo, J. Manuel Colmenar, Gabriel Kronberger, Stephan M. Winkler, Oscar Garnica, and Juan Lanchares. Data Based Prediction Blood Glucose Concentrations Using Evolutionary Methods. Journal of Medical Systems, 41(9))142, 2017. ISSN 1573-689X. doi: 10.1007/ $10916-017-007882 [ttp://tp.//.aaa.oo///...0////00
Roman Hovorka, Valentina Canonico, Ludovic Chassin, Ulrich Haueter, Massimo Massi-Benedetti. Marco Orsini Federici, Thomas R Pieber, Helga C Schaller, Lukas Schaupp, Thomas Vering, et al. Nonlinear model predictive control of glucose concentration subjects with type diabetes. Physiological measurement, 25(4):905, 2004.
Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for r. Journal of statistical software, 27:1-22, 2008
Diabetes Research in Children Network (DirecNet) Study Group. Use of the direcnet applied treatment algorithm (data) for diabetes management with a real-time continuous glucose monitor (the freestyle navigator). Pediatric diabetes, 9227142 2008
12

---

Published as a conference paper at ICLR 2024
Redy Indrawan, Siti Saadah, and Prasti Eko Yunanto. Blood Glucose Prediction Using Convolutional Long Short-Term Memory Algorithms. Khazanah Informatika Jurnal Ilmu Komputer dan Informatika, 7(2):90 95, 2021. ISSN 2477-698X. doi: 10.23917//khif..7i222. //journal 1 s. ums. ac 1 d/ ndex. httpe//wwwpppp/////aaac/////////
URL https:
Mehrad Jaloli and Marzia Cescon. Long-term Prediction of Blood Glucose Levels in Type 1 Diabetes Using a CNN- -ased Deep Neural Network. Journal of Diabetes Science and Technology. pp. 198229682210022785, 2022. ISSN 1932-2968. doi: 10. 10.117719922962210227555 URL
hittps:/ddos
11.7/ http://pw..orr11111.111116226662
Ioannis Kavakiotis, Olga Tsave, Athanasios Salifoglou, Nicos Maglaveras, loannis Vlahavas, and Ioanna Chouvarda. Machine Learning and Data Mining Methods in Diabetes Research. Com- putational and Structural Biotechnology Journal, 15:104-116, 2017. ISSN 2001-0370. doi:
2..0066/.csbj22011... PMC52570267
https:: /wwwwwww.. ncbi .icbbi.... nlm. nih, htt.g////mc///ariileessss
Hun-Sung Kim and Kun-Ho Yoon. Lessons from Use of Continuous Glucose Monitoring Sys- tems Digital Healthcare. Endocrinology and Metabolism, 35(3):541-544 2020. ISSN 2093-596X. doi: 10.3803/EnM..0020.77. articles/PMc7520S5
https: www ncbi n m. n h. www.goo//ooggoomm /mc//
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. 35th International Conference Machine Learning (ICML), pp. 2796-2804, 2018.
Alejandro Laguna, Paolo Rossetti, F Javier Sapuudia-Blasoo, Josep Vehi, and Jorge Bondia. Experimental blood glucose interval identification of patients with type I diabetes. Journal of Process Control, 24(1):171-181, 2014a.
Alejandro Laguna, Paolo Rossetti, F Javier Ampudia-Blasco. Josep Vehí, and Jorge Bondia. Identification of intra-patt patteetttintt variability in the postprandial response of patients with type diabetes. Biomedical Signal Processing and Control, 12:39 12:39-46,20 2014b.
Saúl Langarica, Maria Seatriguez-Fernandeed Felipe Núñez, and Francis Doyle. A meta-learning approach to personalized blood glucose prediction in type diabetes. Control Engineering Practice, 135:105498, 2023. ISSN 0967-0661.
doi: 10.
1054499 105498. URL https
7/www. www.sscienceiireec.o. http://ccc........o...o.oo//////
com www.sceeeeeaaeeeeeeeeeeeeeeeeeee
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trude, pp. 9-50. Springer, 2002.
ED Lehmann and T Deutsch. A physiological model of glucose-insulin interaction. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society Volume 13: 1991, pp. 2274-2275. IEEE, 1991.
Kezhi Li, John Daniels, Chengyuan Liu, Pau Herrero, and Pantelis Georgiou. Convolutional Recurrent Neural Networks for Glucose Prediction. JEEE Journal of Biomedical and Health Informatics, 24 (2):603-613 613, 2020. ISSN 2168-2208 doi: 10. 20..........99484444
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Plister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37): 1748 1764, 2021.
Chengyuan Liu, Josep Vehi, Nick Oliver, Pantelis Georgiou, and Pau Herrero. Enhaneing Blood Glucose Prediction with Meal Absorption and Physical Exercise Information. arXiv Preprint, (arXiv:1901.07467), 2018. URL/tttp://axxiv http://aaxii.oorraa////aa//
org haagg//bs/9990 19011 07467
Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Mare Breton, Boris Kovatchev, and Claudio Cobelli. The uva/padova type 1 diabetes simulator: new features. Journal of diabetes science and technology, 8(1):26-34, 2014.
Cindy Marling and Razvan Bunescu. The ohiotldm dataset for blood glucose level prediction: Update 2020. In CEUR workshop proceedings, volume 2675. pp. 71. NIH Public Access, 2020.
13

---

Published as a conference paper at ICLR 2024
John Martinsson, Alexander Schliep, Bjöm Eliasson, and Olof Mogren. Blood Glucose Prediction with Variance Estimation Using Recurrent Neural Networks. Journal of Healthcare Informatics Research, 4(1):1-18, 2020. ISSN 2509-498X. doi: 10. (00077/441666-010-00055555 U
//doi.org/10 0. 1007 s 666- 000-66011--000 9-00059-
Nelly Mauras, Roy Beck, Dongyuan Xing, Katrina Ruedy, Bruce Buckingham, Michael Tansey, Neil White, Stuart A Weinzimer, William Tamborlane, Craig Kollman, et al. A randomized clinical trial to assess the efficacy and safety of real-time continuous glucose monitoring in the management of type 1 diabetes in young children aged to; 10 years. Diabetes care, 35(2): 204-210, 2012.
Sadegh Mirshekarian, Hui Shen, Razvan Bunescu, and Cindy Marling, LSTMs and Neural Attention Models for Blood Glucose Prediction: Comparative Experiments on Real and Synthetic Data. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE
Engineering in Medicine and Biology Society, 2019:706-712, 2019. ISSN 2375-7477. doi: 10. 1109/EMBC.2019. 88856940. PMC7890945/1
URL https::///// //wp:://wwwww.. www. wwbi. nlm. nih wtt.hhhgwwm/m///a.ioioo
Mario Munoz- Organero Deep Physiological Model for Blood Glucose Prediction in TIDM Patients. Sensors (Basel, Switzerlandd, 30814) 2020. doi: 10.3390/s20143886. URL htt.hhtps::// www
ncbi. nlm. nih. http/ppneaattiilll//////////////
Natalie Nanayakkara, Andrea Curtis, Stephane Heritier, Adelle M Gadowski, Meda E Pavkov, Timothy Kenealy, David R Owens, Rebecca L Thomas, Soon Song, Jencia Wong, et al. Impact of age at type 2 diabetes mellitus diagnosis on mortality and vascular complications: systematic review and meta-analyses. Diabetologia, 64:275-287, 2021.
C. Novara, N. Mohammad Pour, T. Vincent, and G. Grassi. A Nonlinear Blind Identification Approach Modeling of Diabetic Patients. JEEE Transactions Control Systems Technology, 24(3):1022-1100, 2016. ISSN 1558-0865. doi: 10. 10.119//CST.222122222777
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. 8th International Conference on Learning Representations (ICLR), 2020.
Mwaffaq Otoom, Hussam Alshraideh, Hisham M. Almasaeid, Diego López-de Ipiña, and José Bravo. Real-Time Statistical Modeling of Blood Sugar. Journal of Medical Systems, 39( 19)) 2015. ISSN 1573-689X. doi: 10. 0009166015-0311
URL
Silvia Oviedo, Josep Vehí, Remei Calm, and Joaquim Armengol. A review of personalized blood glucose prediction strategies for TIDM patients. International Journal for Numerical Methods in Biomedical Engineering, 3386), 2017. ISSN 2040-7947. doi: 10.1002/cnm.225 2833. URL
http::://onnineelee
wi wey. com/ htmm//do//b///lln http://abb/....00///...// 12020002/cmmm 2833
Francesco Prendin, Simone Del Favero, Martina Vettoretti, Giovanni Sparacino, and Andrea Facchinetti. Forecasting of Glucose Levels and Hypoglycemic Events: Head-to-Head Compari- son of Linear and Nonlinear Data-Driven Algorithms Based on Continuous Glucose Monitoring Data Only. Sensors, 1165):1647, 2021. ISSN 1424-8220. doi: 10.3390//21051647. URL
https: // www..mpp.. com 424
Maximilian Reymann, Eva Dorschky, Benjamin H. Groh, Christine Martindale, Peter Blank, and Bjoern M. Eskofier. Blood glucose level prediction based on support vector regression using mobile platforms. 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2990-2993, 2016. ISSN 1558-4615, doi: 10.1 10.109//MMB..221...5555555
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for stregularlyssample time series. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Renat Sergazinov, Mohammadreza Armandpour, and Irina Gaynanova. Gluformer: Transformer- based personalized glucose forecasting with uncertainty quantification. IEEE ICASSP, 2023.
14

---

Published as a conference paper at ICLR 2024
Ge Shi, Shihong Zou, and Anpeng Huang. Glucose-trackingg A postprandial glucose prediction system for diabetic self manaagmmnn.. In 2015 2nd International Symposium on Future Information and Communication Technologies for Ubiquitous HealthCare (Ubi-HealthTeeal pp. 1-9. IEEE, 2015.
Bharath Sudharsan, Malinda Peeples, and Mansur Shomali. Hypoglycemia Prediction Using Machine Learning Models for Patients With Type 2 Diabetes. Journal of Diabetes Science and Technology. 91):866990 2015. ISSN 1932-2968. doi: (LL1177119322988445554444 1ttg//0.1177712222533333
URL https: //oo
Qingnan Sun, Marko V. Jankovic, Lia Bally, and Stavroula G. Mougiakakou. Predicting Blood Glucose with LSTM and Bi-LSTM Based Deep Neural Network. In 2018 14th Symposium on Neural Networks and Applications (NEUREL), pp. 1-5, 2018. doi: 10. 10.1109/EERREL.20111 8586990.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147. PMLR, 2013.
William PTM van Doom,, Yuri D Foreman, Nicolaas Schaper, Hans HCM Savelberg, Annemarie Koster, Carla JH van der Kallen, Anke Wesselius, Miranda T Schram, Ronald MA Henry, Pieter C Dagnelie, et al. Machine learning-basee glucose prediction with use continuous glucose and physical activity monitoring data: The maastricht study. PloS one, 16(6)ee2253125, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Youging Wang, Xiangwei Wu, and Xue Mo. A novel addptive--eiighied--vrer framework for blood glucose prediction. Diabetes technology & therapeutics, 15(10):792-801, 2013.
Ruth S Weinstock, Stephanie N DuBose, Richard Bergenstal, Naomi Chaytor, Christina Peterson, Beth A Olson, Medha N Munshi, Alysa JS Perrin, Kellee M Miller, Roy W Beck, et al. Risk factors associated with severe hypoglycemia in older adults with type diabetes. Diabetes Care, 39(4):603-610, 2016.
JM Wójcicki. "j"-index. a new proposition of the assessment of current glucose control in diabetic patients. Hormone and metabolic research, 27(01):41-42, 1995.
Zimei Wu, C-K Chui, G-S Hong, and Stephen Chang. Physiological analysis on oscillatory behavior of glucose-insulee regulation by model with delays. Journal of theoretical biology, 280(1):1-9, 2011.
Charles C Wykoff, Rahul N Khurana, Quan Dong Nguyen, Scott Kelly, Flora Lum, Rebecca Hall, Ibrahim M Abbass, Anna M Abolian, Ivaylo Stoilov, Tu My To, et al. Risk of blindness among patients with diabetes and newly diagnosed diabetic retinopathy. Diabetes care, 44(3):748-756, 2021.
Jinyu Xie and Qian Wang. Benchmarking Machine Learning Algorithms on Blood Glucose Prediction for Type Diabetes in Comparison With Classical Time-Series Models. IEEE Transactions Biomedical Engineering, 67(11)3101-3312, 2020. ISSN 1558-2531. doi: 20.. 2975959.
He Xu, Shanjun Bao, Xiaoyu Zhang, Shangdong Liu, Wei Jing, and Yimu Ji. Blood Glucose Prediction Method Based on Particle Swarm Optimization and Model Fusion. Diagnostics, 12(12): 3062, 2022. ISSN 2075-4418. doi: 10.3390/diagnnsttiss1222222 1230062. URL https:t////// www.. ncbi. n nlm. nih. wwth.govppcccccc http://rticleeee..ee http:///arrrttleee//////333333
Jun Yang, Lei Li, Yimeng Shi, and Xiaolei Xie. An ARIMA Model With Adaptive Orders for Predicting Blood Glucose Concentrations and Hypoglycemia. IEEE Journal of Biomedical and Health Informatics, 235)).125--1260, 2019. ISSN 2168-2208 doi: 10.1109/JB11 2018..2800088
15

---

Published as a conference paper at ICLR 2024
Konstantia Zarkogianni, Konstantinos Mitsis, M-T Arredondo, Giuseppe Fico, Alessio Fioravanti, and Konstantina Nikita. Neuro-fuzzy based glucose prediction model for patients with type diabetes mellitus. IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pp. 252-255, 2014.
Yan Zhang, Tim A Holt, and Natalia Khovanova. A data driven nonlinear stochastic model for blood glucose dynamics. Computer methods and programs in biomedicine, 125:18-25 2016.
Taiyu Zhu, Kezhi Li, Pau Herrero, and Pantelis Georgiou. Personalized blood glucose prediction for type diabetes using evidential deep learning and meta-leaming. IEEE Transactions on Biomedical Engineering, 70(1): 193-204, 2022a.
Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, and Volkan Cevher. Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization). Advances in Neural Information Processing Systems, 35:36094-36107, 2022b.
16

---

Published as a conference paper at ICLR 2024
A DATASETS
Previous works. We summarize previous work on the CGM datasets in Table 7
Table 7: Summary of the glucose prediction models by dataset and model type. We indicate "open" for datasets that publicly available online, and "proprictary" for the ones that cannot be released.
Dataset
Diabetes
Deep
Shallow
Physiological
DirecNet, 2008
Type
30
He & Wang 2020
Eren Fren-Oruklu et al. Balakrishnan et al
2010
20133:
Chen &
sai (010
Anthimopoulos
Type
20
Sun et al.
2018
al. (015))
Mauras et al.
Type
146
Indrawan
et al.
2012)
2021
Georga et al. (0099)
Type I
15
Georga et al. 2015
Marling
&
Type
12
Deng et al. 2021 ;
van
Doorn et
Bunescu (220
van Doorn et al.
2021
2021 Zhu al.
2022a y Martinsson
et al. (022))
Dubosson al.
Type
Munoz-Organee Organero 2020
2018
Aleppo
Type
168
Jaloli (2022)
Cescon
(017))
Type
40
Fox et al. (2018);
2018
Armandpour et al
(2021) Sergazinov
et al 2023)
Cescon 2013
Type 1
59
Jaloii & Cescon 2022
Total (open)
Simulation
NA
NA
Li et al. 2020): ; Lan-
Reymann
et Boiroux
Boiroux
et
al.
garica et al, (2023):
(2016):
(2012
Bock et al.
Liu al. (2018)
al.
2012
Otoom
2015
Calm al.
2015
(2011)
Pereda
ct al. 2012 Fang
ct al. (2015) ; La
guna et al.
20 4b)
Proprictary
NA
1-851
Xu et al. 2022
Prendin
et
al.
Gyuk et al.
2019
Li et al.
2020
222T
1 Yang et al.
Novara et al.
2016
Prendin
2019%
Sudharsan
Bock Duun-Hennksen
2015
(2021):
Alibbrti
al.
(2015): Hr
et al.
(2019); Liu
dalgo et al. (2017):
et al. (2013): La-
et
al.
(2018):
Efendic
al.
guna et al. (201
4a
Ben
Ali
al.
Botwey ct al. (2014): : Wang
(2014):
Wu al. 201
(2018):
Shi al.
Zhang
2016
(0015
ct al.
Zarko gianni et al. (0014)
2013
Total (propri- etary)
13
Access. The datasets are distributed according to the following licences and can be downloaded from the following links:
17

---

Published as a conference paper at ICLR 2024
Broll 2022
License: GPL-2
Source: link
2. Colás et al. (2019)
License:
Creative Commons 4.0
Source:
link
Dubosson al. (0088
License:
Creative Commons 4.0
Source:
link
4. Hall et al. (2018
License:
Creative Commons 4.0
Source: link
Weinstock et al.
2016
License:
Creative Commons 4.0
Source:
link
Covariates. We summarize covariate types for each dataset in Table 8 For each dataset, we extract the following dynamic known covariates from the time stamp: year, month, day, hour, minute, and second (only for Broll). Broll provides covariates aside from the ones extracted from the time stamp. Colas, Hall, and Weinstock provide demographic information for the patients (static covariates). Dubosson is the only dataset for which dynamic unknown covariates such heart rate, insulin levels, and blood pressure are available.
Table 8: Covariate information for each dataset.
Covariate
Broll
Colas
Dubosson
Hall
Weinstock
Age
Static
Height
Total
0
48
38
Year
V
Dyn.
Kn.
Month
/
Total
5
Unkn.
Insulin
/
Dyn.
Heart Rate
Total
11
0
0
18

---

Published as a conference paper at ICLR 2024
B ANALYSIS
B.I VISUALIZED PREDICTIONS
We provide visualized forecasts for the same 5 segments Weinstock et al. (2016) data for the best performing models: linear regression, Latent ODE (Rubanova et al.) 2019). and Transformer (Vaswani et al. 2017) on Task I (accuracy), and Gluformer (Sergazinov et al. 2023) and TFT (Lim) 2021 Task (uncertainty). For the best models on Task 2, also provide the estimated confidence intervals or the predictive distribution, whichever is available. For visualization, we have truncated the input sequence to 1 hour (12 points); however, note that different models have different input length and require at least hours observations to forecast the future trajectory.
105.
102.5
102.5
100.0
170
230
140
97.5
97.5
220
96.0
95.0
150
92.5
90.0
100
130
200
180
102
105.0
100
(mgalt)
160
102.0
320
230
140
95
97.5
220
150
120
100
90.0
130
200
189
100
TRANSTORMER
180
160
230
140
160
129
90
90
100
38
300
180
GLUFORMER m113
Glucos (mgilt)
260
160
11.0
130
260
140
100
100
100
120
80
340
1.0
30
10
10
5
10
5
5
5
10
5
220
360
Ciscose (mgilt)
280
1.0
130
300
120
240
1.0
100
120
200
100
80
-10
-5
10
-10
-5
-10 -5
10
-10
10
-10
Time minute Intervats)
Time minute Intervals)
Time (In minute Intervals)
Time (In minute Intervals)
Time (in minute Intervals)
Figure 5: Model forecasts on Weinstock et al, (2016) dataset.
B.2 PERFORMANCE
For reference, we include results for all models both with and without covariates evaluated on ID and OD splits in Table on Task (accuracy) and in Table 10 Task 2 (uncertainty quantification).
19

---

Published as a conference paper at ICLR 2024
Table 9: Model results the data sets for Task (accuracy).
p(|2))
Data
Colas
Dubosson
Hall
Weinstock
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
ARI
X
ID
10.53
8.67
5.80
4,80
13.53
11.06
8.63
7.34
13.40
11.25
x
OD
11.75
9.71
5.91
4.87
18.75
14.58
8.22
6.97
15.87
13.34
min (ID, ODY%
+11.8%
+1.75%
+35. 18%
-4.94%
+18.51%
X
ID
11.68
9.71
5.26
4.35
12.07
9.97
7.38
6.33
13,60
11.46
ID
9.95
8.41
5,56
4.60
12.41
10.03
7.84
6.66
13.39
11.34
LIN
Improv:
-14.08%
+5.65%
+1.3%%
+5.63%
-1.31%
X
OD
11.98
9.83
5.33
4.41
15.69
11.90
7.86
6.62
15.58
13.09
OD
23.30
16.80
5.54
4.57
203114.47
67548.59
14.22
10.02
15.66
13.16
Improv.
+82.7%
+3.8%
+930929.9999
+66.07%
+0.54%%
min AID, ODY%
-0.45%
71%
+5.52%
+14.36%
X
ID
12.80
11.50
6.42
5.49
21.18
19.09
7.58
6,55
13.63
11.61
13.89
11.87
6.37
5.46
20.89
18.55
8.05
7.02
13.77
11.77
XGB
Improv.
+5.88%
-0.61%
-2.08%
+6.67%
+1.18%
X
OD
9.76
8.72
6.18
5.32
17.57
15.42
7.49
6.52
15.36
13.04
OD
9.67
8.56
6.36
5.47
17.44
15.46
8.20
7.11
15.55
13.43
Improv.
-1.37%
+2.91% 1
-0.26%
+9.25%
+2.09%
min AIDD ODY%
-29. 19%
-3.47%
-18.11%
-0.84%
+12.51%
GLU
X
ID
14.19
12.55
8.17
7.12
21.74
19.40
7.74
6.69
14.07
12.09
X
OOD
16.70
14.82
6.94
6.03
23.48
20.70
8.17
7.04
15.94
13.65
min (ID,, ODY%
+17.85%
15.17%
+7.37%
+5.39%
+13.08%
LAT
X
ID
14,37
12.32
6.28
5.37
20.14
17.88
7.13
6.11
13.54
11.45
OOD
14.96
13.05
5.64
4.84
15.12
7.71
15.06
12.72
min A(ID, ODY%
+5.03%
-9.94%
14.559%
+8. 12%
+11.12%
X
ID
13.79
12.07
5,93
5.04
17.45
14.79
7,68
6.57
13,29
11.21
/
ID
16.20
14.64
9.09
8.03
30.43
27.97
8.16
7.10
13.41
11.31
NEN
NYN
Improv.
+19.36%
+56.31%
+81. +81174%%
+7.13%
+0.9%
X
OD
14.64
12.77
5.68
4.83
18.20
15.59
7.74
6.62
14.52
12.24
OD
15.66
14.01
7.56
6.65
37.35
33.52
8.59
7.53
14.40
12.12
Improv.
+8.35%
+35.49%
+110.09%
+12.34%
-0.91%
min AID, ODY%
-3.8%
-17.01%
+4.86%
+0.78%
+7.29%
ID
13.73
11.07
5.62
4.54
18.37
15.49
7.92
14.32
11.76
14.68
12.43
6.51
5.27
18.43
15.51
8.42
7.06
14.97
12.30
Improv.
+9.53%
+15.9%
+0.22%
+6.52%
+4.55%
X
OD
12.43
10.23
5,51
4.47
17.50
14.53
8.12
6.76
15.25
12.50
OD
13.25
11.17
5.79
4.68
17.19
14.43
8.93
7.44
15.47
12.67
Improv.
+7.91%
+4.84%
-1.22%
+10.01%
+1.41%
min AID, ODY%
.9.91%
-11.11%
-6.84%
+2.41%
+3. 18%
X
ID
15.12
13.20
6.47
5.65
16.62
14.04
7.89
6.78
13.22
11.22
/
ID
12.83
11.27
8.44
7.77
27.43
24.40
7.49
6.42
14.46
12.61
TRA
Improv.
-14.89%
+33.93%
+69.419%
-5.23%
+10.87%
OD
14.04
12.28
5.97
5.24
15.71
12.98
88
7.07
14.15
11.91
OD
13.76
13.13
7.26
6.59
34.11
28.21
7.40
6.29
15.59
13.58
Improv.
-1.59%
+23.61%
+117.27%
-10.23%
+12.1495 14%1
min AID, ODY%
-7.06%
-14.62%
-6.52%
-1.57%
+6.58%
B.3 FEATURE IMPORTANCE
Based on the performance results reported Tables 9 Task 1 (accuracy) and in Table 10 XGBoost (Chen & Guestrin 2016) is the only model that consistently works better with inclusion of extraneous covariates, improves in accuracy on out datasets and uncertainty quantification on out of datasets. TT lists the top 10 covariates selected by XGBoost for each dataset. For time-varying
20

---

Published as a conference paper at ICLR 2024
Table 10: Model results the data sets for Task 2 (uncertainty quantification).
Broll
Colas
Duhossoo
Hall
Weinstock
K(la
Data
ldk.1
Cal.
Lak.t
Col.1
Idk.1
Cal.
[ike
Cal.
Ldk.1
Cal.
ARI
ID
-9.3
0.11
-9.30
0.10
-10.47
0.10
-9.1
0.10
-10.21
0.12
OOD
-10.06
0.07
-9.38
0.08
-10.44
0.08
-9.66
0.07
-10.32
0.12
min AID, OD!
-1.30%
36.06%
-0.00%
19.95%
40.28%
-2.07%
+1.52%
29.922
-107%
40.06%
ID
-9.89
0.12
-9.19
0.15
-10.10
0,18
-9.56
0.10
-10.14
0.11
ID
-9.87
0.13
-9.17
0.19
-10.15
0.21
-10330
0.19
-00.12
0.11
LIN
Improv.
+0.15%
+5.99%
+0.25%
+24.47%
-0.41%
+11.39%
-7.67%
+97.36%
+0.13%
-1,67%
OD
-9,95
0.15
-9.16
0.15
-10.11
0.17
-9.53
0.10
-10.22
0.11
OD
-10.24
0.55
-9.16
0.17
-12.08
0.48
-10.42
0.23
-11.13
0.21
Improv.
-2.88%
+256.79%
+0.01%
+10.19%
-1.52%
-181.06%
-9.26%
+130.96%
-8.92%
+87.0%
min AID, ODVS
-0.65%
-44.98%
+0.33%
8.88%
00005
7.62%
+0.35%
+4.43%
0.85%
-311%
ID
-9.94
-9.42
0.30
10.55
0.07
-9.68
0.09
10.20
0.11
ID
-10.06
0.07
-9.40
0.09
10.54
0.06
-9.70
0.09
10.21
0.10
Improv.
-1.22%
+0.59%
+0.12%
-7.2%
+0.13%
6.98%
40.1%
-1.3%
-0.1%
4.62%
OD
-10.05
0.11
-9.66
0.09
-10.22
0.07
-9.56
0.08
-10.28
0.11
OD
-10.03
0.11
-9.38
0.08
-10.20
0.07
-9.53
0.10
-10.31
0.10
Improv.
-0.04%
+1.75%
-0.2%
-7.0%
+0.13%
-1.62%
+0.31%
+21.99%
-0.34%
-4.89%
min (ID. OD1S
+0.29%
+67.42%
+0.59%
-1.55%
+3.17%
+5.02%
+1.77%
14.37%
-0.83%
+2.16%
ID
-2.11
-1.07
0.14
2215
0.06
-1.56
0.05
-250
0.08
OOD
-1.96
0.11
-1.61
0.10
-1.17
0.12
-1.44
0.06
-2.41
0.09
MID, ODY
+6.72%
1006.70%
-5.33%
29.83%
64%
+83.63%
+7.69%
+9.24%
33%
+6.23%
ID
25.29
0.36
10.47
0.25
-52.18
0.42
20.24
0.30
26.15
0.33
OOD
-28.75
0,38
-8.80
0.24
-30.19
0.44
-18.19
0.36
-30,08
0,40
min AID, ODVS
-3.667
+6.5%
+15.99
-101%
+42.44%
+10.12%
+20 58%
15.03%
20.72%
ID
-10.01
0.12
-932
0.11
-10.37
0,10
-9.62
0.09
-10.13
0.11
ID
100.37
-9.48
-10.80
0.08
-9.63
0.07
-10.13
WII
Improv.
-3.63%
-3777%%
-1.88
+91.12%
-4.19%
21.68%
-0.07%
24.77%
-0.01%
-5.46%
OD
-10.08
0.10
-9.26
0.11
-10.18
0.12
-9.49
0.08
-10.20
0.12
OD
-10.21
0.06
-9.66
0.14
-11.10
0.20
-9.58
0.06
110.19
0.11
Improw.
-1.3%
34.64%
-117%
+24.578
99.0%
+66.46%
.0.4%%
14.44%
+0.14%
8.1%%
min AID, ODV
+1.55%
-66.3%
+1.23%
-34.00%
+1.76%
+15.16%
+1.88%
14.79%
-0.55%
+4.17%
ID
0,16
0.07
0,23
0.07
0.07
0.30
0.16
0.08
0.06
Improv.
-%
+94.6%
-56
+114.61%
-9
+7.57%
-%
+16.44%
-%
-21.55
OD
0.09
0.26
0.08
0.08
OD
0.23
0.09
0.35
0.08
0.05
Improw.
-%
+57.74%
-%
+0.35%
-%
+37.5%
-%
-1.4%
-%
-35.43%
min (ID,, ODYs
-22.83%
-%
14.
-6
+10.56%
+9.88%
-66
9.51%
ID
-9.99
0.23
9.37
0.21
-10.36
0,12
-9.60
0.13
10.12
0.11
-10.11
.9.5
0.31
10.668
0.18
-9.60
0.10
10.115
TRA
Improv.
-1.21%
-6.84%
00.9%
+45.69%
-3.05%
+48.29%
+0.05%
-2.4%%
-0.34%
+0.16%
OD
.9.88
-9.30
0.22
-10.09
0.14
-9.47
0.15
-10.17
0.12
OD
-10.02
0.11
-9.36
0.22
-10.63
0.25
-9.49
0.08
-10.20
0.12
Improw.
-0.41%
-3.288%
-0.65%
+1.0%
-5.32%
+73.94%
-0.16%
45.49%
-0.33%
+2.63%
min A/D,, ODYs
+0.93%
47.95%
+0.92%
29.49%
+2.59%
+17.77%
+1.34%
11.335
-0.3%%
+8.99%
features, such as the 36 heart rate observations in Dubosson, the maximum importance across the input length is considered as the feature importance. Below, we provide a discussion on the selected features.
Among features available for all datasets, dynamic time features, such as hour and day of the week, consistently appear in the top important features across all datasets. This could serve as indication that patients tend to adhere to daily routines; therefore, including time features helps the model to predict more accurately. At the same time, patient unique identifier does not appear to be important, only appearing in the top 10 for Broll (Broll only has 7 covariates in total) and Colas. This could be indicative of the fact that differences between patients is explained well by other covariates.
Dynamic physical activity features such heart rate and blood pressure only available for Dubosson. Based on the table, we see that medication intake, heart rate and blood pressure metrics, and physical activity measurements are all selected by XGBoost highly important.
21

---

Published as a conference paper at ICLR 2024
Demographic and medical record information is not available for Broll and Dubosson. For the rest of the datatsets, we observe medication (c.g. Vitamin D. Lisinopril for Weinstock), disease indicators (e.g. Diabetes T2 for Colas, Osteoporosis for Weinstock), health summary metrics (Body Mass Index for Colas), as well as indices derived from CGM measurements (e.g. J-index (Wójcicki. 1995) being selected as highly important.
Table 11: Top-10 10 features with importance weights selected by XGBoost for each dataset.
Corotio
Irpertaioe
Cwarto
Imppotsnc
Couatas
Impwrrznee
Conoras
Irportscc
Ubwarter
Imporzase
Des of veck
5/01 141
Day wokk
000233
Blu
0,003990
Malar CGM
0.00911
Day of wook
5
Sacual
6/005556
Glyonnia
0000
boun Narabiliyy bodes
03903355
lideo
0.001211
Heur
20079
Miszic
C1883110
Dabone
02189122
Poster
EU903266
High
CJ811576
03881194
Polkee
0/00072
Plaice
0.000154
Antsy
00902552
Fieq Lowe CCM
0.001153
Breaik dy Sttccciionn
200055
Body Mars laces (BMI)
Heat Rtac
EU802597
S Pro-Dubeaee CUM
Cremx dacae
038021-3
Age
00000055
Blad Presure
00001130
Coeflicke of COM Vartalie
0.00775
Mear Checka Day
2/00137
B.4 STABILITY
Reproducible model performance is crucial in the clinical settings. In Table 16
we report standard
deviation of MAE across random data splits. As expected, the smallest datasets (Broll et al. (2021) and Dubosson al. (2018)) have largest variability. The number of patients in Broll et al. (2021) and Dubosson et al. 2018) is 5 and 9, respectively, thus randomly selecting subject for OD test set has
large impact on the model performance as the training set altered drastically.
Prior works on deep learning has found that initial weights can have large impact on the performance (LeCun al., 2002; Sutskever al.) 2013] Zhu et al. 2022b). Therefore, we re-run each deep learning model 10 times with random initial weights for each data split and report the average. We also report standard deviation of deep learning model results across random model initializations (indicated in parentheses). We find that good initialization indeed matters as we observe that the results differ across re-runs with different starting weights. Such behavior could be undesirable in the clinical settings the model training cannot be automated. The Transformer is the only robust deep learning model that consistently converges to the same results irrespective of the initial weights, which is reflected in near 0 standard deviation. At the same time, Transformer- based models such as Gluformer and TFT do not exhibit this feature.
We include standard errors of each metric: RMSE (Task 1) in Table 12 MAE (Task 1) in Table I6 likelihood (Task 2) in Table 14, and calibration in Table 15 For deep learning models, there are sources of randomness: random data split and model initialization. Therefore, report two values for standard deviation: one across data splits (averaged over model initializations) and one for model initialization (averaged across data splits).
B.5 DAYTIME VERSUS NIGHTTIME ERROR DISTRIBUTION
We provide daytime and nighttime error (MAE) distribution for all models and datasets in Figure 6 In general, we note that for larger datasets (Colas and Weinstock), the difference in daytime and nighttime error distribution appears smaller.
22

---

Published as a conference paper at ICLR 2024
Table 12: Standard error of MSE across data splits and model random initializations.
p((x)
Data
Broll
Dubosson
Weinstock
ARE
ID
110.90 21.95
33,60 0.68
183.11 40.58
74.52
179.54 +- 3.14
0D
138.08 +- 52:73
34.97 5.65
351.53 + 227.93
67.55 +- 25.18
251.97 + 18.59
ID
136.49 +- 13.04
27.70 0.33
145.65 +- 50.12
54.51 +- 1.67
185.04 +- 2:53
LIN
ID
99.04 + 7.45
30.86 0.71
154,04 + 32.62
61.45 + 3.42
179,38 + 2.56
OD
143.43 4- 50.18
28.41 4- 2.55
246. 19 148, 20
61.78 4- 17.90
242.19 4- 21.51
OD
542.88 617.76
41255489546000
825 1251777188.777
9119211
20005
ID
163.85 6.84
41.23 1,08
448.43 28.93
97,45.41.55
185.87 2.30
XGB
ID
192.97
436.29
64.82
189.50 2.7.8
OD
1- 0.777
+-
46.15
23605 1.27
OD
90.56 + 5.14
40.43 3.09
301.16 + 9.07
67.22 5.03
241.65 + 3.12
GLU
ID
201.57 + 1.24 (20.19)
66.69 + 1.32 (5.83)
472.51 + 43.48 (56.73)
59.98 0.47 (1.92)
198,06 + 4.93 (5.01)
OD
278.74 + 6.65 (10.23)
48,10 + 2.06 (3.13)
551,14 + 175.34 (56.50)
66,74- (3.37)
254.05 + 8.37 (10.83)
LAT
ID
206.55 + 18.61 (75.95)
39.38 + 0.55 (3.5(1)..0)) (1.36)
405.47 + 39,58 (68.69)
50.84 0.59 (1.15)
183.46 0.70 (1.84)
OD
223.91 21.59 (67.04)
31,86 0, (1.42)
301.97 +- 104.32 (71-2)
59.38 1,66 (1.57)
226.72 + 14.59 (7.39)
ID
190.20 8.89 (9.07)
35,20 0.52 (0.26)
304.80 63.70 (1.16)
59.02 0.61 (0.27)
176.50 4.41 (L.95)
NI
(26.54)
82.55 21 48
926.26 183.16
0.73
(0.399
QD
214.25 6.19 (10.83)
32.22
(0.15)
331.18+ 128.00 (2.35)
3.90 (0:21)
210.94
(2.16)
OD
245.25 7-3667 (37.93)
57.18
13.68 (4.05)
1395.29
(67.57)
73.77
2.47 (1.7())
207.37
(0.95)
ID
188.68 47.62 (125,97)
31,58 + 0.79 (1.50)
337,31 + 6.85 (43.82)
62.66 (3.633.63)
205 19 10.63 (13.71)
TET
ID
215,41 4-5.0 (32.20)
42.39 4- 0.36 (0L00)
339.65 +- 0.71 (35.06)
70.83 (3.25)
224, 6.01 (5.91)
OD
154,46 23.5 (66.2)
30.40 + 1.41 (1.861
306. 19 58.47 (46.48)
65.95 6.84 (4.04)
232. 66 26. 29 (14.68)
OD
175.62 16.08 (20.39)
33.55 1.0 (0.00)
255.58
18.26 (23.85)
79.73
(5,17)
239,46 +- 17.02 (5.88)
TD
228.61
66.99 (0.00)
41,92 3.73 (0.00)
276.33 39,70 (0.00)
62.22 0.99 (0.00)
174.87 13.02 (0.00)
TRA
ID
164.85
(0.00)
71.18+ 0.35 (0.00)
154,38 (0.00)
L.8 (0.00)
209.02
(0.00)
197.08
113.45
300.25
OD
189.34 4- (0.00)
52.66 (0.00)
1163.43 672.662228888 (0:00)
54.77 2.65 (0:00)
243.19 4- (0.00)
Table 13: Standard error of MAE across data splits and model random initializations.
p(-(z)
Data
Broll
Colas
Dubosson
Hall
Weinstock
ARI
ID
8.67 0.74
4,80 + 0.04
11.06 + 0.98
7.34 + 0.16
11,25 0.10
OD
9.71 = 1.93
4.87 = 0.38
14.58 +: 4.75
6.97 + 1.19
13.34 = 0.52
X
ID
9.71 1 0.37
4.35 1 0.03
9,97 1.00
6.33 1: 0.09
11.46 1 0.11
LIN
ID
8.41 0.24
4.60 0.04
10.03
11.34
X
OD
9.83 133++1.55 1.58
4,41
11.90 + 3,51
6.91
13.09
OD
16.80 + 9.45
4.57 0.19
67548.59 + 135072.57
10.02 - 6.68
1351
ID
11.50 + 0.31
5.49
19.09 + 0.32
0.09
11.61 0.08
XGB
ID
11.87 1: 0.24
5.46
18.55 + 0.82
7.02
11.77 0.16
OD
8.72 0.45
5.32
15.42
6.12
13.04 0.04
OD
8.56 + 0.30
5.47
15.46 +: 0.35
0.26
GLU
ID OD
12.55 = 0.07 (0.67)
2.12= (0.37)
19.40 + 1.11 (1.27)
6.69 = 0.03 (0.11)
12.09 - 0.17 (0.18)
14.82
0.26 (1.09)
(0.21)
20.70
(1.05)
7.04 (0.18)
13.65 0.22 (0.32)
LAT
ID
12.32 = 0.60 (2.15)
5.37 = 0.04 (0.12)
17.88 +: 0.85 (1.59)
6.11 = 0.04 (0.08)
11.45 + 0.01 (0.19)
OD
13.05 1 0.59 (1.87)
4.84 0.05 (0.13)
15.12 1: 2.76 (1.92)
6.61 0.12 (0.10)
12.72 1 0.40 (0.25)
ID
12.07 0.33 (0.31)
(0.02)
14.79 1 1.60 (0.05)
6.57 0.03 (0.02)
11.21 0.16
NHI
ID
14.64
1.57 (0.81)
8.03 1.40 (0.36)
27.97 = 3.06 (0.57)
7.10 = 0.03 (0.07)
11.31 0.09 (0.02)
OD
12.77
= 0.18 (0.35)
4,83 = 0.01 (0.02)
15,59 + 3.33 (0.04)
6.62 1 0.19 (0.02)
12.24 1 0.25 (0.07)
OD
14.01 2.32 (1.03)
6.65 1.03 (0.27)
33.52 10.07 (0.83)
7.53 0.19
12.12 1 0.37 (0.03)
ID
11.07 1 1.17 (2.85)
(2.85)
4.54 = 0.05 (0.12)
15.49 1 0.05 (1.23)
6.61 1 0.12 (0.20)
11.76 1 0.35 (0.43)
IFT
ID
12,43
(0.89)
(0.00)
15,51
(0.83)
7,06
(0.19)
12.30 0.15 (0.17)
OD
10.23
(1.91)
0.11 (0.15)
14,53 + 1.26 (1.11)
6,76 = 0.34 (0.21)
12.50 + 0.77 (0.45)
OD
11.17 = 0.55 (0.59)
4.68 = 0.12 (0.00)
14.43 + 0.36 (0.63)
7,44 0.65 (0.26)
12.67 = 0.43 (0.17)
ID
13.20
1: (0.00)
(0.00)
14.04
(0.00)
6.78
(0.00)
11.22
(0.00)
TRA
ID
11.27
+ 0.45 (0.00)
7.77 - 0.15
24.40 + 2.69 (0.00)
6.42 + (0.00)
12.61 + 0.43 (0.00)
OD
12.28
= 0.83 (0.00)
5.24 = 0.28 (0.00)
12.98 + 2.91 (0.00)
7.07 = 0.14 (0.00)
11.91 + 0.72 (0.00)
OD
12.13
0.03 (0.00)
6.59 0.36 (0.00)
28.21 - 8.72 (0.00)
6.29 = (0.00)
13.58 = 0.06 (0.00)
23

---

Published as a conference paper at ICLR 2024
Table 14: Standard error of likelihood across data splits and model random initializations.
p(-(a)
Broll
Colas
Duhosson
Hall
Weinstock
ART
ID
.9.33 0.02
-9.30 1 0.05
-0.47 1 0.35
9.81 1 0.15
-10.21 0.02
OD
-10.06 + 0.05
-9.38 = 0.04
-10.44 = 0.03
-9.66 0.56
-10.32 0.05
-9.99 0.01
-9.19 0.01
-10.10 0.16
0.05
-10.14
LIN
ID
-9.87 + 0.03
-9.17 0.01
-10.15 = 0.17
-10.30 1.47
-10.12
0.14
-9.16 0.06
-10.11 0.28
0.17
0.0.22
OD
-10.24 1: 0,30
-9.16 0.06
-12.08 3.94
-10,42 1.49
-11.13 11.13:11 1.83
.9.94 0.02
4.02
-10.55
-9.68 9.68+0001
-10.20 0.00
XGB
ID
-10.06 + 0.06
-9.40
-.0.54
-9,00
-10.21 0.00
-10.03
-9.3
+0.222
9.56
-10.28 0.00
OD
-10.03 + 0.01
9.38
-10.00
-9.33
-10.31 0.00
LAT GLU
2.11 1 0.10 (0.24)
-1.07 1. 0.111 (0.19)
22.15 1.00 (0.22)
-1.56 1 0.00 (0.10)
2,50 1 0.02 (0.05)
-1.96 + 0.08 (0.27)
-1.61 + 0.03 (0.
-1.17 1.53 (1.69
-1.44
(0.1))
-2.41 + 0.01 (0.05)
-25.29 # 1.96 (5.68)
-0047 (0.16)
-52.18 (9.54)
-20.24 = 0.39 (0.19)
-26.15 = 0.03 (0.07)
2:31 (6.38)
0.25 (0.12)
-30.19
(5.20)
-18.19 0.46 (0.16)
-30.08 1.49 (0.14)
TD
-10.01 0.0.1 (0.01)
-9.32 + 0.01 (0.00)
-10,37 + 0.04 (0.00)
-9.62 0.01 (0.00)
-10.13 + 0.00 (0.00)
NHI
-10.37 0.07 (0.05)
-9.88 0.10 (0.02)
-10.80 (0.01)
9..33 (0.01)
-10.13 0.00 (0.00)
OD
-10.08 + 0.00 (0.01)
-9.26 0.01 (0,00)
10,18 (0.00)
-..9 (0.00)
-10.20 0,03 (0.00)
-10.21 + 0.13 (0.04)
0.36
-11.10 (0.04)
9.08 (0.01)
-10.19 0.03 (0.00)
-999 + 0.09 (0.00)
-9.37 + 0.04 (0.00)
-00,36 + 0.04 (0.00)
9.00 (0.00)
-10,12
0.00 (0.00)
TRA
ID
-10.11 4 0.11 (0.00)
-9,55 + 0.00 (0.00)
10.68 + 0.08 (0.00)
9.60 0.00 (0.00)
100.15 0.00 (0.00)
-9.8 + 0.03 (0.00)
99.00 (0.00)
100009 0.06 (0.00)
9.02 (0.00)
-10.17 0.03 (0.00)
OD
-10.02
+ 0.01 (0.00)
9936 0.01 (0.00)
-10.63 + 0.27 (0.00)
99.49 (0.00)
-10.20 0.03 (0.00)
Table 15: Standard error of calibration error across data splits and model random initializations.
p(-lar
Data
Broll
Colas
Dubosson
Hall
Weinstock
ARI
ID
0.11 + 0.01
0,10 = 0.01
0,10 + 0.01
0,10 + 0.02
0,12 = 0.01
OD
0.07 0.02
0.08 0.01
0.08 +: 0.05
0.07 0.01
0.12
0.12 0.01
0.15
0.02
0.10 0.00
0.11
LIN
ID
0.13 0.02
0.19 0.01
0.21 + 0.02
0.29
0.11 0.00
OD
0.15 + 0.04
0.15 + 0.01
0,17 + 0.02
0.10 0.02
0.11 = 0.00
OD
0.55 + 0.39
0.17 0.02
0.48 - 0.58
0.23
0,21 0.20
ID
0.07 0.01
0.10 0.00
0.07 + 0.01
0.11 = 0.00
XGB
ID
0.07 0.01
0,09 0.01
0.06
0.00
0.10
OD
0.11 0.01
0.09 1 0.00
0.07 + 0.01
0.08
0,11 0.00
OD
0.11 + 0.01
0.08 0.01
0.07 + 0.01
0.10 0.01
0.10 0.00
LAT GLU
ID
0.05 + 0.01 (0.01)
0.04 (0.03)
0.06 = 0.00 (0.02)
0.05 + 0.00 (0.01)
0.08 0.00 (0.01)
0.11 + 0.01 (0:03)
0.10 (0.02)
0.12
(0.05)
+ 0.01 (0:01)
0.09 0.00 (0.01)
ID
0.36 +: 0.03 (0.05)
0.25 (0.03)
0.42 0.02 (0.03)
0.30 +: 0.01 (0.02)
0.33 (0.02)
OD
0.38 0.01 (0.05)
0.02 (0.03)
0.44 =:: 0.11 (0.08)
0.36 4 0.04 (0.03)
0.40 (0.03)
ID
0.12 4: 0.02 (0.01)
0.11 0.00 (0.00)
0.10 + 0.00 (0.00)
0.09 + 0.01 (0.00)
0.11 +.00 (0.00)
NHI
ID
0.07 + 0.01 (0.01)
0.21 0.03 (0.04)
0.08 + 0.04 (0.02)
0.07 + 0.01 (0:00)
0.00 (0.00)
OD
0.00
0.00 (0.00)
0.12 : (0.00)
0.01 (0:00)
0.12 (0.00)
OD
0.06 0.01 (0:02)
0.14 (0.03)
0.20 + 0.07 (0.04)
0.06 + 0.01 (0:01)
0.10
ID
0.16 0.06
0.07 0.02 (0.03)
0.23 = 0.07 (0.10)
0.07 + 0.02 (0.02)
0.033..3300033
TPF
ID
0,30 +: (0.12)
0.16 (0.03)
0.25 1 0.03 (0.09)
0.08 1 0.01 (0.02)
0.06 0
OD
0.15 4 0.08 (0:09)
0.00 (0.04)
0.26 1 0.04 (0.13)
0.08 0.01 (0.03)
0.08 10003 (0.04)
OD
0.23 1 0.05 (0.10)
0.09 (0.02)
0.35 0.04 (0.10)
0.08 0.01 (0.02)
0.05
ID
0.23 + 0.07 (0.02)
0.09
0.12 0.01 (0.00)
0.13 + 0.01 (0.00)
0.11 0.01 (0.00)
TRA
ID
0.21 4 0.05 (0.02)
0.31 0.11 (0.04)
(0.02)
0,18 : 0.04 (0.01)
0.10 + 0.00
0.00
OD
0.19 + 0.03 (0.01)
0.14 + 0.03 (0.01)
0.15 + 0.01 (0.00)
0.00
OD
0.11 4 (0.0)
0.22 (0.02)
0.25 0.09 (0.03)
0.08 + 0.01 (0.00)
0.0 (0.00)
24

---

Published as a conference paper at ICLR 2024
Lineer Hegression Broll Night Errors
Linear Regresiion Coles
Linwer Regrassion Dubosson
6.34
Lineer Begnesiin Hall
2.56
Linwrr Regressim- Weinslock
6.35
3.0
0.5
6.35
0.00
3.55
6.08-
e:21
0.2
032
10
20
10
50
25
T:0 152
8:00
10
100 120
0.0
50
101
151
300
252
AGBomsi- Broil
XG28ooat- Colls
XLSooatt Debosssion
XGBouat- hal
XGBBooat- -Weistock
0.22
0.86
6.05
433
0.55
6.35
3.00
062
0:54
6:31-
380
50
30
$50
50
100 325 150
0:00
20
60
60
100 122
0.00
3300-25550 7007
Guufermer Broll
Gutomer- Colas
Guuformer Dubosson
Glutermer- Hall
Gutomme Welndtock
0:55
0.05
0.6-
0.2
0.55
6.00
332
6.32
3.3
3.2
6422
0.34
6:21-
381-
8:05
3.00
41
57
0000
193
150
205
6.00
60
0.00
Latent ODE Broil
Latent ODE- Coles
Latent ODE Dubosson
Latent ODE -all
Latent ODE -Weinsook
0.2
0.14
Denaty
0.34
6:25
3.30
3.55
1.6
6.10
0.54
162
6:00
033
6:21
100
3.00
10
8:0
87 1691.2
6.00
190
120
0.00
310
150
200
TFT Broll
0:335
005
TFT Dubosson
6.56
TFT Hall
3.6
We hotock
Denaity
0:25
6.00
0.6
0.5
4.2
013
6.00
0.00
60
0.000
33
20
110
206120143
100
120
030--
130122 100155000
NHTS Dubosson
Hall
Wainstock
032
5.6
0.00
536
3.5
162
033
6:35
532
20
30
166
320
40
53
210
150
Transformer- Broll
Tranformer Colos
Trandfomer.
Transfommrr Hall
Tranfforme Welnstock
3.56
0:35
0.2
8.00
Dennit
6:35
052
0.00
052
50
3.22-
46 Accurscy (MAE)
30 20
220 20046
43
60 360
1003 2530035526 Accuracy (MAI)
Accuracy (MAE)
Accuracy (MAC)
Accuracy (MAE)
Figure 6: Distribution of daytime (9:0AAM 9:00PM) versus nighttime (9:00PM 9:00AM) errors (MAE) for models with no covariates on the ID set.
25

---

Published as a conference paper at ICLR 2024
C REPRODUCING RESULTS
C.1 COMPUTE RESOURCES
We conducted all experiments on single compute node equipped with 4 NVIDIA RTX2080Ti 12GB GPUs. We used Optuna (Akiba et al., 2019) to tune the hyperparameters of all models except ARIMA and saved the best configurations in the config/ folder of our repository. For ARIMA, we used the native hyperparameter selection algorithm (AutoARIMA) proposed in Hyndman & Khandakar (2008). The search grid for each model is available in the lib/ folder. The training time varied depending the model and the dataset. We trained all deep learning models using the Adam optimizer for 100 epochs with early stopping that had a patience of 10 epochs. For AutoARIMA, used the implementation available in FFdeerico Garza (2022). For the linear regression, XGBoost (Chen & Guestrin 2016), NHiTS (Challu et al., 2023) TFT (Lm et al. (021), and Transformer ( /Vaswani al. 2017). used Darts (Herzen al., (022) library. For the Gluformer (Sergazinov
ict al. )023) and Latent ODE (Rubanova et al. 2019) models, we adapted the original implementation
available on GitHub.
The shallow baselines, such as ARIMA, linear regression, and XGBoost, fit within 10 minutes for all datasets. Among the deep learning models, NHiTS was the fastest fit, taking less than hours on the largest dataset (Weinstock). Gluformer and Transformer required 6 to 8 hours to fit on Weinstock. Latent ODE and TFT were the slowest to fit, taking 10 to 12 hours on Weinstock on average.
C.2 HYPERPARAMETERS
In this section, provide extensive discussion of hyperparameters, exploring their impact on forecasting models' performance across studied datasets. For each model, we have identified the crucial hyperparameters and their ranges based on the paper where they first appeared. We observe that certain models, such as the Latent ODE and TFT, maintain consistent hyperparameters across datasets. In contrast, models like the Transformer and Gluformer exhibit notable variations. We provide a comprehensive list of the best hyperparameters for each datasets in Table 16 and provide intuition below.
Linear regression and XGBoost (Chen & Guestrin, 2016). These models are not designed to capture the temporal dependence. Therefore, their hyperparameters change considerably between datasets and do not exhibit any particular patterns. For example, the maximum tree depth of XGBoost varies by 67% ranging from 6 to 10, while tree regularization remains relatively consistent.
Transformer Vaswani et al., 2017), TFT (Lim et al., 2021), Gluformer (Sergazinov et al., 2023). Both TFT and Gluformer are based on the Transformer architecture and share most of its hyperparameters. For this set of models, we identify the critical parameters to be the number of attention heads, the dimension of the fully-connected layers (absent for TFT), the dimension of the model (hidden_size for TFT), and the number of encoder and decoder layers. Intuitively, each attention head captures a salient pattern, while the fully-connected layers and model dimensions control the complexity of the pattern. The encoder and decoder layers allow models to extract more flexible representations. With respect to these parameters, all models exhibit similar behavior. For larger datasets, e.g. Colas, Hall, and Weinstock, we observe the best performance with larger values of the parameters. On the other hand, for smaller datasets, we can achieve best performance with smaller models.
Latent ODE (Rubanova et al. 2019). Latent ODE is based on the RNN (Sutskever et al. 2013) architecture. Across all models, Latent ODE the only one that consistently shows the best performance with the same set of hyperparameter values, which we attribute to its hybrid nature. In Latent ODE, hyperparameters govern the parametric form of the ODE Therefore, we believe the observed results indicate that Latent ODE is potentially capturing the glucose ODE.
NHiTS (Challu et al., 2023). In the case of NHiTS, its authors identify kernel sizes as the only critical hyperparameter. This hyperparameter is responsible for the kernel size of the MaxPool operation and essentially controls the sampling rate for the subsequent blocks in the architecture. larger kernel size leads model to focus more on the low-rate information. Based our findings, NHiTS selects similar kernel sizes for all datasets, reflecting the fact that all datascts have similar patterns in the frequency domain.
26

---

Published as a conference paper at ICLR 2024
Table 16: Best hyperparameters for each model and dataset selected by Optuna Akiba al. (2019).
For models that support covariates, we indicate best hyperparameters with covariates in parantheses.
Hyperpurameler
Broll
Colas
Dubosson
Hall
Weinstock
LIN
lnn
192.00 (12.00)
12.00 (12.00)
12.00 (12.00)
84.00 (60.00)
84.00 (84.00)
len Ir
84,00 (96,00)
120.00 (144.00)
168.00 (36.00)
60.00 (120.00)
84.00 (96.00)
0.51 (0.39)
0,51 (0.88)
0.69 (0.65)
0.52 (0.17)
0.72 (0.48)
subsample
0.90 (0.80)
0.90 (0.90)
0.80 (0.80)
0.90 0.90 (0.70)
0.90 (1.00)
min child weight
2.00 (1.00)
(3.00)
5.00 (2.00)
3.00 (2.00)
5.00 (2.00)
XGB
colsample_hyree hytrce
0.80 (1.00)
0.90
0.80 (1.00)
0.90 (0.90)
1.00 (0.90)
max_h
9.00
7.00 (5.00)
10.00 (6.00)
6.00 (6.00)
10.00 (6.00)
garmma
0.500)
0,50 (0.50)
0,50 (1.50)
(1.00)
(1.50)))))
alpha
0.12 (0.20)
0.22 (0.06)
0.20 (0.15)
(0.17)
0.27 (0.16)
lambda
0.09 (0.02)
0.24 (0.09)
0.28 (0.09)
0.13 (0.02)
0.07 (0.03)
estimators
416.00 (288.00)
352.00 (416.00)
416.00 (480.00)
256.00 (320.00)
416.00 (320.00)
in_len
96.00
96.00
108.00
96.00
144.00
max samples.err.s.s
100.00
150.00
100.00
200.00
100.00
GLU
d.model
512.00
384.00
384.00
384.00
512,00
heads
4.00
12.00
8.00
4.00
8.00
fen
512.00
512.00
1024.00
1024.00
1408.00
num_enc Jayers
1.00
1.00
1.00
1,00
1.00
num somnddee_ayers
4.00
1.00
3.00
1.00
4,00
in_len max samples te
48.00
48.00
48.00
48.00
48.00
100.00
100.00
100.00
100.00
100.00
latents
20.00
20.00
20.00
20.00
20.00
LAT
rec diims
40.00
40.00
40.00
40.00
40.00
rec_aayers
3.00
3.00
3.00
3.00
3.00
gan_layers
3.00
3.00
3.00
3.00
3.00
units gru_units
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
in_len
96.00 (144.00)
132.00 (96.00)
108.00 (120.00)
144.00 (120.00)
96.00 (96.00)
max samples. per.us
50.00 (50.00)
100.00 (50.00)
50.00 (50.00)
100.00 (50.00)
800.00
kemne sizes
5.00 (3.00)
3.00 (3.00)
3.00 (2.00)
4.00 (5.00)
4.000)))0)
NHI
drupout
(0.09)
0.18 (0.13)
0.06 (0.16)
0.05 (0.19)
0.13
Ir
0.00 0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
00.00)))
batchasize
64.00 (32.00)
32.00 (48.00)
32.00 (48.00)
48.00 (48.00)
(32.000
Ir.ooos
16.00 (10.00)
2.00 (16.00)
2.00 (12.00)
2.00 (4.00)
16.00 (2.00)
len
168.00 (96.00)
132.00 (120.00)
168.00 (120.00)
96.00 (132.00)
132.00 (108.00)
samples per
50.00 (50.00)
200.00 (100.00)
50.00 (50.00)
50.00 (50.00)
200.00 (50.00)
hidden_size
80.00
256.00 (32,00)
240.00 (240.00)
160.00 (64.00)
96.00 (112.00)
TFT
num num_attention_hen
(3.00)
3.00 (3.00)
2.00 (1.00)
2..003..0)
3.00 (2.00)
drupout Ir
0.13 (0.23)
(0.11)
0.25 (0.24)
(0.15)
(..1()
0.00 (0.01)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
batch size
32.00 (32.00)
32.00 (32.00)
64.00 (32.00)
48.00 (32.00)
48.00 (48.00)
max grad norm
0.53 (0.03)
0.98 (0.80)
1.00 (0.09)
0.43 (0.66)
1.00 (0.95)
inJen max, max_ samples.peelsss
96.00 (108.00)
108.00 (120.00)
108.00 (156.00)
144.00 (132.00)
96.00 (96.00)
50.00 (50.00)
200.00 (00
50.00 (50.00)
200.00 (150.00)
50.00 (50.00)
d_model
96.00 (028.00)
64.00 (128 00)
32.00 (64.00)
(64.00)
128.00 (128.00)
heads
32.00)
2.00 (4.00)
2.00 (2.00)
4,00 (4.00)
2.00 (4.00)
num encoder layers
4.00
3,00 (4.00)
1.00 (2.00)
1.00 (1.00)
2.00 (1.00)
TRA
num, decoder
1.00 (2.00)
3.00 (1.00)
1.00 (1.00)
1.00 (3.00)
4.00 (4.00)
dim_feed feedforward
448.00 (160.00)
480.00 (128.00)
384.00 (384.00)
96.00 (192.00)
64.00 (448.00)
drapout Ir
0.10 (0.04)
0.12 (0.20)
0.04 (0.00)
0.01 (0.13)
(0.19)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
batch size
32.00 (32.00)
32.00 (32.00)
32.00 (0)
48.00 (48.00)
32.00 (48.00)
Ir epochs
16.00 (20.00)
8.00 (18.00)
6.00 (20.00)
14.00 (4.00)
4.00 (4.00)
max max_grad_mmg noorg norm
0.67 (0.89)
0.83 (0.82)
0.21 (0.10)
0.43 (0.23)
0.42 (0.19)
27

---

Published as a conference paper at ICLR 2024
D CHALLENGES
In addressing the challenges associated with the implementation of our predictive models in clinical settings, recognize three pivotal obstacles. Firstly, the challenge posed by computing power necessitates a strategic refinement of our models to guarantee their effectiveness on devices grappling with resource limitations and potential disruptions internet connectivity. The delicate balance between the complexity of the model and its real-time relevance emerges as critical factor, especially within the dynamic contexts of diverse healthcare settings.
Secondly, the challenge of cold starts for new enrolling patients presents significant hurdle. We acknowledge the importance of devising strategies initialize and tailor the predictive models for individuals who are newly enrolled in the system. This consideration underscores the need for a dynamic and adaptable framework that ensures the seamless integration of our models into the continuum of patient care.
The third challenge pertains to data privacy and transmission. address this, our models must either possess on-device training capabilities facilitate the secure and anonymized transmission of data to external servers. This emphasis on safeguarding patient information aligns with contemporary standards of privacy and ethical considerations, reinforcing the responsible deployment of models clinical practice.
28


Reviews:
[{"review_id": "XfqcW0CZMW", "reviewer": "Reviewer_5DZS", "summary": "The paper demonstrates the initiative and efforts in creating a public benchmark data repository for research in continuous glucose monitoring (CGM), with a curated collection of diverse CGM datasets, popular data tasks, bench-marking protocol and baseline models performance comparisons from existing literature. Code repository is available as supplementary material.", "questions": "1. How will new datasets be added to the repository?\n2. How could other researchers contribute to the repository?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Originality: the paper demonstrates an initiative in public data repository of ML application in CMG research\n- Significance: The initiative of creating a public data repository of CGM research documented in the paper is of great value to the both clinical and ML research communities for experiment reproduction, bench-marking new methods and potential application adoption.  \n- Quality: The paper is well-written with inspiring research questions. Model comparisons are performed on multiple datasets with inspiring research questions and discussion on results in part 5.\n- Clarity: The paper is well-organized with problem formulation, related work, dataset and data tasks description, benchmarking protocols and detailed discussion.\n- Code repository is provided with the paper submission", "weaknesses": "- Cross validation results will be preferred than simple train/test/validation splitting in benchmark model performance comparison.\n- Automated machine learning (AutoML) could also help in benchmarking performance across all datasets with more ML models and pre-processing pipelines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0tnICUMM8V", "reviewer": "Reviewer_HdpX", "summary": "The manuscript analyses continuous glucose monitoring (CGM) data and glucose prediction models for diabetes management. It addresses the challenges associated with accurate glucose prediction, evaluates various models, and offers a curated collection of public CGM datasets. The paper focuses on two main tasks: (1) enhancing predictive accuracy and (2) improving uncertainty quantification in glucose prediction. It discusses the impact of dataset size, patient composition, and covariates on model performance and generalization. The study emphasizes the need for cautious use of glucose predictions in diabetes management and proposes future research directions.", "questions": "I recommend authors consider below suggestions/limitations:-\n1) Are there any potential biases in the selection of the public datasets, and how might they impact the generalizability of the results? Authors should discuss about the quality of the selected CGM datasets assessed, and what criteria were used to determine data quality.\n2) The paper discusses the impact of covariates on model performance. Can you provide insights into which specific covariates had the most influence on the predictions and whether their quality was uniform across all datasets? And, how does the decision to omit certain models due to their lack of support for covariates affect the overall model comparison?\n3) Perhaps authors should consider including, at least, in the supplemental material about any potential risks or challenges in directly applying the findings from the study to clinical settings especially when translated into practical clinical applications for healthcare providers and individuals with diabetes.\n4) The paper mentions not considering additional model-specific tuning for benchmark models. Could such optimizations significantly enhance the models' performance, and why were they excluded from the evaluation?\n5)  How diverse were the patient populations in the selected datasets in terms of demographics, disease severity, or other relevant factors, and how might this diversity affect the results and generalization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The paper addresses the important issue of model generalization to new patient populations, providing evidence of the challenges associated with individual-level variation.\n2) The key strength of the manuscript is it presents a valuable resource for the diabetes research community by offering public CGM datasets, standardized tasks, benchmark models, and detailed performance analysis. It also provides valuable insights into the factors affecting glucose prediction, including dataset size, patient populations, and time of day. It highlights the varying impact of covariates on model performance across different datasets.\n3) I believe the study follows rigorous principles of reproducibility, ensuring that results are consistent across different data splits and model re-runs. It also focuses on fair comparisons by considering out-of-the-box model performance.", "weaknesses": "Some terms and concepts in the manuscript, such as ARIMA, Latent ODE, and Bayesian optimization, may be challenging for readers without a deep background in machine learning and statistics. Also, the choice of benchmark models might not be exhaustive, and the paper could benefit from a more extensive discussion of the models' suitability for different scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "upNGHOBuL1", "reviewer": "Reviewer_XK2E", "summary": "This paper presents a curated list of continuous glucose monitoring (CGM) datasets with a list of standard tasks for the problem of CGM both in terms of predictive accuracy and uncertainty quantification. It also provides benchmarking results using classic to modern methods on the datasets. Detailed analysis and discussion of the impacts of different factors on performance are provided.", "questions": "1. What is the motivation for some analysis you did in the paper? For example, what is the motivation for exploring the impact of time of day on accuracy? Why should researchers care about the analysis you did in Section 5, especially under the context of CGM?\n2. How is this work different from other review papers mentioned in related works?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The authors provide systematic benchmark results of different CGM models on multiple CGM datasets. Their evaluation methods seem to be reproducible and fair.\n2. They provide diverse analyses of the benchmarking results, including the impact of dataset size, generalization of the models, and impact of time of day.", "weaknesses": "1. The writing of some parts is not clear and easy to understand. For example, when describing Task 2, it is not clear what “uncertainty” actually means here.; the caption of Figure 3 is also not clear enough about what the “green block” refers to with more than one green element in the figure; on Page 8, the authors did not provide an exact definition of in-distribution and out-of-distribution test, making the readers have to guess what it means by the authors.\n2. I am concerned about the level of contribution of this work. Many of the conclusions of the paper are not novel. For example, people should already know “Healthy subjects demonstrate markedly smaller errors” and “using simpler shallow models when data is limited”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)