Physiome-ODE: A Benchmark for Irregularly Sampled Multivariate Time-Series Forecasting Based on Biological ODEs
Authors: Christian Klötergens
Vijaya Krishna Yalavarthi
Randolf Scholz
Maximilian Stubbemann
Stefan Born
Lars Schmidt-Thieme
Venue: ICLR 2025 Poster (2025)
Keywords: Irregular Time Series
ODE
Primary area: datasets and benchmarks
TL;DR: An Irregular Time Series Forecasting Benchmark created based on Biological ODE models stored in the Physiome Model Repository.

Abstract: State-of-the-art methods for forecasting irregularly sampled time series with missing values predominantly rely on just four datasets and a few small toy examples for evaluation. While ordinary differential equations (ODE) are the prevalent models in science and engineering, a baseline model that forecasts a constant value outperforms ODE-based models from the last five years on three of these existing datasets. This unintuitive finding hampers further research on ODE-based models, a more plausible model family.
In this paper, we develop a methodology to generate irregularly sampled multivariate time series (IMTS) datasets from ordinary differential
equations and to select challenging instances via rejection sampling. Using this methodology, we create Physiome-ODE, a large and sophisticated benchmark of IMTS datasets consisting of 50 individual datasets, derived from real-world ordinary differential equations from research in biology. Physiome-ODE is the first benchmark for IMTS forecasting that we are aware of and an order of magnitude larger than the current evaluation setting of four datasets. Using our benchmark Physiome-ODE, we show qualitatively completely different results than those derived from the current four datasets: on Physiome-ODE ODE-based models can play to their strength and our benchmark can differentiate in a meaningful way between different IMTS forecasting models. This way, we expect to give a new impulse to research on ODE-based time series modeling.

Published as a conference paper at ICLR 2025
PHYSIOME-ODE: A BENCHMARK FOR IRREGULARLY SAMPLED MULTIVARIATE TIME SERIES FORECASTING
BASED ON BIOLOGICAL ODES
Christian Klötergens ISMLL & VWFS DARC
Vijaya Krishna Yalavarthi
Randolf Scholz
ISMLL
ISMLL
University of Hildesheim Hildesheim, Germany k1 kloetergens @immllss@mmmmmml dde
University of Hildesheim Hildesheim, Germany
University Hildesheim
Hildesheim, Germany
Maximilian Stubbemann ISMLL & VWFS DARC University of Hildesheim Hildesheim, Germany
Stefan Born
Lars Schmidt-Thieme ISMLL VWFS DARC
Institute of Mathematics TU Berlin Berlin, Germany
University of Hildesheim Hildesheim, Germany
ABSTRACT
State-of-the-art methods for forecasting irregularly sampled time series with miss- ing values predominantly rely on just four datasets and few small toy examples for evaluation. While ordinary differential equations (ODE) are the prevalent models in science and engineering, a baseline model that forecasts a constant value outper- forms ODE-based models from the last five years on three of these existing datasets. This unintuitive finding hampers further research on ODE-based models, more plausible model family. In this paper, we develop methodology to generate irreg- ularly sampled multivariate time series (IMTS) datasets from ordinary differential equations and to select challenging instances via rejection sampling. Using this methodology, we create Phys come- a large and sophisticated benchmark of IMTS datasets consisting of 50 individual datasets, derived from ODE models developed by research in Biology. Physt amme-DDE is the first benchmark for IMTS forecasting that
of and an order of magnitude larger than the current evaluation setting. Using Physiome - ODDE we show qualitatively completely different results than those derived from the current four datasets: on Physi ooee-omee-ee deep learning methods based on ODEs can play to their strength and our benchmark can differentiate in a meaningful way between different IMTS forecasting models. This way, we expect to give new impulse to research on irregular time series modeling.
INTRODUCTION
Over the past decade, substantial research has been devoted to fully observed and regularly sampled time series, often referred to regular (multivariate) time series. However, certain scenarios, observations occur at irregular intervals and variables are measured independently resulting in sparse multivariate time series with many missing values. The resulting time series are known as irregularly sampled multivariate time series with missing values, or simply IMTS.
With regular time series forecasting being well-established topic, there exist a variety of sophisticated methods and standardized benchmarks (Godahewa et al. [202 Gilpin| 2021 Bauer et al.) (021))
On the other hand, there currently no IMTS forecasting benchmark published. Creating such benchmark demands a high number of publicly available time series datasets with gemuinely irregular observation processes. However, only a few of those exist. Although they are closely related to each other, most evaluation sections of IMTS forecasting papers (Biloš et al. 2021 (Schirmer et al. 2022 Yalavarthi et al. 2024, Zhang et al. 2024) are based on at least a subset of these datasets. Hence, stands to reason that they induce dataset bias. Additionally, show that simple baseline,

---

Published as a conference paper at ICLR 2025
which always predicts a constant value independent of time, is competitive with or even outperforms complex neural ODE models on these datasets. Hence, it appears questionable whether the currently used datasets are indeed well-suited for forecasting.
To address this limitation, we introduce Physiome- -ODE, a wide benchmark of IMTS datasets consisting of 50 individual datasets, derived from ordinary differential equations from biological research, that are stored in the Physiome Model Repository (PMR). Biological processes are well- suited for generating IMTS datasets, they are inherently multivariate and irregularly measured in real-world experiments. Additionally, the PMR provides Python implementations of many of these an automated manner. While Biology researchers create their models based on very few and non-published observations. they enable us to create an arbitrary number of time series which relate to possible measurements of a real-world phenomenon. Phys Home-ODE is the first benchmark for IMTS forecasting that we are aware of and an order of magnitude larger than the current evaluation setting of just four datasets.
models, allowing us to create Phys somemmmeee
Furthermore, evaluate the complexity of the different forecasting datasets, we introduce simple metric called Joint Grudient Deviation (JGD), which measures the gradient variance of ODE solutions. We will show that our benchmark consists of datasets of different complexity and covers wide range of different JGD values.
Finally, evaluate current IMTS forecasting methods Phys thhysiom--oee ome-ODE and show that it includes many datasets on which neural ODE-based models significantly outperform the time-constant baseline model. Furthermore, member of the neural ODE model family actually emerges as the overall most accurate model on Phys oome-ODE However, the datasets Phys Physiome-Ooi -ODE are diverse enough that no single model is the most accurate for every dataset.
Physi i ome-omee-ooo -DEE is a significant step forward to standardized IMTS forecasting and to monitor research progress. Our contributions include the following:
1. We introduce simple baseline model that is restricted to making constant forecasts. On traditional IMTS forecasting evaluation, this simple baseline shows competitive or even better forecasting accuracy when compared to ODE-based models.
2. We propose Joint Gradient Deviation (JGD), simple score designed to approximate the difficulty of time series datasets for forecasting.
3. We create Phys i OmE- a large and diverse benchmark of IMTS forecasting datasets. Maximizing JGD, we select and configure ODE models present in the Physiome (Yu et al., 201I) repository. Physi tome oome ome-ODE is the first benchmark for IMTS forecasting that we are aware of.
4. We evaluate state-of- tftte-of-th-art art IMTS forecasting models on Physicme-OB Our experi- ments show that the datasets included Physi ome-ommeomm are diverse enough to highlight different strengths of competing models. This results in different performance rankings competing models for each dataset and the absence of a single best model. In that, Physiome -DDE differs from the currently used IMTS forecasting datasets, which is expected to provide a fresh impulse to research on IMTS forecasting models.
5. We share our code on GitHub
com http://ggthtabbcmmmm//otttteeeee
Physiome-One ODE)
2 IRREGULARLY SAMPLED TIME SERIES FORECASTING
An irregularly sampled multivariate time series (IMTS) is sequence X= ((ts, B))) (R (R U [Na =: X' of pairs where & is the observation timepoint and U; is an observation event with maximum of C-many variables (or channels) being observed. We use
indicate the space of sequences. We call Uie E R an observed value and Uie = NaN a missing value. ,ORY
Q = (1ORY < )==1:K E (R x (1,..., "),.,,"))* = Q is the query sequence where 1 is a queried
(future) timepoint and ORY is queried channel. Y = (yi,..., OK) E R =: y is the forecasting answer where yı is the ground truth forecast value for Qu-
Then the IMTS forecasting problem is defined as follows: given a sequence of triples ((X,, Qn, Yw) Y= drawn from random distribution P where X, E X' is an irregularly sampled
2

---

Published as a conference paper at ICLR 2025
Table 1: Test MSE for forecasting next 50% after 50% observation time. OOM refers to of memory. We highlight the best model in bold and underline the second best. indicates that we show the results from Yalavarthi al. (2024)
Model
USHCN
PhysioNet-2012
MIMIC-III
MIMIC-IV
GRU-ODE
1.017=0.325
0.653=0.023f
0.653+0.023f
0,439 1
LinODEnet
0.662+0.126
0.411#0.001*
0.531 0.022
0.336+0.002
CRU
0.730#0.264
0.467=0.002
0.619+0.028%
00M
Neural Flow
1.01420.336
0.506=0.0021
0.651#0.0171
0.465 0031
GraFITi
0.636#0.161
0.401=0.001
0.491 0.014
0.285=0.002'
GraFITi-C
0.875+0.204
0.407#0.001
0.543+0.024
0.324+0.002
time series, Qn E Q is a query sequence and Y, E y is a ground truth answer, as well as loss function 8: y y + E, e.g., the mean squared error between two equilength sequences, the task is find model Y X' x Q y such that the expected loss between the ground-trute forecasting answer Y and the model prediction ((X, Q) is minimal:
p( E(x. E(,
2.1 CURRENT FORECASTING MODELS
In our experiments we compare five recent models. Four of these five network architecture are variants of the neural ODE (Chen et al. 2018), which uses an ODE, that is defined by a neural network, to infer the hidden state in- in-between observations. At observation times the neural ODE-based estimation of the hidden state is updated on the actual observation. For more details, we refer to the papers. We use the following models:
CRU-ODE-Bayes (De Brouwer et al., 2019) utilizes a continuous version of Gated Recurrent Units (GRU) as the ODE defining network f. Additionally, GRU-Bayes applies GRU-based Bayesian update to the hidden state at observations. LinODEnet (Scholz et al. 2022) restricts f to be linear functions but encodes data non non-linearly into latent space. At observations LinODEnet updates its hidden state with a so-called nonlinear KalmanCell, a module inspired by Kalman Filtering (Kalman 1960). Continuous Recurrent Units (Schirmer al., 2022) replaces the neural ODE with a stochastic differential equation (SDE). CRU is able to infer the hidden state at any time in closed form using continuous-discrete Kalman filtering, instead of solving neural ODE, Neural Flows (Biloš et al.) (202T) infers the solution curve of an ODE directly with invertible neural networks. GraFITi Yallavarthi et al., 2024) substantially different from neural ODE. It encodes the observations from IMTS into graph structure and infers forecasts by solving graph completion problem with a graph attention network. On the established evaluation datasets GraFTTi significantly outperforms the neural bDEeb models in terms of forecasting accuracy.
3 TIME-CONSTANT APPROACHES OUTPERFORM ODE- APPROACHES IN THE CURRENT EVALUATION SCENARIO
Current Evaluation Scenario. Traditionally, IMTS forecasting has relied on datasets like Phys- ioNec2012 Sil et al. 2012), MIMIC-III (Johnson et al. 2016), and MIMIC-IV (Johnson et al. 2021]) obtained from Kaggle challenges designed for IMTS classification. Additionally, De Brouwer et al.)
(2019) proposed to create an IMTS forecasting task, based on USHCN (Menne et al. al.] 2016). a fully observed regularly sampled time series dataset containing climate data. We present characteristies of these datasets in Table 3 (appendix).
The GraFITi model the state-of-the-art on PhysioNet2012. MIMIC-III, MIMIC-IV, and USHCN, demonstrating a significant performance advantage over neural ODE-based methods Yalavvrthhi et al.) 2024). To further analyze this gap, we designed a constant version of GraFITi (GraFITi-C), which is restricted make the exact same forecast for every query time point. This is achieved by training GraFITi model that cannot access the actual query time point (f (FORY), ), but is instead always provided with constant dummy query time point.
3

---

Published as a conference paper at ICLR 2025
The results shown in Table I indicate, that neural ODE- models are indeed outperformed by this allegedly easy-to-beat baseline on PhysioNet2012 and MIMIC-IV. On MIMIC-III the best performing ODE-based model achieves a lower average test MSE than GraFITi-C, however the difference is within the standard deviation. GraFITi on the other hand, consistently outperforms its constant variant. However, it also cannot separate in by drastic margin. These findings could relate to important covariants, which are not contained in the dataset. For instance, it is impossible to predict a patient's blood glucose level without information about any food intake.
We designed thysiome-ote ODE to be an extension to the currently used evaluation protocol. For our ODE generated datasets, we can guarantee that all necessary covariants are given. Furthermore, we can take full control about the simulated observation process.
GENERATING CHALLENGING IMTS DATASETS FROM ODES
Sampling ODEs to Generate IMTS Data. In science and engineering, processes that evolve in continuous time are usually described by differential equations, in the simpler cases by ordinary differential equations (ODEs). All possible observations of such a process with channels over duration 7 at any time can be captured by time-varying function x: T) RC as x(().- An ODE characterizes such a function implicitly by two conditions:
1. A relation between observation time I, observation value xx()) and the instantaneous change in observation values r'(1) (t), i.e. the first derivative of x. This relation is described by function F: [0, x RC x IRC + IR called the system.
2. The initial values xp E RC of the function at time = O:
F(1,,((),z'(1)) = 11 E [0, 7),
<=(0) 250
(2)
Note that ODEs can be defined including higher order derivatives, but w.l.o.g. one can assume they are of first order, since the higher order system be transformed into first order system by change of variables (Teschl 2012). In a few special cases the solution æx() can be computed analytically. For example, the oneddimensional system ((t), xt)) == or ar(() x (t) characterizes the exponential growth function ætt) = ex xg. However, in most cases, it can only be computed by a numerical ODE
solver.
Specific parameters that occur in a system like the growth factor a in our example are called constants and one writes F(t, 2' (t); a) with a R- to denote a system with A-many constants. Triples (F, a, xp) of systems, constants and initial values can be turned into simulator, i.c. generative model, by complementing them with two more components:
1. A sampling process s for observation time points and channels that can generate finite sequences ((t,,c1,,... channels e; (l....,,C, C).
cr)) of pairs of observation times i; E [0, T" and observed
2. A noise model PNoISE for the observation values, that is conditional distribution PNOISE (z: TRUE) of the observed value given the ground truth value.
Then one can generate irregular sampled instances (li, c, x oBs ( underlying function x with missing values and affected by observation noise simply via:
1i=1: of the (only implicitly given)
((tz, (i) - ())
x
TRUE = ode-solve(FF F, a, Fo. tz)a,
2 ~ Proisu (x)
OBS
OBS
x
TRUP
(3)
Identifying Challenging Time Series. While any ODE implicitly describes time-varying function x: [) - R, not all such functions challenging to forecast: For example, some just deseribe very simple convergence processes that approach a limit value over a long time. In order to discover challenging time series, we propose measure the mean gradient deviation (MGD), that is the time-normalized 1/2 norm of s' (l) around its mean:
MGD(x) = min V (2(()) -)2
1
(4)
4

---

Published as a conference paper at ICLR 2025
Note that the minimum is obtained precisely for the mean gradient
1 c T /- d'()d x(() -x))
(S)
For a linear test function z(t) = at + b, the MGD zero, while for sine wave z(t) = sin(wt), the MGD scales linearly with the frequency To identify not just individually challenging time series, but whole distributions (i.e. a generator p), one can compute the expectation of MGD. We found that the MGD grows super-exponentially with the Lipschitz constant, which is deseribed in more detail in Appendix
With MGD one does assess only the heterogeneity within each function, not heterogeneity between different functions, which is an important precondition for an interesting data set for machine learning tasks. To capture this second aspect, we propose the mean point-wise gradient deviation (MPGD):
MPGD(p) = T / (od)))(=((())ee T [.V
- Eyy-ly dt (6)
We combine both measures multiplicatively as joint gradient deviation (JGD) for distributions of functions, i.e. function generators:
JGD(p) == MPGD(p) Ba MGD(())
Note, that we can approximate MGD with a finite number of samples:
Lemma For a function E C² ((0, 7)) and e > such that T E N we consider the divided differences
x(t) xtt c)
2t
diff.
t= e, 2e,.....
It then holds that the numerical estimator (td[((a))))) [(+))==::)) = V E deviation of the divided differences converges to the MGD of the function:
(xx - 8)2 of the standard
sidf = MGD(2)
(8)
We can also approximate MPGD with a finite amount of samples under assumptions we explain Appendix B
Lemma 2. The mean point-wise gradient deviation of distribution p of functions can be approxi- mated on a sample (nn)n of N -many sequence from on a fixed grid of time points as follows: Given the divided differences:
[t) xt ()) t= = cc T
diff Fnt
Then the average of the point-wise std. of the divided differences converges the MPGD:
stdl
MPGD(p) for c 0 and N almost surely
T
The proofs can be found in Appendix B Due Lemma and Lemma mate the MGD, MPGD and therefore the JGD based on the set of evenly spaced sequences X E NXT, where En,t denotes the value at time-step in the n-th sequence, with F, p. Given diff = 1n,441 - Fn,t, holds that JGD(p) MPGD(p) * E MGD(x.)) and furthermore:
approxi-
MGD(2,) d
MPGD(P)
std[(2 )=
stdl[
(10)
Lemma 1 and Lemma 2 thus allow approximate the JGD-score by sampling the values of several functions on shared grid from a generator. definition on the full function guarantees that will always get the same results (in the bounds of the approximation emror).
5

---

Published as a conference paper at ICLR 2025
p poooeddd de oo
DUPOI JGD: 2.697
LENOI JGD: 0.178
GUP01 JGD: 0.014
NELO1 JGD: 0.007
Demonstration of time series realized by 4 ODEs of different prediction difficulties without adding any noise. Each line / color represents a channel. Trajectories are shown for a duration of the respective dur shown in Table
Figure 1:
JGD for ODE models. To simplify the initial definition, we considered the JGD for a univariate function x: [0, T] + R, and we interpret single channel within multivariate ODE system as such function. Computing the JGD for a multivariate ODE model requires combining the JGD values of each channel. We found that simply averaging all channels disadvantages models with a high number of channels. To address this, compute the mean of the ten channels with the highest JGD. For models with ten or fewer channels, we revert to using the mean of all channels. Note that JGD is only a useful measure when applied to standardized data, otherwise the units are incomparable. Consequently, we normalize each channel to have a mean of 0 and standard deviation of 1 over all time steps and time series instances.
Varying Single ODE Instances. From the scientific literature one often gets single ODE instances: triplet (F, a' 20 of parametrized ODE system F, its parameters/constants a and the initial conditions I.mm all three deseribing some specific experiment. To generate data by sampling only from such a single ODE instance will create functions that differ only in observation time points and channels and in the noise. These instances are too similar to each other, without noise there is only one function, and hence the MPGD of the distribution would be zero. We therefore propose to also carefully vary (i) the initial conditions To, (ii) the ODE constants a and (iii) the total duration T and of the ODE sampling process. We sample each from a distribution that yields the values given the scientific literature on expectation and is controlled by a spread parameter a each:
To Pinitial L Oimitial)
a ~ Prons
LIT cromst)
T ~~ -Paur (Jurr
(11)
Let be the two-stage function generator that first samples tuples (zu, T) from their respective distributions and then yields as function x the solution to the ODE with initial values To. constants and running for duration of T. Instead of blindly choosing the spreads a, we optimize them w.r.t.
the JGD-value of the generator:
(a initial: T const: dur)
arg max
JGD JGD(p( Cinitial, Cconst, O
(12)
The JGD of each generator is estimated by 100 randomly sampled fully observed time series with a sequence length of 100. Since we are interested in the difficulty of the forecasting task, we compute the JGD on the final 50 steps of sequence only. We search miniaa from the range of {0.1, 0.3, 0.5}. Jcanst from (0.05, 0.1, 0.3), and der from (0.33, 1, 3.3, 10, 30). The unit of ouur varies for each ODE model, it is given by Physiome (Yu et al. 2011) and listed in T (appendix..
Creating the Physiome-ODE Benchmark. In the Physiome database we identified N 208 multivariate ODE models (Fmh " _..TT"") "mmi:, that are defined by automatically generated Python code. We ranked all ODE-models with according to their optimized JGD-value (see Table 4 and selected the highest 50 datasets for Physi comeODD For each of the final datasets, we created a dataset with 2000 instances each. To further increase the range of initial states, we create datasets of lengths aur but with 200 observation steps. For each time series instance, we sample the initial time step fumeet ~ {t,) 1 and then select the next 100 steps for each time series instance. Consequently, each time series instance has a duration of 1 o While the time series instances 2
sampled regularly at first, we create IMTS instances by randomly dropping observations with chance of 80%. For Proise we select to add e ~ (0, 0.05). In Figure we show four trajectories contained in Physi ome -DDE with the respective JGD.

---

Published as a conference paper at ICLR 2025
Handling exploding ODEs. Varying the constants and initial states of the ODEs can lead to incon- sistent states, which could never occur in real world scenarios. In general, this is not an issue regarding the machine learning perspective of inconsistent states and constants can lead to extreme values which are magnitudes greater than values that occur in other time series instances. These extreme values will dominate the training and evaluation loss in a problematic manner. We prevent this by rejecting all (Finitial- Consst durr) containing any values greater than 10 times the channel-wise standard deviation in the 100 samples created for evaluation. If such values nevertheless occur within the final 2000 samples created for mhysiome-ann we drop the respective instances. We follow the same protocol to handle errors thrown by the ode-solver.
experiments. Occasionally however, certain combinations
5 EXPERIMENTS
Baseline models. We compare on 5 recent IMTS forecasting models, namely GRU-ODE-
Bayes (De Brouwer et al. 2019). Neural Flows (Bilo3 et al. 2021). Continuous Recurrent
Unit (CRU: Schirmer et al. 2022), LinODEnet (Scholz et al., 2022 Mione et al. 2024) and
GraFITi (Yalavarthi et al. 2024). GRU-ODE-Bayee Neural Flows, CRU and LinODEnet are
differential equations based models whereas GraFITi is graph based model. We also introduce naive model called GraFITi-Constee (GraFITi-C) which predicts a fixed value for all the future time points in all the target channels for instance.
Experimental Protocol. In our experiments models have to predict the last 50% of the time series after observing the first 50%. We use 5-fold cross validation, and for each fold we split the data into training, validation and test set with a ratio of 70:20:10. Additionally, we resample the sparse observation mask to transform each instance into an IMTS. The evaluation metric is the mean square error (MSE). For simplicity, we randomly sample 10 hyperparameter configurations and fitted each model on a single fold per dataset, selecting the configuration with the lowest validation MSE. The winning configuration per model is then trained and evaluated on all 5 folds (see Appendix F). We run the benchmark experiments on a cluster of 4 NVIDIA 4090 GPUs with 24 GB.
Empirical Results. The numbers presented in Table are the mean and standard deviation of MSE for 5 folds. Overall, LinODEnet (Scholz et al. [022] Mione et al., 2024) is the best rest-performing model, winning the most datasets and having the highest average rank. On the other hand, the average rank of GraFITi (2.20) is close to that of LinODEnet 10) Surprisiggly, GraFITi-C performs better than Neural Flows and GRU-ODE-Baye like DUP01 01 and DOK01 1 are too hard to forecast, hence the complex models cannot improve upon the accuracy of a constant model. Other datasets are too simple that even a constant model yields MSE close to 0.0025, which is the lowest value possible to achieve due to the normal-distributed noise with a variance of 0.05.
and has an average rank of 2.86. This is because datasets
In order to check how good our JGD score can measure the difficulty in forecasting of a time series we plot JGD score vs the best MSE score achieved in Figure 2] It can be observed that JGD fulfills its purpose of finding challenging and interesting ODE models. More specifically JGD and the test MSE of the best performing model are correlated with a Spearman coefficient of 0.77.
6 EXTENSIONS AND LIMITATIONS
Currently, most datasets in Phys i ome-ODE still show a good potential for improvement: even the best performing methods are far away from the Bayes error of 0.0025 MSE. In case authors of future methods get the impression that their method could benefit from larger datasets, i.e., more instances per dataset than 2000, they easily can generate larger ones with 10,000, 100,000, etc. instances.
P Phys serssiom-oonn ODE is not based on observation data as in these areas observed data usually is expensive to get, and thus only limited amounts exists or are publically available. at least. In fact, we scanned through the respective papers of our top 50 ODE models and could not find the original experimental data for single one. Hence, one wants to conduct machine learning experiments on these complex and rich biological processes one is forced to create them using the ODE models describing the process.
For brevity, denote this GRU-ODE in the tables.
7

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results on various baseline models. Physi Physiome-ODE datasets ranked by JGD-score. We highlight the best model in bold and underline the second best.
Dataset
JGD
GRU-ODE
LinODEnet
CRU
Neural Flows
GraFITi
GraFITi-C
DUP01
2.697
1.037= 0.047
0.9642 0.036
0.958t 0.038
I 1.030 0,046
0.955= 0.087
0,951# 0.036
JELO1
2.580
1.017= 0.014
0.9492 0,015
0.939t 0.015
1.000s 0.013
0.942s 0.020
0.935 0.016
DOK01
2.277
1.01 0.004
0.996 0.003
0.985# 0.005
0.998: 0.005
0.005
0.982+
INAOI
2.218
10088 0.010
1.009+ 0,011
1.005+ 0.010
I 1.008 0.008
1.004 0.010
1.004 0.009
WOLOI
1.973
0.952= 0.035
0.806= 0.027
0,814 0.029
0.841 0.029
0.787= 0.028
0.784# 0.030
BORG1
1.795
0.794z 0.034
0.719 0.021
0.715: 0.020
0.743 0.026
0.712s 0.022
0,709: 0.022
HYNO1
1.548
0.883 0.000
0.672= 0,044
0.665+ 0.053
0.636# 0.045
0.625= 0.043
0.619* 0.046
JELO2
1.271
0.816z 0.049
0.6932 0.031
0.674: 0.028
0.779: 0.028
0.029
0.687: 0.027
DUP02
1.202
0.895 0.055
0.740s 0.042
0.722+ 0.046
0.890s 0.081
0.728 0.044
0.718 0.046
WOL02
0.895
0.854: 0.010
0.663= 0.015
0.6534 0.017
0.685= 0.012
0.014
0.645=
DIPOI
0.735
1.035+ 0.023
0.832# 0.087
0.985+ 0.025
1.014 0.025
0.985= 0.030
0.982+ 0.029
VANOL
0.407
0.32 0.023
0,250z 0.006
0,2534 0.005
0.250= 0.006
0.246G 0.005
0.242z 0.006
DUP03
0.254
0,874k 0.089
0.6324 0.044
0.6224 0,047
1.098 0.447
0.627= 0.043
0,744z 0.042
BEROI
0.179
0.594 0.054
0.27%: 0.020
0.280 0.016
0.398 0.014
0.300= 0.018
0.342+ 0.018
LENOI
0.178
1.028 0.060
0.387: 0.071
0.754 0.157
1.00% 0.061
0.970z
LIOI
0.113
1.067= 0.015
0,084 0.009
0.175 0.020
0.979 0.049
0.202s 0.013
0.742 0.010
LI02
0.097
0.723+ 0.080
0.434+ 0.044
0.4374 0.046
0.674+
0.058
0.458+
REVOI
0.081
1.091 0.075
0.597= 0.061
0 0.602+ 0.049
0.978= 0.042
0,674 0.055
0,855 0.050
PUROI
0.049
0.703z 0.058
0.106z 0.006
0,3531 0.083
0.6542 0.102
0.153e 0.006
0.4762 0.020
NYG01
0.047
0.57 0.074
0.358 0.071
0.403+ 0.092
0.442 0.094
0.344; 0.065
0.366 0.047
PUR02
0.044
0.830 0.066
0,280 0.028
0.293+ 0.076
0.723+ 0.077
0.322 0.021
0.511 0.023
0,042
0.851
0.441
0.4091 0.049
0.701 0.077
REFOI
0.035
0,266* 0.068
0.045# 0,012
0.051 0.008
0.045# 0.011
0.033 0.007
0.039 0.012
VILOI
0.028
0.51 0.053
0.374: 0.021
0.3734 0.039
0,500= 0.060
0.344: 0.044
0.378+ 0.042
KARO1
0.023
0.193 0.013
0,034= 0.008
0.044+ 0.012
0.0692 0.009
0.041= 0.013
0,078: 0.011
SHOOI
0.023
0.260s 0.017
0.057: 0.006
0,095 0.010
0.092s 0.014
0.062: 0.013
0.055 0.013
BUTOI
0,020
0,583+ 0.172
0.254= 0.074
0.317 0.108
0.441 0.153
0,281= 0.071
0.324 0.091
MALOI
0.019
0,420+ 0.061
0.018= 0.007
0.064+ 0.007
0.052= 0.002
0.020= 0.004
0.054+ 0.005
0.019
0.114: 0.035
0.022s 0.003
0.0461 0.014
0.033
0.025:
0.026# 0.002
BUTD2
0.016
0.483+ 0.047
0.207= 0.056
n,282+ 0.042
0.329+ 0.068
0.248= 0.052
0.256+ 0.039
MITOI
0.015
0.005: 0.001
0.003= 0.000
0.003+ 0.000
0.008: 0.004
0.003= 0.000
0.003+ 0.000
GUPO1
0.014
0.153z 0.041
0.018 0.007
0.057:0.017
0.0432 0.017
0.041 0.006
0.0351 0,006
GUY01
0.013
0.036 0.004
0.006 0.005
0.0042 0.001
0.024z 0.015
0.005: 0.003
0.004# 0.001
PHIO1
0,013
0.674 0.030
0.131= 0.014
0.133+ 0.020
0.635# 0.158
0.222 0.013
0.345 0.015
GUY02
0.013
124 0.044
0.010= 0.006
0.010 0.002
0.082+ 0.066
0.012 0.009
0.032# 0.015
PULOI
0.013
0.099: 0.024
0.008= 0.004
0.003
0.060
0.001
CALOI
0.013
1.099 0.055
0.078= 0.009
0.158 0.008
0 0.867 0.014
0,179 0.012
0.643+ 0.024
WODOI
0.012
0.612# 0.072
0.154: 0.016
0.113: 0.017
0,510= 0.060
0.1642 0.013
0.344t 0.016
GUP02
0.012
0.870z 0.059
0.469% 0.022
0.444: 0,018
0.577 0.017
0.449: 0.027
0,4612 0.025
MOI
0.012
0.055 0.009
0.004 0.001
0.005 0.000
0.124 0.207
0.003: 0.000
0.003 0.000
LEN02
0.012
0.297+ 0.071
0.039= 0.005
0.059+ 0.012
0.380+ 0.141
0.099: 0.021
0.143+ 0.022
KAR02
0.011
0.515 0.032
0.140z 0.010
0.151+
0.257= 0.016
0.15 0.009
0.252 0.010
SHO02
0.011
0.368+ 0.028
0.037= 0.006
0.0834 0.015
0.109= 0.015
0.006
MACOI
0.010
0.242 0.026
0.020= 0.003
0.065+ 0.006
0.029: 0.011
0.021 0.003
0.019 0.002
IRI01
0.010
0.151 0.052
0.037= 0.003
0.0491 0.010
0.1162 0.006
0.038 0.017
0.097= 0.008
BAGOI
0.010
0.2944 0.041
0.032+ 0.005
0.0464 0.005
0.075 0.012
0.029 0.002
0.109s 0.002
WOL03
0.008
0.859: 0.1 14
0.073s 0.010
0.177+ 0.016
0.479; 0. 107
0. 10105= 0.016
0.247 0.032
WANDI
0.008
0.504 0.031
0.103= 0.012
0.125 0.012
0.345= 0.027
0.119: 0.010
0.232+ 0.015
NELOI
0.007
0.054z 0.012
0.010z 0.001
0.009+ 0.001
0.0602 0.029
0.0072 0.000
0.023 0.006
HUAD1
0.007
0.321 0.046
0.052= 0.004
0.1164 0.007
0.015
0.005
0.115
Wins
2
10
15
Rank
5.9
2.10
2.82
4.82
2.20
2.86
8

---

Published as a conference paper at ICLR 2025
100
Consequently, we build the benchmark on top of real- world ODEs: their system, their constants and their initial values establish a real-world connection that has been created in hundreds of scientific publica- tions. To delineate our benchmark from purely syn- thetic data we therefore call it semi-synthetic.
Best MSE
10-1
The fact that the datasets are generated by ODEs implies that some general ODF-based models are expressive enough to mimic the generating process and perform well for large amounts of data. For example, LinODEnet can represent a Koopman- type (Koopman 1931] Koopman and v. Neumann 1932) linear approximation of any nonlinear ODE under some assumptions. However, being expressive enough does not mean that these models are best when trained on a limited amount of data. Other, non-ODE models can beat ODE-based models with limited training data, even the ground truth is an ODE. In fact, a main purpose of the henchmark is to questions about the best forecasting models for data generated from (unknown) ODE ground truth. Despite our limitation to ODEs from biology the perspective is forecasting for many scientific domains where dynamics are in principle governed by ODEs.
10-2
10
10 1
10
JGD
Figure 2: Test MSE of the best performing model vs JGD-score across 50 datasets.
The inductive bias of ODE-based models (LinODEnet (Scholz et al., 2022 is rather weak. Being based on ODEs does mean that these models would only perform well on data generated from ODEs. When one spells out the expressiveness of LinODEnet in terms of basis functions, it would just mean that after nonlinear transformation the time dependent functions can be expressed by some basis functions given as solutions of some linear ODE, The representation is entirely trainable and nothing about the time series in question is hard-coded apart from smoothness.
The number of instances per dataset contained in Phys iome-ODE is deliberately small (2000). Our goal was not to create huge datasets just for the sake of being huge, but to create datasets for the purpose of research: the smallest datasets that would allow complex, ODE-based models to demonstrate their strengths. However, it is trivial to create an arbitrary number of time series, larger datasets are desired.
While this work focuses on IMTS forecasting, Physiome- ODE can be easily extended to regular time series forecasting. In Appendix G. we conduct experiments with three datasets created with Thysiome-Onn
Our results show that these datasets have an interesting property: They force models to learn channel dependencies. This contradicts with the results traditional evaluation datasets, in which models like PatchTST (Nie et al. 2023) are more effective when they ignore inter-channel correlations.
7 PRIOR TIME SERIES FORECASTING BENCHMARKS
Monash time series forecasting archive (Godahewa al., 2021) The Monash repository is the first benchmark for time series forecasting. It encompasses 11 openly-accessible multivariate time series datasets drawn from various domains, including energy, banking, and nature. Some of these datasets may include missing values. However, the sampling process is regular. Therefore, the missing values can easily be imputed, so that methods designed for regular time series forecasting can be applied. Hypothetically, we could use the 11 multivariate time series contained in the Monash archive to create IMTS datasets for an IMTS forecasting benchmark. However, most of these multivariate datasets are actually multiple univariate time series stacked on top of each other and/or single long time series that is segmented to create multiple time series instances. Other datasets used to evaluate regular multivariate time series forecasting (Nie et al. 2023; Zeng et al.] 2023 Zhou et al.) 202I), are not well-suited for IMTS forecasting for similar reasons. In fact, Nie et al. (2023) showed that models benefit, when they ignore channel correlations, hinting that these barely carrying

---

Published as a conference paper at ICLR 2025
useful information in the respective datasets. In IMTS forecasting however, modeling these channel correlations a crucial property of the models, due to the high number of missing values.
Chaotic ODEs benchmark for time series forecasting Closely related Physi ome ODE, Gilpin 202T) provides a forecasting benchmark created with chaotic ordinary differential equations that mannly originate from Physics. Similar the Monash repository, each chaotic ODE creates single long trajectory, without varying the constants. In contrast, Physions cetrssime-DD contains 2000 different trajectories with varying constants and initial states. Furthermore, the inherent chaotic nature of these equations often presents forecasting challenge, even in the regularly sampled and fully observed forecasting setting. On the other hand, IMTS forecasting challenging due the sparse observation of a time series. We assume that combining these two factors of difficulty, would surpass any current methods forecast modeling ability. This is closely related to what we observe on the most difficult dataset of Physi ome-ODE. We support assumption, by example experiment on the Respective experimental details are given in Appendix H and results in Table Similarly, certain Physi i ome ODE datasets with a high JGD (e.g. DUPO1), no method can significantly outperform the simple baseline GraFITi-C.
a famous chaotic ODE, which is a part of Gilpin (2021)ss benchmark.
PDEBench (Takamoto et al. 2022) Related to applying differential equations to create datasets, there exists a benchmark with 11 spatio-temporal datasets named PDEBench. Here, models are trained to estimate the parameters of an underlying partial differential equation (PDE) used to generate such dataset, instead of forecasting. Current IMTS forecasting architectures cannot utilize the spatio-temporal information inherent in PDEBench's datasets.
CONCLUSION
We introduced Physiome-OUE the first ever IMTS forecasting benehmark. datasets are created with ODE models, that were defined in decades of research and published on the Physiome Model Repository. Our experiments showed that LinODEnet and CRU actually better than previous evaluation on established datasets indicated. Nevertheless, we also provided a few datasets, which models are unable to outperform constant baseline model. We believe that our datasets, especially the very difficult ones, can help to identify deficits of current architectures and support future research on IMTS forecasting.
REPRODUCIBILITY STATEMENT
Our experiments can be reproduced by following the instructions provided our Git repos- itory: https://githun.
There are two options to obtain the datasets from Physiome-ODE. ODE. The first is regenerate everything using the code and instructions in the repository. The second is to download the data from Zenodo httta:///enodo..or//errod////
10

---

Published as a conference paper at ICLR 2025
REFERENCES
D. W. K. Andrews. Consistency in Nonlinear Econometric Models: A Generic Uniform Law of
Large Numbers. Econometrica, 55(6):1465, Nov. 1987. ISSN 00129682. doi: 10.2307/1913568.
URLhttpss //www/ wtt.:stttoooooooo...ooooo sttg:/table///// 91935356888 19111918811111i1.. or.. crossref
A. Bauer, M. Züfle, S. Eismann, J. Grohmann. N. Herbst, and S. Kounev. Libra: A Benchmark for Time Series Forecasting Methods. Proceedings of the ACM/SPEC International Conference on Performance Engineering, ICPE '21, pages 189-200, New York, NY, USA, Apr. 2021. Association for Computing Machinery, ISBN 978- 145503-8194-9, doi: 10. 00.144//427722134500244
M. Biloš, J. Sommer, S. S. Rangapuram, T. Januschowski. and S. Günnemann. Neural Flows: Efficient Alternative to Neural ODEs. In Advances in Neural Information Processing Systems, volume 34, pages 213377 Curran Associates, Inc., 2021.
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018
S.-A. Chen, C.-L. L. N. Yoder, S. O. Arik, and T. Pfister. TSMixer: An AlI-MLP Architecture for Time Series Forecasting, Sept. 2023.
E. De Brouwer, J. Simm, A. Arany, and Moreau. GRU-ODE-Bayes, Continuous Modeling of Sporadically-Obsere Time Series. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
W. Gilpin. Chaos as an interpretable benchmark for forecasting and data-driven modelling. In PPhrtty-Fifthh Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Aug. 2021.
R. W. Godahewa, C. Bergmeir, G. Webb, R. Hyndman, and P. Montero-Manso. Monash Time Series Forecasting Archive. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1, Dec. 2021.
R. Jennrich. Asymptotic Properties of Non-Linear Least Squares Estimators. The Annals of Mathematical Statistics, 40(2):633-66) 643, 1969. ISSN 0003-4851. doi: 2214//om//1177699
A. Johnson, L. Bulgarelli, T. Pollard, Horn, L. A. Celi, and R. Mark. MIMIC-IV. PhysioNet, 2021. doi: 10. (0.1366//RGGFXX
A. E. W. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng. M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark. MIMIC-III, freely accessible critical care database. Scientific Data, 3(1):160035, May 2016. ISSN 2052-4463. doi: 10. 1.318/data.220166
R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35 Mar. 1960. ISSN 0021-9223. doi: 10.1115/1.3662552
B. O. Koopman. Hamiltonian Systems and Transformation in Hilbert Space. Proceedings of the National Academy Sciences, 1155):31--318, May 1931. doi: (0.1073/pnas.17.55 17.5.315.
B. O. Koopman and J. v. Neumann. Dynamical Systems of Continuous Spectra. Proceedings the National Academy of Sciences, (3))2555-,,3 Mar. 1932. doi: 10.1073/pnas. 10..pna.......55.... 18.3.255.
M. J. Menne, J. Williams, and R. S. Vose. Long-Term Daily and Monthly Climate Records from Stations Across the Contiguous United States (U.S. Historical Climatology Network). Technical Report osti: 1e:492999 ediac:NDP -DP0019; doi: (001100.33/CDAA//CL.NDD Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) (United States); CDIAC, Jan. 2016.
Environmental System
F. M. Mione, L. Kaspersetz, M. F. Luna, J. Aizpuru, R. Scholz, M. Borisyak, A. Kemmer, M. T. Schermeyer, E. C. Martinez, P. Neubauer, and M. N. Cruz Bournazou. A workflow management system for reproducible and interoperable high-three thoouuhrouhuuttttt self-driving experiments. Computers & Chemical Engineering, 187:108720 Aug. 2024, ISSN 0098- 1354. doi: 10. co..0iee...mm.hemenngg 2024. 1887100
11

---

Published as a conference paper at ICLR 2025
Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam. A Time Series is Worth 64 Words: Long- term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
M. Schirmer, M. Eltayeb, S. Lessmann, and M. Rudolph. Modeling Irregular Time Series with Continuous Recurrent Units. In Proceedings of the 39th International Conference on Machine Learning, pages 19405. PMLR, June 2022.
R. Scholz, Born, N. Duong- Trung, M. N. Cruz- Bournzouu and L. Slhmidt--hiemma Latent Linear ODEs with Neural Kalman Filtering for Irregular Time Series Forecasting. Sept. 2022.
I. Silva, G. Moody, D. J. Scott, L. A. Celi, and R. G. Mark. Predicting in-hospital mortality of ICU patients: The Physionet/Comppuiin in cardiology challenge 2012. In 2012 Computing in Cardiology, pages 245 248, Sept. 2012.
M. Takamoto, T. Praditia, R. Leiteritz, MacKinlay, F. Alesiani, D. Pflüger, and M. Niepent. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Informa- tion Processing Systems, 35:1596-1611 2022.
G. Teschl. Orlinary Differential Equations and Dynamical Systems. American Mathematical Soc., Aug. 2012. ISBN
V. K. Yalavarthi, K. Madhusudhanan, R. Scholz, N. Ahmed, J. Burchert, S. Jawed, S. Born, and L.. Schmidt-Thieme. GraFITi: Graphs for Forecasting Irregularly Sampled Time Series. In J. Wooldridge, J. G. Dy, and S. Natarajan, editors, Thirty-Figtth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelli- gence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 16255-16263. AAAI Press, 2024. doi: 10.1609/AAAL V38115.29560.
T. Yu, C. M. Lloyd, D. Nickerson, M. T. Cooling, A. K. Miller, A. Garny, J. R. Terkildsen, J. Lawson, R. D. Britten, P. J. Hunter, and P. M. F. Nielsen. The Physiome Model Repository 2. Bioinformatics, 27(5):743-744 Mar. 2011. ISSN 1367-4803. doi: 10. 1000//boonformmttssstt22.....
A. Zeng, M. Chen, L.. Zhang, and Q. Xu, Are Transformers Effective for Time Series Forecasting? In AAAI, 2023.
W. Zhang, C. Yin, H. Liu, Zhou, and H. Xiong. Irregular Multivariate Time Series Forecasting: Transformable Patching Graph Neural Networks Approach. In Proceedings of the 41st International Conference on Machine Learning, pages 60179-600 791661 PMLR, July 2024.
H. Zhou, S. Zhang, Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 11 (06-11115, 2021.
12

---

Published as a conference paper at ICLR 2025
A REAL WORLD DATASET STATISTICS
Table 3: Statistics of evaluation datasets used by De Brouwer et al. (019 Bilos al. (2021) )
Schirmer et al. (2022). Max. Len. refers to the maximum sequence length among samples, Max. Obs.
refers to the maximum number of non-missing observations among samples. Sparsity refers the
percentage of missing values over all samples.
name
Instances
Channel
Max. Len
Max. Obs
Spars.
USHCN
1.114
5
370
398
78.0%
PhysioNet-2011 2012
11.981
37
606
80.4%
MIMIC-III
21.250
96
97
677
94.2%
MIMIC-IV
17.874
102
920
1642
97.8%
B PROOFS
B. PROOF OF LEMMA
1
Proof. Due Equation
it is sufficient to show that
(o((1) (x(T) (a(T)-x(()) 2
std [ ep2 qp
dl
(13)
1=
T
Due the mean value theorem, there exists Ge,e E - [t, with a(t) s(t)) x' (Gr,e) for all
t = e, ...,.... T. Given N = T and HN = N
1
Et=e,e.... , we thus have
AN N
Σ
(Se,e)
(14a)
1
T
x' (Gre)
(14b)
T
c-00
x' (((t)d) de
(14c)
T k
x (T) (0))
(14d)
T
due to the definition of the Riemann integral. Furthermore, we have
std diff 122
Σ
(4) (1) -
(15)
N
Therefore, again by the mean value theorem and the definition of the Riemann integral:
Σ
(z)
Σ
a' 2(((r,)2)
(16a)
N
N
1Ec,2.....T
t=c,
T
x'(
(16b)
T
tc,,
00
x' 2((1)2 dt
(16c)
Consequently,
std|z c-2
1
d'(t2
(I(T)) - x(0))
(17)
T 0
T
2'(1) (x(T) - -x(0)) 2
T dt.
(18)
T |=0
1=0
13

---

Published as a conference paper at ICLR 2025
B.2 ASSUMPTIONS AND PROOF OF LEMMA
For the lemma to hold we make the technical assumption:
(A) The distribution p of continuous functions [0, Φ: θ x [-1, + R, (0, + @00 t), with a compact parameter space O C F. and probability 8é measure jr O. We require that the map oz
given by continuous map
is continuous, too.
We sometimes use Tg for -00,-) and z'g for at (0,-). This is restriction for the differential equations we consider, as we can always obtain solution on [-1,7], given parameters (including x(0) zo).
Sampling from this distribution is done by sampling from the probability measure on θ A sequence of i.i.d random variables On. E N>o, distributed according to μ4, defines a sequence of random functions (En)nENsor Zn (t) @(0.,, f), as in the statement of the Lemma
For the proof will need uniform law large numbers (ULLN). There exist many versions of such theorems (e.g. (Jennrich, 1969), or (Andrews) 1987)), but we prefer to state and prove version with strong assumptions appropriate for our setting.
a continuous function on compact metric spaces è and I. Let 0,, N>u be i.i.d. random variables with values in θ, all distributed according to a probability measure μ on Then
Lemma 3 (Uniform Law of Large Numbers). Let f
N
Σ
f(Oitt) N ((0,t)d(()
(19)
N
uniformly in E T almost surely.
Proof. As θ I is compact and f continuous, f is even uniformly continuous. Hence, for any >> O. we can then choose a such that
d(0, 0') < and (t,t') < - (f(",t)) f(0,t) <e
(20)
Where d is the metric on O. Now by compactness of I, finitely many d-balls in cover it, i.e. there are
(1y-.., la E I, such that the B = I,...,k cover 1. For each (4g)==1:k can apply strong
law of large numbers to obtain almost surely convergence of N EN f0,,, 1,) to So f(0, t,), du(0).
Hence, almost surely, the following holds: For >> O choose No 0 such that for all natural numbers N 2 No and all j E {1,...,,k) k:):
1
5(0.4))) (o66))-((000 1(,,))dm00) <E
Σ
(21)
N
Now for any & E there is a E (1,...,,k) k) such that
1 - <e and
11e - -(0,)))((()))
<E
(22)
N
The triangle inequality yields
N / [(0, ((,)))du(0)
Σ f(0,,t) N
<
(23)
1=1
for all E I, which proves the claim.
Proof. (Lemma Let ,,, i E Nso be i..d. random variables with values in O all distributed according to a probability measure / on O.
We define @ the larger interval [-1, T), instead of [o, T), in order one-sided difference quotients.
14

---

Published as a conference paper at ICLR 2025
Define :b: : [0,
[0, by
(((.))-0(0-1 o(0,t-6)
if e > 0
(b 6.
(24)
De
at (0,t) t)
if
For 6 > O the function is obviously continuous as a composition of continuous functions. Let now be e 0 and t E [0, 7", E e. We claim that v is continuous at (0,0,t). ät is uniformly continuous on its compact domain of definition.
Let be greater than zero. We can choose a 7y > 0 such that for d(0,0') < 7 and |t - L'| < we
have
do de (0', σt (0, t)
Now chose (0','', with d(o, 0¹) and d(t,t') 7/2 and 0 0 é' <//2. If é' 0, we directly obtain
d6 (0', ,t'
06
",(0',0,,) - (0,0,t)] = 06
(0, t) <8
(25)
01
If é' > 0, the mean value theorem implied that there is a E (t' ,(,') such that:
06
00 at (0, 1) <
10 10'0'''','' -00,0.1)
(0', -)
(26)
0t
as t(T,t) S d(t,t') + d(t', t) é +2/2< 7/2 7/2 y This establishes the continuity of tb Therefore, eb is also uniformly continuous on its compact domain of definition.
Define ;: [0, 1] x [0,7] + by
((ee 1 (0),e,
(27)
We also define N. [0, x 10, T + R
1 N
(c, = N
Σ (0,,c, e,
(28)
The uniform law of large numbers shows that almost surely N converges uniformly Y for N + Do. Analogously define E: [0, 1] + R by
E(e,t) / (00,e, (00,e,1)ddu(())
(29)
We also define N. [0. 10, + IR
1 (c, EN,et
Σ c0),
(30)
N
Again, the uniform law large numbers shows that almost surely EN converges uniformly E for N + 0o. Hence,
std (+((),,, /V/ (E,tt) (E,tt)
(31)
converges almost surely uniformly to /E(ett) (eE, (addoe-(VV((E, for N + oc.
On the other hand by uniform continuity
(*) ())==
std 0, 1)) [))==(1;))
(32)
uniformly
(E(e,t) t)e,tt)z
c
stde (0, t))] std (t)
(33)
uniformly
15

---

Published as a conference paper at ICLR 2025
Now, given some 8 0, we choose an e' and almost surely an No such that for all 0 < <E< <e<d and 2 No:
std std e, (,)) std(( (.)
$ 8
(34)
(0((0))))
(v)) std (t)
8
(35)
(9((0,7))
and for 1, (t
(td(( =))== = 50
(36)
Therefore,
T / c, T /'" std x' (t) dt < 28
1
/"
(37)
and for T/e N the Riemann sum with step size e differs from the integral by at most T8, i.c.
e T/e
T / std[(w((,, ())== dt
Σ
e, <
(38)
T
the notations of the lemma, with + Jo² stdaop x de) = MPGD((p we get:
diff stdl( JE - MPGD(2) 36
(39)
T
Thus proving the claim that the left term of the difference converges almost surely for - 0 and N + oo the right term of the difference. Please that N and were chosen independently. The limits can be taken simultaneously or in arbitrary order.
16

---

Published as a conference paper at ICLR 2025
C RELATION OF MGD AND LIPSCHITZ CONSTANT
Regarding the relationship between the MGD and the Lipschitz constant of the vector field, we can consider the standard example of a linear ODE: a(t) = <t),, for which the Lipschitz constant is
precisely a. For simplicity, assume the initial condition z=0) = 1. Hence, the solution is xxt) = eat. Then, by (4) we have с = (ee - 1), and hence:
T / [+(()) cl1 dt
1
a
20T
oT
a
2T
T
27
27
7
a
0
T
(V2
Vgearr
MGD O
In particular, for this example, the MGD grows super-exponentiall with the Lipschitz constant. This result can be extended to an upper bound for general systems:
1. By the Mean Value Theorem, there exists & (0, such that c =(T)-=(0) =x{E) ((((E))
T
2. Therefore, [|=(t) cll2 = (t)(t) - =())2 = [l((((t)))))))))))))((()))))l[) - (P][=(t)=2(( - x2
3. We can bound the latter quantity via Grönwall's Lemma:
[I=(t) x<))ll
a(s) ds
f(((s) ds
g [I(((())))] ds
assuming f. 2
(I((x(s)) ((x(E)) ([(()()))
$ (t - E) (1f((=(()) +
L (|x(s) x((\)l ds
=a(t)
-B(s) =a(s)
s (t E) (lf((((((E)
c((--))
via Grönwall
And if & 2 t, we analogously get ||x(t) x(E) < (E - - ([((((()))))ee4 (15-)) In particular, note that by the same method:
If((((E)|I = (1x(T) 1
x(0) = ()(()(0)))
17

---

Published as a conference paper at ICLR 2025
5. We use this result to bound the integral:
T / [1+(1)
c11 di S / ||((1) - xxE)]|2 di
L
T
I?
(1) f [x((1) x(E))1? dt { [[((1) - x(E))1 di
T
L
(f (5- t)² e dt { (t E)2 22 dt
[1f(
$ T
(f(((E))) (2LT - +
2L ) 22.T
1
2L
x (1 2 0(122
O (1(((())))))TT")
6. Plugging this back into the definition of the MGD, we get:
1 {' [=((t) - c|f² - OOV 2( V T
1
MGD
18

---

Published as a conference paper at ICLR 2025
D DATASET DESCRIPTION
Table 4: Optimized Datasets created with generated Python code published on Physiome. A t-unit of " refers to Physiome giving the time unit as dimensionless, We entered - dur. staate conss and JGD, if we were unable to create dataset with method due to errors thrown by the ode solver or the occurence of odeeexplosions (see Section
Model
Domsin
Channel
Constants
Time Unit
T
d-state
d-const
ATVL
Calcium- Caleium-Dymanice Dynaanics
0.05
2.697
JELO1
Endocrine
2
15
30.0
0.5
0.05
2.58
DOKO1
Electrophysiology
0.05
2.277
INA01
Eloctrophysiology
58
10.0
0.5
0.05
2.218
WOL01
Signal-Transdees Transduction
9
19
min
10.0
0.05
1.973
BORO1
Calcium Dynamics
3
min
30.0
0.05
1.795
HYNU1
Metabolism
22
60
min
30.0
0.1
0.05
1.548
DUP02
Calcium- Caleium-Dynamics
3
19
min
30.0
0.1
0.05
1,202
WOL02
Metabolism
7
min
30.0
0.895
DIFUI
Electropbysioloyy
50
10.0
0.1
0.1
0.735
YANDI
Metabolism
38
10.0
0.05
0.407
DUP03
Calcium- Dynamics
2
10.0
0.05
0.254
BERD1
Transuccaansotiion
11
22
30.0
0.1
0.3
0.179
Endocrine
LI0I
Flectrophysiology
20
30.0
0.3
0.05
0.113
LI02
Electrophysiology
10.0
0.05
0.097
REVOI
Immunology
day
30.0
0.081
PUR01
Neurobiology
21
mBS
30.0
0.05
0.049
NYG01
Flectrophysiology
51
0.33
0.047
PUR02
Neurobiology
3
21
30.0
0.1
0.05
0.044
Flectrophysioloyy
REE01
Metabolism
4
21
30.0
0.5
0,05
0.035
VILOI
Signal- Transduction
9
h
20.0
0.028
KAR01
Signal- Transuction
21
10.0
0.05
0.025
SHOO
Excitation contraction Comnaiinn
105
ms
30.0
0.05
0.023
BUTOI
Electrophysiology
4
24
30.0
0.05
MALOI
Endocrine
9
51
day
30,0
0.5
0.05
0.019
Electrophysiology
0.019
BUT02
Electrophysiology
4
25
30.0
0.3
0,05
0.016
MUTОI
Immunology
7
day
30.0
0.015
GUPOI
Endocrine
4
30,0
0.05
0.014
GUYD!
Cardiovacalar Cirzulation
2
min
30.0
0.05
0.013
PHI01
Sreadian-Rhythme
3
0.013
GUY02
Candiovaseular Circulation
2
II
min
30,0
0.3
0.05
0.013
PULO1
Andiovaeular-Ciruulee
min
30.0
0.05
0.015
CAL01
Coll-Cycle
43
min
30.0
0.5
0.013
WOD01
Immunology
3
30.0
0.1
0.012
GUP02
Endrerine
4
30,0
0.1
0.012
MDI
Bardivaasular-Cirelltt
2
558
min
30.0
0.3
0.05
0.012
LEN02
Endocrine
3
KARUZ
Segna-Transductiinn
13
15
3.3
0.1
0.1
0.011
SHO02
Excistion-Conrracio--Cooo Comulliiiionn
105
ms
0.05
0.011
MACOI
Ion Transport
7
35
0.33
0.05
0.01
IR101
Elecinphysiology
0.33
0.1
0.01
RAGDI
Signal-Transian Transduction
9
30
3.3
0.06
0.01
WOL03
Mctabolism
13
28
min
30.0
0.5
0.1
0.008
WAN01
Caleium-Dynamiia
NELD1
Immunology
4
7
day
30.0
0.1
0.3
0.007
HUA01
Signal-Tran Transductonono
37
20.0
0.007
SVE01
Cell-Cycle
58
min
0.007
GUY03
Cardooascula-Chcclaiio
2
8
min
30.0
0.5
0.05
0.006
Cell-Cycle
min
LELO1
Signal Transduction
10
38
h
30.0
0.5
0.05
0.006
GARo1
Cell-Cycle
min
COU01
Elecinphysiology
49
30,0
0.05
0.005
AUT0I
Cardiovascular Circulation
2
570
min
3.3
0.05
0.005
GUY04
Sadiovascula-CCruule
min
0.05
0.005
RAZ01
Myofilameut-Mccha Mechanics
3
19
30.0
0.1
0.1
0.005
GRA01
SNF01
Calcium-Dynamicn Dynamics
Calcium-
5
26
1.0
0.5
0.3
0.005
WOD02
Immunology
day
0.005
WODD3
Immunology
4
day
30.0
0.004
NEUDI
Immunology
3
8
day
10.0
0.3
0.3
0.004
FAROL
Signal-Trandduction
0.004
SNE02
Calcium-
Calcium-Dynaniin Dynamics
3
13
30.0
0.3
0.1
0.004
Signal Transduction
Continued on next page
19

---

Published as a conference paper at ICLR 2025
Model
Domain
Channel
Constants
Time Unit
T
distale
dicumst
ATVL
BER02
Electrophysiology
0.004
GUY05
Cardiovascular Circulation
3
13
1tn
30.0
0.3
0.05
0.004
MCAD1
Electropyysiooggy
115
30.0
0.004
MCA02
Elecrophysiology
13
30.0
0.3
0.004
NOVO!
Cell Cycle
45
min
30.0
0.3
0.004
YAMO1
Excination Comppliiioo-Colling
5
20
3.3
0.004
RAL01
Mctabolism
142
mio
30.0
0.1
0.3
0.004
Electrophysiology
0.004
CAMO1
Myolilament Mechanics
4
10
30.0
0.3
0.05
0.004
VIS02
Electrophysiology
75
mB
10.0
0.05
0.004
SNYOI
Calcium Dynamics
27
30.0
0.003
NOROI
Flectrophysioogyy
69
0.33
0.1
0.08
0.003
LEL02
Signal-Transien Trandutoanoottioon
3
30.0
0.003
MAL02
Eloctropbysiology
29
86
30,0
0.5
0.05
0.003
UED01
Signal-Trran Transduction
CHRO1
Cell Cell-Cycle
36
136
min
10.0
0.5
0.3
0.002
POTOI
Endocrine
97
h
30.0
0.1
0.002
CHIR02
Cell Cycle
71
min
30,0
0.05
0.002
PAROI
Ton-Trane Don-Transport
7
35
3.3
0.05
0.002
JELOS
Endocrine
0.002
GOOD1
Cirvadian-Rhee Rhythms
2
6
30.0
0.5
0.3
0.002
Immunology
day
CS101
Cell-Cycle
14
93
min
10.0
0.5
0.1
0.002
НAToI
Signal Transduction
S0
30.0
0.1
0.002
VAS0I
Metaholism
30,0
0.002
MIT02
Electrophysiology
2
ms
30.0
0.3
0.3
0.002
Myotlament-Mechhame
REIDI
Signal-Transee Transduction
4
11
1.0
0.1
0.3
0.002
GUY06
Cardiovascular Circulation
GALOI
Electrophysiology
3
28
30.0
0.5
0.05
0.002
LEM01
Endocrine
28
day
30.0
0.05
0.002
SRI0I
Cell-Cycle
25
min
30,0
0.06
0.001
FABO1
Electrophysiology
25
81
ms
10.0
0.3
0.05
0.001
Caleium-Dynamme
min
LEM02
Endocrine
3
28
day
30,0
0.5
0.05
0.001
Immunology
day
HAU01
Endocrine
4
14
min
30.0
0.5
0.1
0.001
BOR02
Calcium-
Caleium-Dyuamice Dynamics
3
min
0.33
0.05
0.001
WOD04
Immunology
2
30.0
0.001
FINO1
Eloctropbysiology
27
61
ms
0.33
0.1
0.3
0.001
CHA01
Metabolism
HALOI
Cell Cycle
4
7
month
30,0
0.1
0.05
0.001
BAROI
Cell-Cycle
min
0.05
0.001
RAPO1
Caleiun-Dynamies
57
h
10.0
0.5
0,05
0.001
KOMOI
Immunology
2
h
30.0
0.1
0.001
NOV02
Coll-Cyele
31
min
0.05
0.001
NOVE3
Coll-Cycle
13
43
min
10.0
0.5
0.05
0.001
VEM0I
Signal Transduction
MTFO1
Flectrophysiology
5
14
0.33
0.1
0.3
0.001
COR01
Electrophysiology
0.05
0.001
BON01
Immunology
3
day
30.0
0.5
0.3
0.001
YATOI
Cell-Cycle
2
5
day
30.0
0.05
0.001
BON02
Immunology
day
0.001
BONG3
Immunology
3
8
day
30,0
0.5
0.3
0.001
Endocrine
min
17.01
2
30
10.0
0.1
0.3
0.0
VIL02
Signal- Signal-Transdone Transductionn
min
30.0
0.05
0.0
BER03
Electrophysiology
4
21
30.0
0.0
BER04
Elecirophysioloyy
21
ms
30.0
0.3
0.0
BERUS
Electrophysiology
KOM02
Endocrine
3
13
day
30.0
0.1
0.1
0.0
0.0
RATOI
Endocrine
3
13
30.0
0.5
0.1
0.0
BUB01
Electrophysiology
ms
20.0
0.1
0.0
GUY07
Cardiovascular Cireulation
4
27
30.0
0.05
0.0
HUNDI
Electrphysinlogy
70
ms
1.0
0.1
0.1
0.0
ROM01
Cell-Cyele
28
min
30.0
0.0
SHO13
Calcium Dynamics
7
40
1.0
0.1
0.1
0.0
GUY08
Bariovaseular-Crruule
min
0.05
0.0
POTO2
Endocrine
5
12
30.0
0.5
0.05
0.0
CIR0I
Cadiivascula-CCiruue
96
mS
30.0
0.05
0.0
STE01
Excitation contraction Coupling
4
30.0
0.0
PFD01
Endrerine
$4
ms
10.0
0.05
0.0
GUY09
Cadiooassula-Cirullle
min
GONDI
Endocrine
II
min
0.33
0,1
0.05
0.0
Cadiovaselar-Ciculdee
min
0.05
0.0
GUY10
Cadiovasala- Cireulation
17
min
30.0
0.1
0.3
0.0
Continued page
20

---

Published as a conference paper at ICLR 2025
Model
Domain
Channel
Constants
Time Unit
T
distale
dicomst
ATVL
POTU3
Endocrine
0.0
CLO01
Metabolism
7
42
0.33
0.1
0.1
0.0
CLO02
Metabolism
7
0.33
0.1
0.0
RANOI
Signal-Transtcction
Transduction
32
30.0
0.1
0.3
0.0
HYРO1
Cardiovascular Circulation
3
560
min
30.0
0.3
0.0
GUYHI
Cardioascular-Crrudbe
3
min
30.0
0.0
SEDO1
Tranl-Tranddocttoo
20
33
min
Electrophysiology
KATOI
Electrophysioooy
9
54
CUJ02
Electrophysiology
17
NIEO1
Electrophssologg
188
CHA02
Flectrophysioogy
38
KAT02
Electrophysiology
9
54
FAVO1
Electropbysiology
80
770
Elecirohhysooooyy
CLO83
Neurobiology
36
125
VINOI
Metabolism
342
DEMO1
Signal Transduction
55
CHE03
Signal-Tran Sagal-Transducttion
20
C0001
Signal-Trame Trandurranooooo
38
NAZ01
Metabolism
8
55
IIN02
Electrophysoolgyy
NA702
Metabolism
8
25
WAN02
Signal Transduction
CUROI
Metabolism
0
21
DAS01
Mctabolism
0
30
VIN02
Metabolism
HEYD1
Metabolism
11
18
Signal Transduction
PRO01
Tigndl-Transductionn
14
24
TRA01
Mctabolism
71
ms
REA01
Metabolism
60
BENO1
Eloctrophysiology
29
69
NOBU2
Electrophyysioooyy
LI03
Electrophysiology
12
37
Pipd
MAC02
Endocrine
4
14
min
MAI01
Cardiovascular Cinculaaion
7
58
GUY12
Cardiovascular Cireulaainnn
210
min
MA102
Cardoovasull-Chcuuaaa
3
21
KYR01
MAC03
Endocrine
7
28
min
Caleium-Dynamice
RICOI
Hignal-Transductte
11
BEN02
Mechanical Constitative Laws
26
PANO1
Excination-contaation wtte-oootnattiooooooiiiii Coupling
MA103
Cardiovascular Serdovvacula-Ciccaaaioo
7
32
Pipd
PMR01
Gene Regulation
0
0
GUY13
Cardiovssular- Cireuulaatoonnoonnn
MOD01
Cuddooascular-Ciiece Cireulation
88
PMRU2
Gene Regulation
0
GUY14
Cardiovascula--Ciee Cireulationnnno
4
min
ESPU1
Electrophysioloy
21
70
Electrophysiology
BOYD1
Flectrophysiology
22
90
ESP02
Electrophysiolgy
ASL02
Electropbysiology
49
OST01
Elecinphysiology
8
31
Electrophysiology
NIE02
Electopbysiology
5
25
Electmphyssology
MAHOI
Electrophysiology
80
SED02
Signal- Trandu-tannuctiion
38
min
21

---

Published as a conference paper at ICLR 2025
E LINKS OF ODE MODELS
Table 5: Model abbreviations with links to their Physiome page. The Physiome page contains information about the ODE model, the respective publication. Furthermore, the generated Python code can also be found there.
Abbreviation
Link
ASLOI
nt hLLps: / /mooels s ommpro ject httpa///.........ooooo.oooo.....
RECps: mww.1o......ommm
cre rect waw.aeaeae..........aaaaaaaaaaaa
AUTOI
Ject LL mmmeeleee phys ort.0//eee...........000000.....
BAGOI
Rttps: mogels........ phys cmeuro oFt. ott.e//.a.......................
BARGI
BAYOI
ht mooals physs ottp://ooooooooooaaoao.......//.
BEAOI
https:/
phys omepro ottp
BENOI
BENUZ
St:ps: mww.1o.ccccoommmmmmmm htwp//mmc....o.ceeeeeeee phwsicneDro wtt.
BENO3
01 LLCps1/10 models aelyssooeeps
BERO1
Rt:p5: moceis. wttpm//wwess.s.cemmoomom www.:ccooooscoooooooooooooooooo. cmeoro westeroooett..............eee rest. httpa///..e............aaaa.....
BERO2
pt
BER03
ht mona phys omepro
BER04
LL.ps:/ models phyy simeeoro
BEROS
RE models. chysicmeere jest, .tc.t..o./eee...... wwtp////e/eeeeeee//oo..o
BON01
St mow.1o..o..ooommoommm www.o/os.......neeeeeee. oct..oo//oooooo//////// http:///w.........../../////..//
BONO2
BONU3
St:p5: mocels.ccmmmmmm..m.mmmmmm.oommmm wtw.mmm...s....oo.ooomoom cmeoro weweeroooett..... ottp.//////oooe//eeeeeooooooo wttp://www....///2//../.........
BORGI
Gt mooals physt cmepro wwt..//rrooooo////////e/
BOR02
ht mode phys omepro
BOY01
RL mocels www.eeeess.............e..ee siomeoro htt.:////........//////////oo/r.
BROO1
RE moosis........m....mmmmmm..om..m shvsicmepro Test, .tt.ettrc/......e..eeeeeeeeoe wwtpo//geeooce////.ooo
St mo6315
BUTUI
BUTO2
nt:p5: mww.ls..ococoommmommmmmmm...m.mm http://mmms..s.......... cmepro wtt.eett.tooo..oeeeeeeeeeeeeeeee
CALDI
Tt mooa phys' onepro Ject p
CAMOI
nL //08:://moees modelo phyy sommpro
CHA01
RE:p5:/ mww.lo..c...oomm.ommmmmmmmooo..m wtwppoeess.........ooooo..o shysicmeers
CHA02
ht mo00 physsssss...s..-... cmepro jeat. http:///...000000000000000/a11e.
RE mww.ls..o.....mo...mmmmmmm.om..m whw.s Test wwtpo//o//ooeeee//ooooo
CHA0
CHEDT
CHED2
nt htt.://w................m..p cmopro wwr.oooooorrroeoooooooooo http:///e0...1111111100.00.000.. 768
CHE03
nt mode phys! onepro fectt
CHE04
AL / mwcelocccees..oommomm phyys simmeuro
75
CIL01
at:p5:/ mww.oo......ooomoooommoomoooo..m www.ss......................... www..physccoo... coeorro
CIROI
Rt mod.ls.......mmm...m physi omepro jest, wttp://pggto///////ptoo 2twp://0e//00111000000/332331011
C1.001
Rt monttt phys omepro pect jet
CLO02
mouels
ject, wttp:///c0111110000000.aaa......
C1.003
ht ht mooals phys omepro weo.ee.o.rr..............
CO001
Rt models physlomepro Lomegro .et.lergee.........oo
COR01
RE mww.looccs....mmo chysicmevre ottp.:///ssssse/e3331333 biccee/e
at:ps: mww.mo......mmmmo...mmmmmmmmom.. phwss omepro
UBbS
CSI01
p mooa physt omepro ject http///xpooo///////p//p/ wrtp:///eee.a..cccaaaaaaa. moodss
CUT01
ht: mode//oeeel..ooooooommmmooo phy.pooeprooooooooooooooooopo Lomepro ject
CUR22
nE PEFp8t7 mode 1s, phys omepro 0
CUR0I
haw.:////eeeeaaaoooooaooooooeoee abt 4a4
DAS01
RL mucels somegio ject oet..
DEMUI
Rt mww.ioo..ss...omo.... www.scceereeee.......oo..eeoeeoe htt.erreee...ooooeeeeeee....11..
DIFO1
nt mww.00......mmmmmm.mm phw.1 omopro wwt..ooggoooooorrroooooo http://wwwepaa.............aa...
DIXO1
RE ptspa:/ mode 16. phys omepro jeot wtw../rrr///oodee/////o/ooo hwwp://///mmmumooooosslsma4/421 Id 2802
DOKOI
nt RULps1 mode wpy.ssssppppp.............y www.ppyysssssse..
2ttp::///55522212222222020001100 9e
DUPO1
DUP02
nE pptp mogels physpphppsspssssep Lomppro ject pec..oroggo http:///pppaeeeencococoo...eeeo.
DUP03
StSps: mwcels..c....o.o..mmmmmmmmooommm shwsicmepss /tt.r//o.......ooo..1...1/000.1. htg.a//wee.e..cccccc.oooooo/eee.
ELED1
nt / mww.is........om.....mmmmm...m.m chweicoeero http://..///poo//////eeeeeee.e..
ESP01
t mooa Baeeeaas1..hhh... phys1 omopro jeat wtt..arrg.....poooooreee wtt.://gg/ogoo//oooooooo htt.
ESP02
nt Sp mode phys
FAB01
AU mwwels.cc.e..oooooommmmmmmooommm httpa//wwes.sseseeeeeoeee physiccevro het.:oooeeeoo..e3333222211333...
FAROI
FAR02
AL LLLpst/ / mucels physlomeero comeur ject, wtt.//orrg/eeee.......... ottpo//gopo.eeee1/22222222////11
Scbue
FAVOL
RE www./s.cc...ooooo/o www.scceeroeeocooooooooooooooooe 2tt.e//eeee..oo//2111111111.11.. wtt.:o/cceoiiescccc.00000000....
FINDI
st mod.ls.o.....oooo.ooo wwt.a//ww.....r.............e...
GAL.11
ht mona physt omepro
wwcprrow......... feat wttp:///..apa....a//000111100
GAROI
01 models. pww.sooeessyssssooooooo omepro h0tpp//eeeeooo////////////111100
GONDI
St models.
http://eeessssss..oooos whysscaeero htt.a//...........oo.oo.........
GOODI
St
whws:
GRAD1
at at www.///ccccc.ooommooo
/
Continued on next page
22

---

Published as a conference paper at ICLR 2025
Abbreviation
Link
GUPO2
GUY01
nttps:/ mooa10. www.seccccoeoeeeeeeec.. wtw.a//www......................
GUY02
Rtsps:/ wwwpcess.pbys..... cme Test. http://.w.......................
GUY03
GUY04
nt p.yys o11.::/0000..............000001.
GUY05
http8:/ mooe
onepro ject, httpprrrg.....oogoooaaa///////ea
GUY06
n1 LLLps: www.mmmccess.oommmmmmmmm
GUY07
Rt:ps: mww.oo..cccoommmmmmmmmmmmmmoommm pbws: cmepro (et..00///0003111155555 (tgpa3/00111111....e.e0001111111
GUY0
Rt modal5 phye ponporooeee org/ wtt.a//w............oo...a.aaa..
GUY09
Rt:ps: mww.wo.c...oommmmmmmmmmmm.oom..m www.mss.swssssecmmcooooo wtt.eepco..........oaeeaaaaaaaaa
GUY10
st
GUYII
nt ht mona phys omepro Tect. wwt.g///pprr/////aa.... hat.a:ahhh0000044444040003330333
GUY12
RL modele. plys sommepro prt.epooooo..ror.......... ottp:///oooa..eeeooo//o////////0
GUY13
RE mww.io..c...oommo..mmmmmmmommm.m whwsicmeero wtt.:o//ee......ascceeo... wtw.araeee....o.ee..............
GUY14
at:ps: moos1o.cc.mmmmmmommmmmmmmmmommmm w.w.s cmepro wtt.t..rr/eooooeeeeee
HALOI
RE patpat// mood 15, phys onepro joopppoooot.... jost, oFgg
Thoabr 2280 350.6 seat
HATOI
Rt:ps: / mww.io.ccs..cmmmmmmmmmmmm.oomm.m www.a/owws..........oo... chysscmeoro 1est. ort..oor//eossee/eeeeee.. http://cceee1111111eeeeee2211110
HAUDI
St moo.ls Phys: cmepro Jest. .rg.
69893788
HEYO1
ht mode phys physlomepro
omepro jeptrojeett.oo ject, wttpo//opare///.eeee... 1535931 (67 12055955 9c0655
HODOI
RL models
HUADI
RE mww.is..c....oooo.... wwwsscaeeroooooooooooooooooooooe Testerrooest..ooo oett.ooo///o/aa/ee2222222
HUNOI
nt moo015 p.y.s1 omopro Test. fest. wtwp/orr/exeooo.. wwtpor//ooo/////ooo www.//h///aaaaac
J6th
HUN02
nE pstpa171 mone 15. phys onepro joot,
s8an
ITYNOI
St:ps: / mww./o..os..ommmmmmmmmmmmmoommmm
HYPOI
phys' onepro ject, .tw.foog//eeee .oggg//pp////////.0. http::///....1tttccaaaaa33aa02
INADI
nt moge 1s why.spooppp comppro ject, /tt.
INA02
SE:ps:/ mwwels..c...oommmmmmmmmmmmoommmm wbwsicaeero wttp.o///eoooeeee/eeeeeee.ee
IRI01
nt mww.is......mmmmm..mmmmmmm...m.m
IZA01
ht monalooaa physt omepro wttp:///pp...///////.../.ooo
JELO1
nt tt mode 16. phys omepro ject orgg///oo//////49 /ttp:/4441100o2222422012200
JEL02
AU 00277/mmoce htt.://oeesssss.sssseem. Shysscmees
JEL,33
Pt /mooe/ooeeeeee phys! onepro peeeproject..ooo wrtpoo/r////////4111111
KAROI
LLLps:/4 mudels wonoyooooooooooooooooooooo
Paecl
KARUZ
St:ps:7 www./s......ooommoom htwp://wwc.....cceeeeeeeeeee
KATO1
St mod. www.ppppsssoo........ cmepro jest, ottpo//.tttooo///o//////////////
73388822
KAT02
ht mone phys' omepro
pocpproee........ ject. wtt..orrrexpc.r/////5..
KOMO1
01 models physsomeppo Lomepro
1822
KOMU2
St:p5: mwdels.co...oommo.mmmmmmm.ooom.m http://mccoess.s....oooo Shysscmeero oot..ooaaeep33311111113....
KYRO1
Lt mooele phys physsoneprs Lomepro (10.0///./00..000/000aaaaaaaaaa0
LEL01
nt models diysicmevre
St:ps: www./o......oommmoommooomoooommm wwwsiceepro wttpo/oo..o.000111......o... wtw.::/.w.......................
LEMOI
ht mooa physt omepro jost.
73cab
LEM02
nt /m phys ommpro wwcpproooot...oooooooo
353
LENDI
LENU2
St:p5: mww.1s..c...oommmmmmmmmmm.oomm.m http://mmsssssssssseee ott.c//ooooootooooo/ooeeeeeee/e.
LEN03
LLLps: models plyy siomegro
LI01
nE modeis.
Stops:/ www.1o......mmmmmmm wtwpo/css..phheemeeeee
1J03
ht mone 1. phys omepro htt.://gg//.ppo/e///aaeee/aeaae.
LIVOI
n1 PLLps1/ mode phyyppppseeeeeee wwwporowe......ooooeoo
9ec 4
M01
jest. wtt...../..0..e//...............
MACUI
St:p5: / mww.ms.ccssoommmmommmmmmmooommmm http://mmmssss...scmmoooom. pttperooeee...rr///sssseeee/ oot.a//psa....0000000o0aaaaa.a.
MAC02
MACIE
Rt moo.is.
MAHO1
nt www.15.0.5...mmmmmmmmmmmmmm
www.onnsss...hssooooo.m www.pphhsoooooo....... omopro www.oooooooorrooooo..oo wwt.prooo....ggg/oooooo.ooo
MATOT
01 pp5t/ mogels wtt.s/ess........e...eeeeee shysicmevro westrroeeee..... .ec.e.er..e..................... 2at.
ht mode phys omepro http:///////pg////11./...33
MA102
MA103
MALOI
St mooals wtt.:/mmcssss.......oo......
MALU2
RE:ps: www.ooiss..ss....eooo..oo..o.. whwsicmeess ott.. http://scccclccuoooaaaaaaaaaaaas
MCA01
nt mooals. phys www.scmeproooo.o..o.oo.oo..ooooe Test,
MCA02
nt www.mmnnann.mmmmmm 5. physs omepro wwt..ooggeecc..reoooooo
MICOI
01 Lt models Lonepro wtt.:o////sppeeeee/e1/03333
bu
MIFOI
2t Rt mocels. wttpooccss...ysseooooooo chysicmeers jest. http:////..ppsuee//eeeeeee...... 1320088 10 1138
MIJ0I
Rt
wew.eeeeeecccccoo 4900
MITO1
ht mona phys omepro poatrroear...o wttpg///ooo.////l//////
MIT02
At models ppyy:: chepro jec .org/ https/exxxareeee
MOD01
nt modals physs jest, wtt.://gg/xgo///eeeooom
15a. 1235266633 9na ues
NA701
RE mat.s///mmee.....m physs omepro wecprroeect...o.. het.eeeoooo....eeeeeoeeeeeeeeeee wtt..orreeoaar///aaee.
NAZ02
01 models wwwssohessseeo...
NELO1
SE Sttps: mocels.
www.:/ow........oooooooooo whwss cmeoroo rest, .est. wttpo////woss/eeeueeoee
whyss 1tt. http:///ee..........ooooaaeaaaa.
NEU01
NIF01
nt pp: /ww.//ooasa...ooooo phys omepro
weoptrooee...... ject.
NIE02
p mouels ppl.seessspyssss.e...... 51mmerro (eCL,0r 1ttp://epp0...1/1111333000
(00) l
NOBO1
ht pa1/ mona 5. physt omepro jeot, wttp://ggg//gooo.. /1
7:1 Tall
NOB02
R pp5 models phy. Lomegro wowpeooooooooooorroooeoo wtt..orr//poo.e//aalaa hww.
NOV01
Contioued next page
23

---

Published as a conference paper at ICLR 2025
Abbreviation
Link
NOV03
NYG01
Rtsps:/
OSTVI
OVED
nt p.ys cmepro 10t.
PAROI
http8:/ mooe o0 040.://0000000000000000001111111
onepro 2
PEDO1
AL AL mww.mmmoeess.oommmmmmmmm
PHID1
RECps: mww.oo.ccccmmmmmmmmmmmmmmmmoommm whws: cmepro
PMRO1
RE Rt moda15 phye omepro jest org/ wwt.000000001110221111.aa..a....
PMRU1
Rt:p5: mww.wo..c..oommm.mmmmmmmm.oom..m httpm//pcocs.s.eseeeeeeee.p wtt.://................1........
PMRO1
St
PMR02
ht mooa phys omepro ject.
PMR02
RL models sommepro Ject httppp/.g....o.rrroo///////////.
POTO1
RE mww.io..c...oomm....mmmmmm.mmm.m
at:ps: mocs1s...mmmmmmm w.w.s cmepro wtt.
POTO3
nF Pttpat/ modd phys omepro jost oFg wttp:////o//o////////////
PROO1
nt:ps: / mww.is.ccc.commmmmmmmmmmm.oommmm www.m/oo..........oommmm Shysicmeoro wtt..ooooeeoossseeeeee..oo. htt.:///we.......e.../.aaaaaa...
PULO1
pt
Phys:
PUROI
nE SE mode phys omepro
RELps:7 models phyyppyyyccoo oiyvico oet..orr//epa..e//eee (et.a//assbaaaaacceaa
ject ject, ottp:///////////////////a/aaaaa1
PUR02
RALO1
nE mww.im........omm.... wwwsscceerree.....c Testorooeeet..co ott. wttpa//www......................
RANO1
nt moo315 phys11 omopro Test, www.torg/e..e...eeeeeeeee.eeeeee wtt..ooo/ooooo/////h.oeo 11t.://01610000019190130033
RAPOI
pt:p mode phys onepro joot
RATOI
nt:ps: / mww.mo..osc.ommmmmmmmmmmmmoommmm http///mcc..........cmmm
RA7DI
Tt7 phys' onepro wew.toorg/eoo. f
REEDI
nL mogels www.pphopppppe... comppro ject wrwp/e/pa...e/beeeeee........ee. Mndeeeo
REI01
RE:ps:/ mwwelo..c...oommmmmmmmmmmmmommmm
REVOI
nt mww.is......momm...mmmmmmm..mm.m www.ppppssscs..e. weste wttp::/ooroooo////////ee///
RICOI
ht monal physt omepro jeat.
ROMO1
ht mode 16. phys omepro ject wtt.o/rrrr//eee/aaaaaa..
SCHO1
AU mwwels.......ooom wbw.ess.oess.sees...oo. Shysscmees
SEDOI
ht /ww.///mooel// phys' onepro
pectrrrectt..oo ject
SED02
AL mudels welyssomeeur
SHO03
St:ps:7 www.wo......ooommmmmm wtwp://mww.......meeeeeeeeeeeeee www.sceeero Test.eeet..oo ott.ee/eee.....o.ooo.....ee///e.
SNEDI
nt mod. www.pphysssooo........ cmepro jest, ott../.....e/////////////
ht mone phys' omepro
wecpprooe......... ject. wttpt/rrgeexx.e//ee_
SNYOI
01 models wwwslooeepeeee..
SRIO1
St:P5: 7 models.co...oomm.... http://ccoe.ss......ccoo Shysscmeero oet. /ttp0//0000/21334eeed000000
STEO1
Ut mooele phys Lonegro oet attp
SVEDI
nE models divsicmevre ottploo//ppssseeeeeeeeoe htt.:///w..../////////a////a....
THA01
at:ps: mww.mo......oommmoommmommmooom.m pwwsicmepro ott.oe.ooooooooooeeoo////ooo
TRADI
ht mooa physt omepro ject jost. htw.::/wwwwppp.....oaaaaaaaaaaa.
UED01
ht: /m phys
p ommpro
VANO1
01
VAS01
Sttp5:/ mww.1s......oomm...mmmmmm.oom..m aat.::cccceoeet...rrrreeeeeeees. oec.aaaacccccccccceeeaeaoaaaooo.
VEMO1
ALLps: models play siomegro jec.lorgee.pppe........... o1t.
VILOI
nE models. chysicmeere
Stops:/ mww.1o......mmmmmmommmmmmmmmmmmm wwwpe/sss..phssomoeeee
VIN0I
ht mone phys omepro jeot jetp.//rg/opoooreooooo /tw.
VIN02
n1 ppl1 mode
VISO1
jest.
VIS02
St:p5: mww.ms.ccssoommmmommmmmmmooommmm www.scceorooeee htt.:///ee...................e..
WANOI
Sttps:/ mocels.........o.... wby.ss...................... shysicnegro
WAN02
RE moo.is.
WODOI
www.onnsss....ys.oooo.mm www.pphhsooooo........ omopro htwprr/oo...o///ooo/////////////
WOD02
nE ht mode phys omepro jet..erg/eeooa......
WOD03
01 pp5t mogels wtt.s/els........ee..eeoeee chysicmevro wew.rroeeee...... /ot..r/r/e/pps////333332 http://ss.eat1111....22222221111
WODOH
WOLDI
St / moo.1s,0....omm oug.a/ae........................
WOL02
cmeoro Test. 4tt.:///........................
WOL03
st modnls.
omepro Jest. 20t.:///000...../////////4414110
YAMDI
t www.anall....ee..e. omepro bttp://ooooo..../.a..a..aa..aaa1
YATO!
0 /ode// someppooooooooooooooooooooooooo Jest 355. 00000000000000000000000001111111
24

---

Published as a conference paper at ICLR 2025
F HYPERPARAMETER SEARCH
We exclusively conduct hyperparameter search on fold 0. For GraFITi Yalavarthi et al. 2024) the hyperparameters for the search are as follows:
The number of layers, with possible values [1, 2, 3, 4].
The number of attention heads, with possible values [1, 2, 4].
The latent dimension, with possible values [16, 32, 64, 128, 256].
For the LinODEnet model (Scholz et 2022) we search the hyperparameters from:
The hidden dimension, with possible values [16, 32, 64, 128].
The latent dimension, with possible values [64, 128, 192, 256].
For GRU-ODE-Bayes (De Brouwer et al. 2019) we tune the hidden size from [16, 32, 64, 128, 256] define the hyperparameter spaces for the search are as
For Neural Flows (Biloš follows:
2021
The number of flow layers, with possible values 2, 4].
The hidden dimension, with possible values [16, 32, 64, 128, 256].
The flow model type, with possible values [GRU, ResNet].
For the CRU (Schirmer et al. 2022) the hyperparameter space is as follows:
The latent state dimension, with possible values [10, 20, 30).
The number of basis functions, with possible values [10, 20).
The bandwidth with possible values [3, 10].
25

---

Published as a conference paper at ICLR 2025
G REGULARLY SAMPLED TIME SERIES DATA EXPERIMENTS
In this chapter, we show important findings related to generating regularly sampled multivariate time series data with Physi come- ODE. discuss the generation process of datasets that were chosen based 2 important criterions. First is the previously mentioned JGD which was shown to be important indicator for the complexity of the dataset. Second criterion is for these datasets to include high number of channels dependency matters more and have more effect when tested on our ODE-datasets compared to current benchmark datasets.
show through the following small experiment, how channel
The datasets used here were DOK01, INA01, and HYNO1 which all have a number of channels greater than 15 with 18, 29, and 22 channels respectively. Furthermore, they ranked highly with respect to the JGD metric having values of 2,277, 2,218, and 1,548 respectively which makes them the most complex high dimensional (more than 15 channels) datasets in our benchmark. Datasets were generated based on the configuration at Appendix [D] generating 100 samples for each dataset based on 100 different initial conditions that are drawn uniformly random from distribution based on the configuratio were split into a train/val/test split of 70/20/10 which similar to the split done normally on the TSF literature for the weather, electricity and traffic datasets (Chen et al. 2023 Zeng al. 2023 Nie ctt al. 2023).
given to the model. Each time series sample spans 300 timesteps. The datasets
We conduct experiments on state-of-the-art Time Series Forecasting models PatchTST (Nie et al. 2023). TSMixer (Chen et al. 2023), and DLinear (Zeng et al.] 2023). For DLinear, there is only univariate model (doesn't capture channel dependency), but for the other models, they can be imple- mented as univariate or multivariate (capture channel dependency) variants. So overall, we label the univariate models as CI (Channel Indepedent), and multivariate models CD (Channel Dependent). These models are tested with the ODE datasets mentioned earlier tuning the hyperparameters for the 3 models through random search of 20 trials for predefined range of parameters.
For PSt we set the following hyperparameter range:
learning rate (range of floats) - [10- 110-7,10-31 1J
number of encode layers (range of integers) [1, 6].
Dimensionality of fully connected layer (categorical) -> [256, 512, 1024].
Embedding dimension of the attention layer (categorical) - [128, 256, 512, 1024].
dropout (range of floats) - [0, 0.9].
fully connected layer dropout (range of floats) - [0, 0.9].
patch size (categorical) - [4, 8).
stride (categorical) [2, 4].
For TSMixer, we set the following hyperparameter range:
learning rate (range of floats) - [10-7,10-2]
number of blocks (range of integers) [].
hidden size of MLP layers (categorical) [32, 64, 256, 1024].
dropout (range of floats) - [0, 0.9).
activation (categorical) - [ReLU, GeLUJ.
For DLinear, learning rate was tuned on the same range of values mentioned above. All models were tuned on the validation set. The results for the best performing model on validation set (lowest MSE error) were reported the test set. We used lookback window of 24 and forecasting horizon of 12 for all the conducted experiments on the ODE datasets, On the other hand, we report the results the mentioned models well-established evaluation datasets such as ETTh1, ETTh2, ETTml, ETTm2, Weather, Electricity, and Traffic on a forecasting horizon of 96. The reported results from Table give us interesting insights about benchmark the regularly sampled time series data.
26

---

Published as a conference paper at ICLR 2025
Table 6: MSE Test loss reported ODE datasets well on of the most popular regular TSF datasets. We highlight the best performing version (CI/CD) of each model in bold. The results were reported on 5 different baselines showing clear trend of CD models being better our generated ODE datasets. On the other hand, for benchmark datasets, this trend is not there as CI models are performing quite well.
Daraset
PatchTST(CI)
PachTSTTCD)
TSM)xerCD) TSMixerCD)
DLincar
INA01
0.099
0.052
0.168
0.062
0.191
0.155
0178
HYNO1
0.001
0.001
0.019
0.902
0.046
ETThl
0.375
0,416*
0.361
0.35y=
0.375
ETTh2
0.274
0.334
0.274
0,275*
0.289*
ETTml
0.290'
0.326
0.285
0.284
0.299*
0.167
Weather
0.152
0.168*
0,145
0,145
0.176*
Electricity
0.130
0.196*
0.131
0.132
0.140*
Traflic
0.367
0.595
0,376+
0.370
0.410*
Results are taken directly for PatchTST, TSMixer, and DLinear from the
respective papers (Nie et al. 2023).
(Chen et al. 2023). and (Zeng al.
2023
We notice from the results how unpredictable the results can be for the benchmark datasets, with CI models being better on some datasets and CD models being better on the rest. Also, one more interesting finding how close the test loss between very simple CI model such as DLinear and the tate-of-theeart models which can be concerning regarding how much channel dependency there on the available benchmark datasets. On the other hand, for our ODE-generated datasets, the results more consistent with clear better performance for CD models over their CI counterparts. Also, the best model overall was noticed to be always a CD model which shows the importance of capturing channel dependency on the ODE generated datasets. One more interesting finding is how worse is a DLinear model on our benchmark datasets where its results are at least 5 times worse than the best performing model. That shows how complex these datasets are, well how multivariate models can clearly benefit from intrinsic relation between the channels on such datasets.
27

---

Published as a conference paper at ICLR 2025
H LORENZ
We conducted a forecasting experiment on the chaotic Lorenz Attractor:
dx aly x)
(40)
dy x(p - 2)
(41)
dl
de zy B2
(42)
dt
Instead of varying the constants do in Physiome-ODE, set the parameters to
p 28
0 10
3
We vary the sample the initial states from x [1,3], y [0, 2]z 20. Following Gilpin (2021) 1), the task is to forecast the final of the IMTS based on the initial 6. Since we always use the exact same constants, we create 200 instead of 2000 time series instances. Outside the mentioned changes, the experimental protocol is the same as the one used in our experiments with the ODEs from Physi i ome-
As shown in Table 7 there is no model which significantly outperforms the constant baseline GraFITi-C.
Table 7: Test MSE IMTS created on the chaotic Lorenz-Attractor
Dataset
GRU-ODE
LinODEnet
CRU
Neural Flow
GraFITi
GraFITi-C
Lovenz
1.33420.097
1.313a0.123
1 1.292a0.877
1.34540.105
1.29420.090
1.30340.079
28


Reviews:
[{"review_id": "of0VlnCAXR", "reviewer": "Reviewer_q9fn", "summary": "The paper introduces Physiome-ODE, a novel benchmark for irregularly sampled multivariate time series (IMTS) forecasting derived from ordinary differential equations (ODEs). Physiome-ODE consists of 50 individual datasets created using biological ODE models. The authors highlight the need for a more challenging and biologically relevant benchmark for IMTS, as existing benchmarks primarily rely on only a few datasets and even simple constant-value baselines outperform complex ODE-based methods. Using Joint Gradient Deviation (JGD) as a metric, they select challenging ODE instances from the Physiome Model Repository, ensuring that the benchmark captures diverse levels of complexity. The paper also provides a comprehensive evaluation of state-of-the-art forecasting models, comparing methods based on neural ODEs with simpler, non-ODE methods.", "questions": "1. How feasible is it for other researchers to replicate Physiome-ODE in environments with limited computational resources?\n2. Could the proposed dataset generation method be adapted to capture personalized features in time series, such as patient-specific characteristics?\n3. How does Physiome-ODE perform against benchmarks like Monash Time Series Archive or PDEBench in terms of practical outcomes for IMTS forecasting?\n4. Given that the generated datasets are semi-synthetic, how closely do the generated ODE solutions resemble actual biological processes observed in real data? Would you suggest any metrics to measure the closeness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Significant Contribution to IMTS Benchmarking: The introduction of Physiome-ODE represents an important step forward in providing a robust and biologically relevant benchmark for irregular time series forecasting, filling a notable gap in the current research landscape.\n\n2. Use of JGD for Dataset Complexity: The introduction of Joint Gradient Deviation (JGD) to measure the gradient variance and dataset complexity is a well-justified and creative way to ensure that the generated datasets vary in difficulty, addressing the shortcomings of existing IMTS datasets.\n\n3. Broad and Diverse Dataset Generation: The benchmark is derived from biological ODEs, which are inherently multivariate and often irregularly measured. This connection to real biological processes makes Physiome-ODE highly relevant for practical forecasting applications, especially in healthcare and biology.\n\n4. Detailed Evaluation of State-of-the-Art Methods: The evaluation results indicate the diversity of the Physiome-ODE benchmark, where different models excel in different scenarios, highlighting no single model as the best for all datasets. This realistic scenario is useful for researchers to understand the strengths and weaknesses of existing methods.", "weaknesses": "1. The paper could benefit from a more detailed comparison against existing benchmarks for IMTS forecasting. While the authors do compare some models to existing datasets (such as MIMIC-IV and PhysioNet), a direct comparison of Physiome-ODE’s added value over these datasets using a common evaluation metric would be more convincing.\n\n2. The majority strategy for selecting challenging ODE instances, although effective in finding complex trajectories, might overlook personalized or localized causal differences that are crucial for domains such as personalized medicine. This lack of granularity could limit the applicability of Physiome-ODE to more individualized forecasting tasks.\n\n3. Creating and using Physiome-ODE is computationally intensive, especially with diffusion-based data generation and the JGD optimization steps. The paper lacks an analysis of how the dataset's computational demands impact its usability, particularly for researchers with limited access to high-performance computing resources.\n\n4. Physiome-ODE is a semi-synthetic benchmark, as the original biological datasets are often not publicly available. This limits the interpretability and direct clinical relevance of the benchmark since it relies on models rather than real patient data. A more comprehensive discussion on the implications of using purely ODE-generated data, including potential biases, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JT6x4vkaer", "reviewer": "Reviewer_ouEs", "summary": "An irregularly sampled multivariate time series (IMTS) forecasting benchmark called \"Physiome-ODE,\" which is derived from biological ordinary differential equations (ODEs), is proposed in this paper. Through the provision of a more extensive and varied collection of datasets, it seeks to overcome the shortcomings of the existing IMTS benchmarks, which are small and unvarying. The authors present Joint Gradient Deviation (JGD) as a metric to evaluate the complexity of datasets, asserting that it gives the benchmark a significant degree of rigor.", "questions": "- The integration of observation noise into this configuration is not adequately explained by Equation (2). While Equation (3) discusses the addition of noise to the generated IMTS data later on, but there is no obvious connection to Equation (2), so it is unclear how the noise is actually added in practice. The data generation process might be more rigorous if the differential equation were explicitly formulated to account for noise, as would be the case with a stochastic differential equation (SDE) framework.\n\n- Simple statistical models (like linear regression and ARIMA) and more complex models (like neural SDEs) are conspicuously absent from the selection of baseline models, which lacks rigor. \n\n- The existence and uniqueness of the solution are not discussed, particularly in Equation (12). The robustness of this optimization is doubtful in the absence of conditions guaranteeing that a unique maximum exists for JGD over these parameters. For example, what are Equation (12)'s spread parameter bounds? \n\n- Can the authors substantiate their assertion that JGD scales super-exponentially with the Lipschitz constant with empirical data or an example? \n\n- Lemmas 1 and 2 (Appendix B) are used to approximate MGD and MPGD with finite samples, but the assumptions on function continuity and differentiability are not stated clearly.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Physiome-ODE offers a novel approach to IMTS benchmarks by generating datasets using ODEs, which is an advancement over the few IMTS datasets currently available. The idea of creating datasets using biological ODEs may benefit the scientific community.\n\n- The paper offers a rigorous mathematical setup, especially when defining the JGD metric and the IMTS problem.\n\n- This paper has a wider empirical foundation because it includes experiments on 50 datasets. This could be a benefit when evaluating model performance variability.", "weaknesses": "- The theoretical explanation of JGD and how it relates to dataset complexity is unclear and excessively detailed. It is difficult to assess the reliability of the claims due to the heavy reliance on mathematical notation without adequate intuitive explanation.\n \n- Although the authors assert that JGD scales super-exponentially with the Lipschitz constant, they offer no supporting data or examples. Furthermore, without a comparison to other well-known metrics like variance or entropy, it is difficult to determine how useful JGD is as a complexity metric. \n\n- The robustness of the results under different experimental conditions is called into question because there is no sensitivity analysis on the numerical solver or noise parameterization. Additionally, there is limited discussion on the generalizability of Physiome-ODE beyond biological applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oB75rhjIr6", "reviewer": "Reviewer_nAcA", "summary": "This paper introduces a new benchmark for multi-variable time series (IMTS) prediction called Physiome-ODE, which is generated based on biological dynamical equations (ODEs). The current evaluation methods for irregularly sampled and missing-value time series prediction mainly rely on limited datasets, which may not accurately assess the performance of models due to their small size and diversity. The authors develop a new methodology to generate and filter challenging IMTS datasets from ODEs, successfully creating a significantly larger and more diverse benchmark than existing evaluation settings. Physiome-ODE consists of 50 independent datasets derived from ODE models used in biology research over decades. By comparing the performance of existing IMTS prediction models on this new benchmark, the authors reveal different strengths among the models and indicate that some current prediction models can demonstrate stronger abilities on Physiome-ODE compared to traditional evaluation datasets. Additionally, the paper proposes a new metric, Joint Gradient Deviation (JGD), to measure the difficulty of datasets, demonstrating that the benchmark can effectively distinguish between datasets and models of different complexities. The introduction of Physiome-ODE not only provides a more comprehensive and realistic assessment platform for IMTS prediction but also promotes future research in this field.", "questions": "- Why is Joint Gradient Deviation (JGD) introduced, and what specific advantages does it offer in creating benchmarks?\n\n- The authors do not adequately address how the Physiome-ODE dataset ensures representativeness and reliability. Without a clear explanation of its distinctive features and validation processes, the dataset's credibility as a benchmark for time series prediction remains uncertain.\n\n- 5-fold cross-validation is used for model evaluation. Is this partitioning sufficiently representative of the dataset's diversity? Could the way the data is split introduce any biases in the results?\n\n- By focusing solely on ODE-based models, the authors fail to incorporate recent advancements like TimeMixer and TimesNet, which may offer alternative or improved performance. This omission results in an incomplete evaluation, as it leaves out potentially competitive models that could impact the study's findings.\n\n- The authors do not clarify why channel-independent models (e.g., PatchTST, DLinear, PDF, SparseTSF) are reportedly more effective on standard datasets, nor do they provide experimental evidence to support this observation. Without a clear rationale or relevant experiments, the paper’s insights into channel dependencies remain unsubstantiated, limiting the strength of its conclusions. Furthermore, this work does not consider the impact of data stationarity on the results.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Physiome-ODE provides a larger and more diverse benchmark, allowing for a more comprehensive assessment of IMTS prediction models.\n\n- By comparing existing models' performance on Physiome-ODE, it is possible to discover some models with stronger abilities in handling irregular sampling and missing values.\n\n- A new metric called Joint Gradient Deviation (JGD) has been proposed, which measures the difficulty of the dataset and helps distinguish between different complexity levels of datasets and models.", "weaknesses": "- The contribution of the proposed Physiome-ODE dataset is not clearly articulated in the manuscript, making it difficult to understand its unique advantages compared to existing time series forecasting benchmarks.\n\n- The study only considers ODE-based predictive models and overlooks more recent models, such as TimeMixer and TimesNet. This limited model selection restricts the breadth of the comparison, potentially missing insights that could be gained from newer approaches.\n\n- Although the study highlights that datasets created with Physiome-ODE encourage models to learn channel dependencies, it does not explain why channel-independent models like PatchTST, DLinear, PDF, and SparseTSF are observed to perform better on traditional datasets. This lack of explanation, coupled with the absence of supporting experiments, leaves the findings incomplete and reduces clarity on model performance differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJhHPbQsIm", "reviewer": "Reviewer_Sy9P", "summary": "The paper presents a set of ODE generated benchmark dataset for Irregularly Sampled Multivariate Time-Series Forecasting task. Furthermore, it proposes a complexity metric (JGD) to evaluate the difficulty of different datasets. Finally evaluated common forecasting methods and a baseline predicting constant time series.", "questions": "1) Did you checked that parameters you modify by resampling from a distribution still meaningful? eg.: parameters have to be positive are positive, parameters have to have a given order (e.g. a > b always hold by biology) are such, logarithmic scale parameters are sampled in logarithmic scale etc.  Same for initial condition. \n2) How the suggested complexity measure assesses Lorenz system for example? \n3) You mention the necessity of normalizing the system values. How about time? What happens if a system timescale changed from measured in sec to measured in hours?\n4) Describe the training protocol what is the loss computed on, how much ahead prediction is evaluated. Best if you create a figure, If it is changing for some methods, describe it specifically.\n\nMinor:\nI would consider calling the \"ode constants\", “ode parameters” as this is more frequently used, or mentioning at least both name, as it can be confusing.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents a new set of baselines for Irregularly Sampled Multivariate Time-Series Forecasting, As there is a need for standard baseline in this area therefore the contribution is significant. \nIt also provides code to test existing methods on the benchmark, and introduces a new metric to assess the complexity of a given dataset. This metric correlates with the lowest prediction error achieved by the tested models.", "weaknesses": "The new benchmark is a standardized way of generating data from already public ODE models. This would still be valuable, but some of the choices are quite ad-hoc including the parameter noise and the initial condition selection.\nThese systems have significantly different sensitivity for different parameters and can change between operation regimes by tiny change of some parameters.  Currently the authors use a common $\\sigma_{\\text{const}}$ and hope for meaningful results. If this not happen (ODE explodes for example) the given time series is simply dropped. This may invalidate the benefit, that there is meaning behind these curated ODEs, by setting physically (or biologically in this case) nonsensical parameters or initial conditions. Expert knowledge should be used to decide what parameters should be fixed what can be modified, and sensitivity analysis should be carried out to create meaningful time series.\n\nThe mean gradient deviation (MGD) as metric of complexity is ad-hoc and questionable. As it is mentioned in the paper it automatically assumes faster oscillatory processes are less predictable. How about a huge (length of the rods is large, so in average slowly moving) double pendulum in its chaotic regime? Is it less complex than a very fast sine wave? Also systems like the Lorenz system changes between regimes (two “wings” of the “butterfly” ) suddenly but quite smoothly evolving in between, it seems this is not captured by the metric.\nCorrelation is found between the error and the composite metric (JGD) but it can be spurious, see bellow. \n\nIt is not clear from the paper how the training and the evaluation of the methods are carried out.  In the Experimental protocol (Line 351-352) It stands  “In our experiments models have to predict the last 50% of the time series after observing the first 50%.” How the ODE models were trained? the loss in train time was computed on similarly long sequence and backpropagated or it was trained on a single point future prediction task. \nIf the later that the methods had no fair chance to learn to, for example, fall back to the constant model, as they clearly trained optimizing short term error, and go out of phase. Due to the comparatively excellent result of the constant model, it needs to be cleared that this is not the case. In this regime fast oscillating systems are less predictable as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)