True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics
Authors: Christoph Jürgen Hemmer
Daniel Durstewitz
Venue: NeurIPS 2025 poster ()
Keywords: Dynamical Systems
Dynamical Systems Reconstruction
Chaos
Recurrent Neural Networks
Foundation Model
Primary area: deep_learning
TL;DR: DynaMix, a novel pre-trained foundation model for zero-shot inference of dynamical systems, outperforms time series foundation models in terms of long-term statistical properties, forecasts, and inference times.

Abstract: Complex, temporally evolving phenomena, from climate to brain activity, are governed by dynamical systems (DS). DS reconstruction (DSR) seeks to infer generative surrogate models of these from observed data, reproducing their long-term behavior. Existing DSR approaches require purpose-training for any new system observed, lacking the zero-shot and in-context inference capabilities known from LLMs. Here we introduce *DynaMix*, a novel multivariate ALRNN-based mixture-of-experts architecture pre-trained for DSR, the first DSR model able to generalize zero-shot to out-of-domain DS. Just from a provided context signal, without any re-training, DynaMix faithfully forecasts the long-term evolution of novel DS where existing time series (TS) foundation models, like Chronos, fail -- at a fraction of the number of parameters (0.1%) and orders of magnitude faster inference times. DynaMix outperforms TS foundation models in terms of long-term statistics, and often also short-term forecasts, even on real-world time series, like traffic or weather data, typically used for training and evaluating TS models, *but not at all part of DynaMix' training corpus*. We illustrate some of the failure modes of TS models for DSR problems, and conclude that models built on DS principles may bear a huge potential also for advancing the TS prediction field.

True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics
Christoph Jürgen Hemmer Daniel Durstewitz!:2a
Deptt of Theoretical Neuroscience, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
Interdiscipllinary Center for Scientific Computing (IWR), Heidelberg University, Germany Facultyy of Physics and Astronomy, Heidelberg University, Heidelberg, Germany {christoph. hemmer, daniel durstewitz)@zi eeere/iiiimmameeeeee -wwnnmminndeeec.
Abstract
Complex, temporally evolving phenomena, from climate to brain activity, governed by dynamical systems (DS). DS reconstruction (DSR) seeks to infer generative surrogate models of these from observed data, reproducing their long- term behavior. Existing DSR approaches require purpose-training for any new system observed, lacking the zero-shot and in-context inference capabilities known from LLMs. Here we introduce DynaMix, a novel multivariate ALRNN-based mixture-of-experts architecture pre-trained for DSR, the first DSR model able generalize zero-shot to out-of-domain DS. Just from provided context signal, without any re-training, DynaMix faithfully forecasts the long-term evolution of novel DS where existing time series (TS) foundation models, like Chronos, fail a fraction of the number of parameters (0.1%) and orders of magnitude faster inference times. DynaMix outperforms TS foundation models in terms of long- term statistics, and often also short-term forecasts, even on real-world time series, like traffic or weather data, typically used for training and evaluating TS models, but not at all part of DynaMix' training corpus. We illustrate some of the failure modes of TS models for DSR problems, and conclude that models built on DS principles may bear a huge potential also advancing the TS prediction field.
Introduction
Context
Forecast
Geometry
Spectrum
DynaMix
Reconstruction of state space geometry Long-term temporal statistics
Mantfol www. www ww Mol
Context
Forecast
True
Attractor Geometry
Power Spectrum
Chronos
Dimension misalignment Long term failuree
1/ wmw. www
Context parroting
Figure 1: DynaMix achieves zero-shot DSR attractor geometry and long-tern fnngtermmae temporal properties (power spectrum) from short context signal while Chronos [2] fails.
Most real-world processes, from atmospheric phenomena and stock markets to brain activity or ecological networks, be deseribed as dynamical systems (DS) [58, 16, 88, 61]. Reconstructing
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

these from observational data, called dynamical systems reconstruction (DSR), has been a long- standing challenge in scientific modeling [101, 24, 321. DSR goes beyond conventional time series (TS) modeling, as we wish to have generative model of the underlying process which exhibits long-term behavior with the same temporal and geometrical signatures (Fig. 1), i.e. the same invariant or "climate' statistics [68, 72], as important in scientific applications. Achieving this usually requires special -hhrro-- training techniques [60, 40] or loss objectives [71, 72, 43, 78] which accentuate the system's 's long-term dynamics. Numerous deep learning approaches for DSR, based on recurrent neural networks (RNNs; [87, 23, 90, 17, 7, 75, 40, 8, 72D,, neural ODEs [18. 46, 1, 48], Koopman operators [15, 57, 65, 4, 62, 29, 93], library -based methods [13, 55, 44, 20, 59], have been advanced over the years. However, all of these require purpose- trainiing on the specific system observed and struggle to generalize beyond their training distribution [34].
Inspired by the strong in-context and zero-shot generalization abilities of L.MSs [11, 28, 22, 19], there has been a push recently to develop models with likewise properties for the time series domain. Time series foundation models like Chronos [2, 3] or Mamba4Cast [6] are pre-trained on a huge database of real-world and artificial time series, and then tasked to forecast novel time series from which snippets are presented in-context, without any parameter fine-tuning. They are not built for DSR, however, and as we show here typically fail to properly capture system's 's long-term behavior and the structure of its attractors (Fig. 1; [102]).
To address this gap, we introduce DynaMix, novel mixture-of -exture-of-experts model designed for DSR and zero-shot forecasting. By training across a diverse range of DS, DynaMix learns transferable representations, enabling it to forecast previously unseen systems without retraining, including their underlying attractor geometries and invariant statistics (Fig. 1). The core features of DynaMix are:
Accurate zero-shot DSR: DynaMix achieves strong generalization across diverse DS, eliminating the need for fine-tuning while maintaining accuracy in reproducing attractor geometry and long-term statistics. No other model tested achieved this.
Multivariate information transfer:
Due to its multivariate architecture, the model ef- ficiently captures dependencies among multiple system dimensions, enabling accurate reconstruction of their coupled dynamics. Further, is neither bound specific dimen- sionality nor to a specific context length, but can flexibly adapt to other dimensions through specific embeddings.
Computational and parameter efficiency: The model reaches high performance with a very lightweight architecture (s= 10k parameters) and small training corpus (34 systems), enabling orders of magnitude faster inference than other foundation models.
Interpretable dynamics composition: The model provides insight into the dynamical composition of reconstructed systems, elucidating similarities between different DS
Time series forecasting:
Beyond DSR, our model is applicable to general time series forecasting where we employ different embedding techniques to obtain data representations accessible to DSR.
We evaluate model multiple benchmark DS and real- world time series, demonstrating superior zero-shot generalization on DSR problems compared to current TS foundation models.
2
Related work
Dynamical systems reconstruction (DSR) In DSR we seek to learn generative models from time series data that represent the underlying system dynamics with all its topological, geometrical, and temporal properties [24, 34, 72, 31, 14, 12]. in that sense providing an approximation to the underlying system's 's governing equations. By definition, such model should not only render viable short-term forecasts, but also reproduce the long-term evolution of the DS it has been trained on, both in state space and in the time domain. Methods approaching this goal have been founded on predefined function libraries, such as Sparse Identification of Nonlinear Dynamics (SINDy; [13, 55, 20, 44, 59),, on reservoir computers [69, 71, 72], neural ODEs [18, 46, 1, 48), Koopman operators [15, 57, 65, 4, 62, 29, 93], or different types of RNNs [87, 23, 90, 17, 7, 75, 40, 8). More important than the architecture itself seems to be their proper training in order to ensure long-term (invariant) statistics of the underlying system are met: Methods like sparse [60, 7] or generalized [40]

---

teacher forcing allow RNN-generated trajectories to 'explore the future' whilst training, yet keep loss gradients in check. Altematively, long-term statisties based on the observed system's 's Lyapunov spectrum, fractal geometry, or invariant measures may be added to regularize the loss [71, 72, 43, 78), but require to compute these properties from the data first. Despite these advances, out-of-domain generalization remains a key challenge in DSR [34]. Meta- and hierarchical learning models, trained across many DS simultaneously, have recently been designed as steps toward foundation models for DSR [10, 64, 47, 100], but still require parameter fine-tuning and lack in-context inference capabilities.
Time series foundation models Large language models (LLMS) exhibit impressive ability to infer patterns from prompts (context) and generalize to novel situations without retraining [11, 28, 22, 19]. although the underlying reasons for this are still matter of debate [53, 92, 99]. This inspired the development of general general-purpose purpose time series (TS) foundation models which could accurately forecast time series from short segment provided 'in-context' [2, 27], without the need of task-specific fine- tuning [21]. One idea is to simply use pretrained LLMs directly as TS forecasters [35, 73, 81, 95, 85], but the inherent differences between textual and many temporal data, such continuity, present sigmificant challenges [84]. To overcome these, transformer based architectures, similar in design to LLMs, such Chronos [2. 3] or TimesFM [21], were specifically pretrained on a large corpus of time series data. Promising zero-shot forecasting capabilities have also been achieved with alternative architectural designs, such Tiny Time Mixers [27] state-space models like Mamba [6].
The success of TS foundation models raised hope these could also be utilized for zero-shot DSR, but the to our knowledge and fell short of a full DSR evaluation [102]: Successful DSR, if present at all, heavily depended on the initial conditions, and mechanisms such as context parroting visually gave the wrong illusion that features of the dynamics had been captured. In fact, here we show that existing TS foundation models generally nos capable of producing valid DSRs (Fig. 1), and highlight some of their failure modes. This is in contrast to the zero-shot DSR foundation architecture we develop here, based on models [8] and training algorithms [60] successful in DSR.
far only previous study on this, based on Chronos, had mixed outcomes
3 Methods
3.1 Model architecture
To enable zero-shot reconstruction of novel DS, develop specific mixture-of-expeer (MoE) architecture that can be pretrained across many diverse DS (Fig. 10), with different experts possibly specializing in different dynamical regimes. As SOTA DSR base model for the experts, we leverage a recent parameter-friendly RNN, which allows for highly efficient DSR training and is designed to yield topologically parsimonious and interpretable representations of DS it is being trained on, the Almost-Linear RNN (AL-RNN; [8]):
21 A Wh' (21-1) + hh.
(1)
The model describes the evolution of M-dimensional latent process 2, E R with linear self- connections A E diag(RR (TR M) weight matrix W E MRMX XMM bias term h E MR,, and *z(z) defined
s (z,) = [2L,. ... PM- max (0, 2M 2M-P41)), ... max zM,))l"
(2)
i.c with ReLU nonlinearity on only P <<< M of the M AL-RNN units. The first N units are interpreted as the network's 's readouts and provide the predicted observations, X {w. = 21,N,1 N where N is the observation dimension.
The selection of AL-RNN experts is achieved through a gating network (Fig. 2), which receives as inputs the (generally multivariate) context time series C= {c,) RNTe as well as the current latent state 2, Both are passed into a state attention mechanism defined as
C C-
(Dz, + 1T
IN
w
att
E TTC
Tat
where D E N is a learnable matrix which maps the latent state into observation space, 6 ~ AO Σ) exploration noise with a learnable covariance E, Fan learnable temperature parameter,
3

---

1 (To.N) column vectors of ones of length Tc.N, respectively, and o(-) the softmax returning normalized weights. Hence, this mechanism computes attention weights based on some distance between projected latent states and actual context observations. At the same time, the context signal C' is processed by CNN, yielding temporal features C RNxTe, These features are then weighted with the attention weights uet and together with the current state 2, further processed by multi-layer perceptron (MLP) followed by a softmax, yielding set of expert weights at any time t:
cap a (LLP(CW, 21. 2
R
(4)
Toxp
where Texp another learnable temperature parameter. The corward-iteratee latent states z/_1 1 of the individual experts
{1, ..., } are then weighted by w. yield the next time step prediction exp 2 Fig. 2 illustrates the whole approach. Note that one key advantage of this Z(+1 = Ej= W j.C compared to other TS foundation architectures is that the context length is flexible by design, i.c. the CNN and the attention mechanism may take input context signal of arbitrary length.
Expert Prediction
Current state Zt E R&M
E++1
x
W
Predicted state
1 IR
AL-RNN
W
Gating Network
Context I NXT.
Attention
Observation
8 11:Nc011 R
MLP
CNN Encoder
W
Figure 2: Illustration of the DynaMix architecture. At test time, only a context signal C' is provided and guides the selection experts to yield arbitrarily long forward predictions of the dynamics.
3.2 Model training
For training, collection of just 34 different chaotic and cyclic DS is used [30], see Appx. A.2 for full specification and Fig. 10 for illustration. From each, multivariate time series X E R xT are simulated (== x 10 in total) and then standardized dimension wise (likewise for the test set), of which the first Tc < column entries are defined as the context signal = X1:7e- The experts initialized l Te: Al + I and forward-iterated until time T. such that they overlap for period A/ with the context. This is to ensure that the model learns to optimally utilize the context for generalization. A MSE loss is then computed across ground truth, 1T and respective model-generatee time series X. The model is trained by sparse teacher forcing (STF; [7, 60]), control-theoretic technique specifically designed for DSR. STF replaces part of the forward-iterated latent states 2, by data-inferred states &, at optimally chosen intervals т, to avoid exploding gradients even for chaotic systems while enabling the DSR model "to explore the future" (see [60] for detailed theoretical motivation). shown Fig. 26, STF with optimal essential for achieving good zero-shot DSR results. Note that STF is only used for training and turned off at test time.
3.3 DSR evaluation & time series forecasting
We used two established measures to assess DSR quality [49, 7, 40, 97, 102, 72, 60, 66, 31]: First, to quantify agreement in state space (attractor) geometry, we employed Kullback-Leible divergence defined across space, Dstsp (see Appx. A.I for details). For assessing agreement in long-term temporal properties, we used the Hellinger distance D, defined across power spectra of the true and -ener-te trajectories. Crucially, to properly assess the long-term dynamics, both these measures were evaluated in the limit of large T 10, 000. In addition, short-term, n-step-ahead

---

mean absolute prediction error (MAE) was evaluated. As has been repeatedly emphasized in the statistical and DSR literature [97, 49, 7, 60, 70, 31], prediction errors defined on time series are often only sensible on short time scales, because of the well known exponentially fast divergence of trajectories in chaotic DS (see Fig. 24).
Empirically observed time series are often just one-dimensional (N* = 1), but come from an inherently much higher-dimensional underlying DS. To work efficiently in a DSR setting, the context signal should represent the underlying DS as good as possible, for which it commonly needs to be lifted into a higher-dimensional space where sets explored by the dynamics ideally become diffeomorphic to those in the system's is true state space. Temporal delay embedding is the most popular technique to achieve this [82, 76]. For 1d time series {ar). d-dimensional embedding can be defined as [50]
emb (Ia, Tt-r: Tt- zz
(5)
where the 7i are time lags, commonly estimated from the autocorrelation of the time series. Alter- natively, if the delay embedding would become too large (d N), augment the empirically observed time series by a variant of the positional encoding common in transformers, defined here as
2Ttl + σ1 1.. sin
)1))...
2ml (- + )
emb
Ze. sin
(6)
where arg manmmax+++mm Tiin Tmin [[ is given provided it exceeds threshold Tmin by the maximal autocorrelation, and random phase de E [0, 3] is assigned to each dimension.
4 Results
4.1
Zero-shot DSR
a
Time Series
Power Spectrum
State Space with Vector Field
State Space
MW WWW
10
203
1014
MMMMMM
10-
500
1000
2003
400
02
Contaod
41
Figure 3: a) DynaMix zero-shot DSR (red) compared to ground truth (lightgray) using a 500-step context (darkgray) for the Sprott M system. b) Zero-shot forecasts for the Selkov DS (true vector field in lightgray) from different initial conditions (red) outside the context range (darkgray). e) DSR quality function of context length for Lorenz-63. d) DSR quality function of the temporal resolution At of the context signal. Error bands STD
Figs. (top) & 3a present example zero-shot DSRs of new chaotic DS, which were not part of DynaMix' training domain, from just provided context signal C (see Figs. 11-15 for further examples). Despite the inherent challenge of predicting chaotic systems, where small differences in initial conditions lead to rapidly diverging trajectories (Fig. 24), DynaMix successfully captures the system's temporal & geometrical behavior, i.e. accurately reconstructs the underlying state space and faithfully recovers the true system's 's power spectrum. Fig. 3b highlights further aspects of DynaMix' reconstructions: Even though not trained on any 2d systems, the model successfully recreates the limit cycle from the 2d Selkov system [79]. Notably, the model even generalizes to new initial
5

---

conditions outside the scope of the context data, i.c. correctly infers properties of the state space beyond the short context trajectory (see Fig. 22 for further examples). Likewise, although trained only on 3d DS, DynaMix also successfully generalizes to sigher-dimensiioal DS, as illustrated for the 6d Lorenz-96 system [56] in Fig. 34 (see Appx. A.6 for details on the setup and comparisons to other models). As further shown in Fig. 3c, the context length necessary to achieve good DSR quickly converges, with as few as 500 time steps often sufficient. Fig. 3d illustrates that our approach also works across surprisingly wide range (about an order of magnitude) of temporal resolutions at which the underlying DS was sampled, confirming its robustness w.r.t. sampling frequency.
We next compared the zero-shot, out-of-domain DSR performance of DynaMix to that of a number of recent TS foundation models, including various versions of Chronos [2], Chronos-2 [3]. Panda [52], Mamba4Cast [6], TimesFM [21], and Tiny Time Mixers
[27]. using the long-term statisties Dttsp and DHr for evaluation (see Sect. A. As seen in Fig. 4a-b, DynaMix clearly and significantly outperforms all of these. In terms of short-term (10-step ahead) forecasts, for which TS foundation models (unlike DynaMix) are optimized, DynaMix performs on par with the strongest competitors in our batch, shown in Fig. 4c. This is especially surprising when compared to Panda, which included variations of our test set DS in its training repertoire, biasing the evaluation in its favor. Table 9 assembles further results on performance comparisons, while Figs. 16-21 provide specific examples. Moreover, a significant limitation of particularly the large Chronos models with millions of parameters is their extreme inefficiency in both inference time and computational costs, compared in Fig. 5. In fact, their inference time can even exceed that of a custom-traines model. In contrast, DynaMix provides a highly efficient alternative, offering superior DSR performance while using orders of magnitude less parameters ( 10k in total) and computation time.
Geometrical Disagreement
Temperal Disagreement
10-step ahead Prediction Error
Oranced
Pinds
px5 uP
ChorastS-arg
Crees is
0.20 Try Tiee Muos
Cheese
Chrose ataaa
Farce
Oreio kutl boo
Choms
Oraceiobbarel
Ohanas
Orevo
Cycnse batan
-1.1
12.44
and
Sradhh
12:30
Man
Try
11.27
C.74
Rbos
Dane
MASE
Figure 4: Zero-shot DSR performance across all 54 test set DS for DynaMix and various TS founda- tion models for context length Tc 2000 (see Fig. 23 for results with Tc 512). MAA of stap (left, geometrical disagreement), DH (middle, temporal disagreement), and MASE (right, short-term prediction error). Statistical testing based on Wilcoxon signed-rank tests.
109
TimesFM
LS-large
Number of Parameters
104
Chroros
botbbbase
15 base
Pardaa
small
boit
Mamba4Cast
15-miri
10"
Tiny Mixers
10a
10
DynaMix
100
ALANN Trainsint
10'
Inference Time (s)
Figure 5: Number of model parameters vs. inference times for zero-shot generation of Lorenz-63 DS with 10000 forecasting steps. All models were provided the exact same context and run on the same hardware. Note the log-scale on the z/y-axis. For comparison, also the time needed for training and inference of custom-trained AL-RNN is shown (turquoise).
To put DynaMix' performance further in context, we also compared it to a couple of custom-trained DSR models, i.c. SOTA DSR models explicitly trained on the context data, incl. the plain AL-RNN

---

trained by STF [8], Neural ODEs [38, 18], and reservoir computers which are commonly used for DSR [72] (see Appx. A.7 for details). Despite the custom trained models' unduly advantage. performing unlike DynaMix only in-domain generalization [38], DynaMix keeps up with, and in some cases even outperforms, them (Appx. Tables 6-8).
4.2 Reasons for the failure of TS foundation models DSR
Why do TS foundation models perform so poorly on DSR problems? Essentially, they face three key issues: First, most TS foundation models cannot efficiently deal with multivariate time series, but treat dimensions independently. However, in a nonlinear DS all variables are usually coupled, and their joint evolution actually defining for the system's 's (long-term) dynamics and attractor states. Thus, ignoring this, as illustrated in Fig. 6a, results in poor long-term accuracy (see also Figs. 16-21). This highlights fundamental limitation: Univariate approaches are inherently insufficient for reconstructing multivariate, interacting DS. Second, TS foundation models are trained for short-term prediction, but not for DSR, providing another reason why they fail to reconstruct the correct long- term behavior. As illustrated in Figs. & 6b, in the longer run they often converge to fixed points or simple cycles where the true dynamics is chaotic, not reflecting the context anymore. Third, context parroting. phenomenon originally identified in LLMs [5], has also been observed in TS foundation models [102]. where they tend to repetitively reproduce the exact input pattern rather than dynamically adapting to the system's true evolution. Thus, TS foundation models simply produce oscillations, where the true dynamics chaotic and therefore by definition irregular and non-repeating (Fig. 6e). This is illustrated in more detail in Fig. 6d for Chronos-te 15 ---ass,, the best of the TS foundation models according to Datsp- sharp, narrow peaks wrongly suggest periodicity, while the true spectrum is usually rather broad and smeared out for chaotic systems, as the one exhibited by DynaMix. We further quantified this by calculating from the forecast time series the maximum Lyapunov exponent [60], measure for the
particularly evident in the power spectrum (Fig. 6d right), where Chronos'
Dimension Misalignment
Long-term Failure
Context Parroting
Mar Soot
w Mm M
m
d
Chaotic
ground truth Chronos
Power specarum
10
0
Awws
10
Ohromna
100
200
300
400
500
trequency
1.0
Power spectrum
1.0
Non- Chaotic
Aese
0.5
0
250
500
750
2000
1250
1500
1750
2000
0.0
Syndeaa Oheens
50
1000
150
200
Time
frequency
Figure 6: Common problems of TS foundation models (red: forecast, darkgray: context, lightgray: ground truth). a) Dimensions are dynamically decoupled. b) Long-term forecasts often converge to simple fixed points or cycles unrelated to the true dynamics. e) Context parroting covers up the true dynamics. d) Example forecasts for the chaotic Lorenz-61 63 system with DynaMix and Chronos-t5- base. Chronos' context parroting results in cyclic repetition the true aperiodic, chaotic dynamics. This is evident in the close-to-O max. Lyapunov exponent (center) and peaked power spectrum (right). e) For comparison, the underlying behavior truly cyclic with prominent peaks in the power spectrum, both Chronos and DynaMix are able to capture it.
fixed pattern, thus inherently missing
7

---

exponential divergence rate of trajectories (see Appx. A.I). For Chronos, the value AChronoe ~ 0.02 close to zero confirms the nearly periodic behavior, while DynaMix produces exponentially diverging trajectories with ADyneMir R 0.96 close to the ground truth value of A6rren663 R 0.87. Hence, unlike DynaMix, Chronos does not reconstruct the underlying dynamics but just repeats the context. On the other hand, if the underlying dynamics truly is cyclic, DynaMix settles into cyclic pattern as well, exhibiting no intrinsic bias, as illustrated in Fig. 6c.
4.3 Dynamical similarity
DynaMix' mixture-of-experts design produces natural similarity measure for comparing the dynam- ies of different systems, by evaluating the relative contribution of each of the experts to the forecast dynamics. As shown in Fig. 7a (bottom), different expert mixtures specialize in specific dynamical patterns or regimes. Hence, we can utilize the expert usage w* across the generated time series to construct a similarity metric S 1
I+W, [0, 1], based the squared 2- 2-Wasserstein distance W between the standardized expert weights w=p of any two systems.
b
5 - 0.86
0.60
Usage
0
40
80
300
116
Attractor
Figure 7: a) Top: Vector fields of different experts which in combination determine the attractor. Bottom plot illustrates the expert usage across time. b) Similarity matrix based on S among all 84 DS in the dataset, sorted via hierarchical clustering, Structure in the matrix indicates there are clusters of DS with high dynamical similarity. e) Examples of two pairs of attractors classified as similar (top). and one dissimilar pair (bottom).
4.4 Time series forecasting from a DS perspective
Table 1: Performance comparison empirical time series in terms of geometrical divergence (Daup). long-term temporal distance (DH), and forecast error (MAE). Best in red, second-best in blue.
System
DynaMix
Chronos-tS-
Mamba4Cast
TTM
TimesFM
Dasp
D
MAE
Day
Dy
MAE
Day
MAE
Desp
DH
MAE
Dasp
Du
MAE
Partially obs. DS
0.02. 0.25
0.39
0.02
0.32
0.39
8.35
0.35
0.76
4.76
0.31
0.90
7.30
0.36
0.71
E
0.22 0.08
4,92
0.23
0.09
3.22
0.80
0.10
3.63
6.36
0.09
4.64
0.24
0.10
3.04
Traffic
0.81 0.21
6.93
0.40
0.10
3.16
1.59
0.34
7.68
1.36
0.25
5.25
0.43
0.11
3.66
Cloud Requests
0.27 0.14
161.58
1.86
0.31
557.35
5,89
0.16
282.95
1.31
0.25
147.54
0.33
0.29
239.31
Weather (Temp.)
0.66 0.09
2.24
4.78
0.21
4.14
6.86
0.57
9.39
1.28
0.12
1.92
6.86
0.13
3.31
Weather (Press.)
0.39 0.19
9.03
6.68
0.23
9.05
7.07
0.52
8.60
7.31
0.26
10.03
9.59
0.60
8.41
Human fMRI
0.17 0.09
0.45
5.78
0.12
0.85
7.97
0.11
1.35
4.87
0.39
1.54
6.19
0.21
1.11
Human EEG
1.07
9.45
0.24
1.10
5.87
0.61
1.12
10.36
0.26
10.07
0.27
We next tested and compared DynaMix on a variety of real-world datasets often used to probe time series models. As discussed in Sect. 3.3, for DynaMix we used an embedding of the typically 1d empirical time series, to better reflect the underlying DS properties (positional embedding for all empirical time series, akin to what done in TS foundation models, and delay embedding for the Lorenz-63). Fig. 8a confirms for 1d time series from the 3d chaotic Lorenz-63 system that this works well and DynaMix, in contrast to all other models, is still able to reconstruct the chaotic behavior. When applied to real-wor world time series, such as traffic, weather, cloud request data,

---

Context
Ground Truth
DynaMix
Chronos
Mamba4Cast
TTM
TimesFM
Partially
ET
M
/w
VM
M
MA
ww
value
wy
m
WWW WOW WIM!
-103
Traffic
d
Cloud Requests
Volue
2000
-00
Time (hours)
Time
(Temp.)
Weather (Press.)
My WWW
-1000
-150
-008
-202
350
500
700
1800
800
Time (days)
(days)
Human IMRI
h
Human EEG
Velue
vgluc
(seconds)
(seconds)
Figure 8: Comparison of DynaMix (red) strongest competitor in terms of D Dstep (other colors) on zero-shot forecasts of various empirical time series (see Fig. 27-30 for comparisons to all other TS foundation models): Forecasts of a) partially (Id) observed Lorenz-63 DS, b) electricity transformer temperature, c) hourly car traffic data with weekly cycle, d) Huawei cloud requests, e) soil temperature development, f) air pressure, g) human functional magnetic resonance imaging (fMRI, h) human electroencephalogram (EEG).
where previous TS foundation models have shown inconsistent results [86]. our model produces accurate forecasts which reflect the dynamics well (Fig. 8b-f, see Appx. A.2 for further details the datasets). Our model seems to excel particularly on human physiological signals, such as functional magnetic resonance imaging (fMRI) or electmeencephalographic (EEG) recordings, where all other models essentially produce meaningless dynamics, at best capturing the mean trend (Fig. 8g.h).
Table quantitatively confirms that DynaMix mostly outperforms the other models tested on the DSR measures. Surprisingly, often even outperforms them terms of the forecasting error MAE, although its training corpus consisted purely of the simulated DS shown in Fig. 10, i.e. did not include any empirical data at all, let alone traffic, cloud usage, electricity, or temperature data! This is in stark contrast to the other TS foundation models tested here, which mostly had examples of such empirical data in their training corpus and hence might be expected to have an edge over DynaMix. Many real- world data, like weather, human iMRL, EEG data, are known to bear signatures of deterministic chaos [60, 51], potentially at least partly explaining DynaMix' strongly competitive forecasts, although it has never seen such type of data in training.
4.5
Which ingredients are most crucial for DynaMix' performance? Ablation studies
Which are the components of the DynaMix framework contributing most to its success in zero-shot out-of-domain generalization? Training by STF (Fig. 26), a sufficiently diverse training corpus, sufficient number of experts (Fig. 25), the attention mechanism, and context preprocessing by the CNN, all appear vital its strong performance (Fig. 9). Replacing the final gating step (the MLP) by simple linear operation and the exact expert model within the class of piecewise linear RNNs, however, had less of an impact. Similar results were obtained when the AL-RNN was swapped for the clipped shallow PLRNN [40], and probabilistic version of the AL-RNN (adding a noise term 2, = -2) + 7. 17 ~ (O, D)) even slightly improved long-term performance, albeit at the cost of some short-term prediction accuracy. Using as experts LSTMs [42], vanilla RNNs or reservoir computers [67] instead led to decay in performance, in line with previous results establishing PLRNNs as SOTA DSR models which allow for efficient training by STF [7, 40, 8]. Hence, it appears that the

---

whole 'DSR package' consisting of 1) architectural choices, 2) training algorithm specialized for DSR, and 3) properties of the DS training corpus, is crucial to DynaMix success.
a
Full model
MLP
CHN
no Attention
CNN Lineer
too few experts
no STF
restricted training set
0.3
0.0
b
probabilishte
clpped chellow
vanill
Reservoir
1.5
1.0
0.5
Figure 9: Ablation studies. a) Effect of removing or altering components of the gating network (no MLP, no CNN, no Attention, CNN to linear layer), including too few experts (J < 5), replacing STF by standard BPTT training, restricting the training corpus Lorenz-type systems only. b) Effect of replacing the AL-RNN by other RNN models. Error bars STD
5 Conclusions
Here we introduce the first DSR foundation model that achieves zero-shot reconstructions of novel DS, accurately reproducing their long-term statistics, just from provided context signal without any retraining or fine-tuning. It hugely outperforms current TS foundation models like Chronos or Panda in terms of DSR quality, although much more lightweight with only fraction (0. 1%) of the number of parameters. Consequently, DynaMix features inference times about an order of magnitude faster than the closest competitor. Most surprisingly, it even outperforms current TS foundation models on empirical data of types it has in contrast to competitors never seen in training, often producing also better short-term forecasts. All of this is achieved with fairly narrow training corpus consisting mainly of 3d synthetic data from chaotic systems, which suggests there may be considerable room for further improvement by extending DynaMix' training to include, e.g., various empirical time series.
One important take-home from this work therefore is that foundation models built based on principles of DS theory may be able to profoundly improve performance of current zero-shot TS forecasters. Most (if not all) empirically observed time series come from some underlying DS, and acknowledging this fact in model training and construction may help to advance the field. Besides DynaMix specific motivated training techniques, and embedding principles, also the fact that it has seen many chaotic systems in training may play role. Most complex DS in nature [16, 88, 61], engineering [80], and society [58, 94] are likely chaotic. Moreover, many chaotic attractors, like the Lorenz-63, feature skeleton of infinitely many (unstable) periodic orbits of all possible periods [37], thus inherently expressing a wide spectrum of temporal patterns.
DSR architecture,
Limitations For now, we have mainly focused rather stationary time series (but see Fig. 8b). Changes in statistical properties over time, tipping points, or widely differing time scales in the data, impose severe difficulties for zero-shot forecasting, illustrated in Fig. 3la.,, which also plague other TS foundation models. This may partly be amended by explicitly including such types of non- stationary and multiscale DS in the training corpus. Another potential solution may be adding explicit filtering and decomposition modules, similar as in Auto- [98] or FEDformers [104]. A proof of concept of this idea is provided in Appx. A.5. Incorrect embeddings. which may not be immediately obvious, can also lead to forecast failures (Fig. 31c). Finally, continuous-tim versions of the experts which can deal with sampling at irregular time points would be another fruitful direction.
Code available at https: h/tgt//ggtttttttt wttp://wwwww..ttwt///eea wawaaaaaaaaaaaaiaaaaaaaaaaa julia (Julia), https:// gi github. com/Duurttenittaaa//yynooe
(Python).
10

---

Acknowledgements
This work was funded by the German Research Foundation (DFG) within Germany's Excellence Strategy EXC 2181/1 -390900948 (STRUCTURES), and through DFG individual grant Du 354/15-1 to DD.
References
[1] Victor M. Martinez Alvarez, Rareş Rosca, and Cristian G. Fälcuțescu. DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control, September 2020.
[2] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang. Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Syndar Rangapuram, Sehastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael
and Yuyang Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: http://ppnneeiieeeeee www.eeeeewwwwweeeeeeeeeeeeeeeeee tete/fforumiiiee id=gerNCVqqtR.
[3] Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, et al. Chronos-2: From univariate universal forecasting. arXiv preprini arXiv:25 10. 15821, 2025.
[4] Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael W. Mahoney. Forecasting Sequential Data using Consistent Koopman Autocncoders. In Proceedings of the 37th Interna- tional Conference on Machine Learning, 2020. URL http: / arxiv. org/ abs//003.. 022360
[5] Emily M Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM conference on fairness, accountability, and transparency, pages 610-623, 2021.
[6] Sathya Kamesh Bhethanabhotla, Omar Swelam, Julien Siems, David Salinas, and Frank Hutter. Mamba4cast: Efficient zero-shot time series forecasting with state space models. arXiv preprint arXiv:2410.09385, 2024.
[7] Manuel Brenner, Florian Hess, Jonas M. Mikhaeil, Leonard Bereska, Zahra Monfared, Po-Chen Kuo, and Daniel Durstewitz. Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems. In Proceedings of the 39th International Conference on Machine Learning, pages 2292-2322 PMLR, June 2022. URL https: brenner22a. html. ISSN: 2640-3498.
mlr press/v162.
[8] Manuel Brenner, Christoph Jürgen Hemmer, Zahra Monfared, and Daniel Durstewitz. Almost- linear rmns yield highly interpretable symbolic codes in dynamical systems reconstruction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 36829-36868. Curran Associates, Inc., 2024. URL https: http::://///cceeeeeeeeeeeee neurips cc cc/paper files/paper -ateeoo77-----prr--e-o Conf conferrnce... .df.
/024/file/
[9] Manuel Brenner, Florian Hess, Georgia Koppe, and Daniel Durstewitz. Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics. In Proceedings of the 41st International Conference on Machine Learning, pages 4482-4516. PMLR, July 2024. URL
attps://prat https://procceedings mlr, press http//222////// 2033/bbennee22a. html. ISSN: 2640-3498.
[10] Manuel Brenner, Elias Weber, Georgia Koppe, and Daniel Durstewitz. Learning interpretable hierarchical dynamical systems models from time series data, In The Thirteenth International Conference Learning Representations (ICLR), 2025. URL https: /ttp://v/////eeee/////ee net/
UpDAxxxss..
[1]] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, et al. Language models are sho-shot learners. Advances in neural information processing systems, 33:1877-
1901, 2020.
[12] Steven I. Brunton and J Nathan Kutz. Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press, 2019.
11

---

[13] Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences USA, 113(15):3932-3997) 2016. ISSN 0027-8424. doi: 10. 10.1073/pnas. 1517384113.
[14] Steven I.. Brunton, Bingni W. Brunton, Joshua I., Proctor, Eurika Kaiser, and J. Nathan Kutz. Chaos as an intermittently forced linear system. Nature Communications, 8(1):19, May 2017. ISSN 2041-1723. doi: 10. 100-0000//1444--7-000000 0er-leee/s1147-017-000000
URL httph:ttt//// h/tp:::://www../// http:://wwwwwwweweeee//eeeeeeeee wwwure...e...... com/ Publisher: Nature Publishing Group.
[15] Steven L. Brunton, Marko BudiKié, Eurika Kaiser, and J. Nathan Kutz. Modern Koopman Theory for Dynamical Systems, October 2021. arXiv:2102. 12086 [es, eess, math].
[16] Gyorgy Buzsaki. Rhythms of the Brain. Oxford University Press, August 2006. ISBN 978-0-19-804125-2 Google-Booe Boale-Boooks-ID. Idz58irprjYC.
[17] Rok Cestnik and Markus Abel. Inferring the dynamics of oscillatory systems using recurrent neural networks. Chaos: An Interdiseiplinary Journal of Nonlinear Science, 29(6):063128, June 2019. ISSN 1054-1500. doi: 00006..... URL https://doi org/ 1063/ 5.5096918.
[18] Ricky Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems 31, 2018. URL
http://arxiv. org//o/g/go abs/ 11///8806.000 1806.07366.
[19] Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta-in-context learning in large language models. Advances in Neural Information Processing
Systems, 36:651 0086651896666 692011 2023.
[20] Alexandre Cortiella, Kwang-Chin Park, and Alireza Doostan. Sparse identification of non- linear dynamical systems via reweighted 11 I1-regularized least squares. Computer Methods Applied Mechanics and Engineering, 376:11366 13620, April 2021. ISSN 0045-7825. doi:
10.10166j.cma.222
13620..
[21] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
[22] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprini urXiv:2301.0023: 2022.
[23] Daniel Durstewitz. A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamies from neural measurements. PL.S Comput. Biol., 13(6): el e1005542, 2017. ISSN 1553-7358.
doi: 10. l.o.ma//alllllpbiiiiio 1005542.
[24] Daniel Durstewitz, Georgia Koppe, and Max Ingo Thurm. Reconstructing computational sys- tem dynamics from neural data with recurrent neural networks. Nature Reviews. Neuroscience, 24(1 24(11):693-710, November 2023. ISSN 1471-0048. doi: 11003//415333022-0077444
[25] Lukas Eisenmann, Zahra Monfared, Niclas Göring, and Daniel Durstewitz. Bifurcations and loss jumps in RNN training. Advances in Neural Information Processing Systems, 36, 2024.
[26] Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In Proceedings the 29th ACM SIGKDD conference on knowledge discovery and data mining, pages 459-469, 2023.
[27] Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam Nguyen, Wes- ley Gifford, Chandra Reddy, and Jayant Kalagnanam. Tiny time mixers (tms)): Fast pre-trained models for enhanced zero/few-shot forecasting of multivariate time series. Ad- vances in Neural Information Processing Systems, 37:74147-74181, 2024.
[28] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 555330833305998 2022.
[29] Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. Neural Networks, 146:272-289, February 2022, ISSN 0893-6080. doi:
12

---

[30] William Gilpin. Chaos as an interpretable benchmark for forecasting and fata-driven modelling. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2022. URL /ttetps://pperreeee
net/ worroriddeenrrli
[31] William Gilpin. Model scale versus domain knowledge in statistical forecasting of chaotic systems. Physical Review Research, 5(4):043252, December 2023. doi: 10. 103/ Phy.RewReescarch55.003333
Publisher: American Physical Society.
[32] William Gilpin. Generative learning for nonlinear dynamics. Nature Reviews Physics, 6(3):
194-206, March 2024. ISSN 2522-5822 5820. doi: 10.1038/s422555 0058042225400440006688 URL https: ///
www. www.nnauur.oooo httpelestcccceeeoooo cacccceees422/ 2//////422244220008808 2. Publisher: Nature Publishing Group.
[33] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105.0666 100.06643, 2021.
[34] Niclas Alexander Göring, Florian Hess, Manuel Brenner, Zahra Monfared, and Daniel Durste- witz. Out-of-domain generalization in dynamical systems reconstruction. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 16071-161 14. PMLR, 21-27 Jul 2024. URL htRp:///:
bittps::/prooo http//poceednggge mlr wwwppreeevvvee 22335/orrng24a html.
[35] Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36: 19622-19635, 2023.
[36] Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, December 2023. trXiv:2332.00073 [es].
[37] John Guckenheimer and Philip Holmes. Nonlinear Oscillations, Dynamical Systems, and Bi- furcations of Vector Fields, volume 42 of Applied Mathematical Sciences. Springer, New York, doi: 10.1007/978-1-4612 1611--141---2 140-2. www.10.../........ 1007/978--711 1-18--1--612-1---0-- URL http::/ttiiiee http:/iiinn..siiinn...eeeee
NY, 1983. ISBN 978-1-4612-7020- 978-1-4612 11--4612-110--- 2-022
[38] Niclas Alexander Göring, Florian Hess, Manuel Brenner, Zahra Monfared, and Daniel Durste-
witz. Out-of-Domain Generalization in Dynamical Systems Reconstruction. In Proceedings of the st International Conference on Machine Learning, pages 1607 16071-16114, PMLR, July
2024. URL htttp:://proceeding mlr. www.tpressvvvee v235/ 3498.
http///2///////// gor.iggiinnnaa.. html. ISSN: 2640-
[39] Christoph Jürgen Hemmer, Manuel Brenner, Florian Hess, and Daniel Durstewitz. Optimal
Recurrent Network Topologies for Dynamical Systems Reconstruction. In Proceedings of the s1 International Conference on Machine Learning, pages 18174-18204, PMLR, July 2024. URL https: ptttps:/pproceedingg
mlr. press ht/p//225//hhh// hww.ereemeeeeeeeeeeeeeeeeeeeeeee 24a. html. ISSN: 2640-3498.
[40] Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized Teacher Forcing for Learning Chaotic Dynamics. In Proceedings of the 40th International Conference
on Machine Learning, pages 13017-13049. PMLR, July 2023. UtLhhtts://proeeeeee mlr. www.ppressvveee http://222hhheeseee html. ISSN: 2640-3498.
[41] Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery, (7(2):788-833 2023.
[42] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9()) 1735-1780, nov 1997. ISSN 0899-7667. doi: 10. 1662necco.. 1997.9.8.1735. 1735.
[43] Ruoxi Jiang, Peter Y La, Elena Orlova, and Rebecca Willett. Training neural operators to preserve invariant measures of chaotic attractors. Advances in Neural Information Processing Systems, 36:27645-27666 2023.
[44] E. Kaiser, J. N. Kutz, and S. L. Brunton. Sparse identification of nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 474(2219):201 80335, November 2018. doi: 10. 10.1098/sspa. 0..33.335.00 Publisher: Royal Society.
[45] Holger Kantz and Thomas Schreiber. Nonlinear time series analysis, volume Cambridge university press, 2004.
13

---

[46] Daniel Karlsson and Olle Svanström. Modelling Dynamical Systems Using Neural Ordinary Differential Equations, 2019. URL https: //hdl. handle net/20. 500 1238002800000 22380/256880 2568877.
[47] Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy. and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In International Conference on Machine Learning, pages 11283-13011 PMLL...... 2022.
[48] Joon-Hyuk Ko, Hankyul Koh, Nojun Park, and Wonho Jhe. Homotopy- training of NeuralODEs for accurate dynamics discovery, May 2023. URL http: pestp://arxiee org/ abs/ 2210. 01407. arXiv:2210.00 111.:222000014077 [physics].
[49] Georgia Koppe. Hazem Toutounji, Peter Kirsch, Stefamie Lis, and Daniel Durstewitz. Identify- ing nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI. PL.OS Computational Biology, 15())ee0072663, 2019. ISSN 1553-7358. doi:
10. 10.137//jurmal.pehh wtt.:mma.ppci.100263...
[50] K. H. Kraemer, G. Datseris, J. Kurths, I. Z. Kiss, J. L. Ocampo-Espindee and N. Marwan. A unified and automated approach to attractor reconstruction. New Journal of Physics, 23(3):033017, March 2021. ISSN 1367-2630. doi: Publisher: IOP Publishing.
URL
https: http:://dooooogggp org/10..0... 1088/ 11111336722268 2/630/abe336
[51] Daniel Kramer, Philine L Bommer, Carlo Tombolini, Georgia Koppe, and Daniel Durstewitz. Reconstructing nonlinear dynamical systems from multi-moda time series. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1 11613-11661 1163. PMLR, 17-23 Jul 2022. URL https: //proceedings. mlr. press wtt2/k1aa2ee2222 hhmml.
[52] Jeffrey Lai, Anthony Bao, and William Gilpin. Panda: A pretrained forecast model for universal representation of chaotic dynamics. arXiv preprint 1.Xi::2500.13755 2025,
[53] Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. In Forty-first International Conference on Machine Learning, 2024.
[54] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2020. URL https: // //tp///vwp//eeee//////ee/ee net/forum? id=rkgz2aEKDr.
[55] Jean-Christophe Loiseau and Steven L. Brunton. Constrained sparse Galerkin regression. Journal of Fluid Mechanies, 838:42-60 67, March 2018. ISSN 0022-1120, 1469-7645. doi: 10. 10.10077/jmm.200 http:ffm..017.882.2 2017.823, Publisher: Cambridge University Press.
[56] Edward N Lorenz. Predictability: problem partly solved. In Proc. Seminar on predictability,
volume 1, 1996.
[57] Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nat Commun, 9())49550, December 2018. ISSN 2041- 1723. doi: 1001038/441467-11--777 arXiv: 1712.09707.
URL
org/ 19777772999777
[58] Benoit Mandelbrot and Richard L. Hudson. The Misbehavior of Markets: A Fractal View of Financial Turbulence. Basic Books, March 2007. ISBN 978-0-465-00466-3. Google-Books.
ID: GMKeUquiPQOC.
[59] Daniel A. Messenger and David M. Bortz. Weak SINDy: Galerkin-Based Data-Driven Model Selection. Multiscale Modeling & Simulation, 19(3):1474-1977, January 2021. ISSN 1540-3459. doi: 10.1137/20M134311
URL https / htapp::///puuaee /epubs..... siam. www..ooo///ee...eeeee..e..o prrg/doi/100 1137/ 20M1343166. Publisher: Society for Industrial and Applied Mathematics.
[60] Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with RNNs. Advances in Neural Information Processing Systems, 35:1 1297-1122311 December 2022. URL
httpp/ppppr/22222 nwwrips ct/ httpp/apprrr_ff waler_files///eeeeeppppppeeeeeee httperrffllleepppe///eee
html.
[61] Peter J. Mumby, Alan Hastings, and Helen J. Edwards. Thresholds and the resilience of caribbean coral reefs. Nature, 450(7166):98-011, 2007.
doi: 10. 10.10388/ URI.
https://doi. org/ wtt.a/oro/.....//nnttetttoo wtt.133//ature0625555
14

---

[62] Ilan Naiman and Omri Azencot. A Koopman Approach to Understanding Sequence Neural Models. arXiv:2102. 07824 [cs, math], October 2021. URL http://arxivv.orggg//////////// ottp.///..aa/./......//// 07824. arXiv: 2102.07824
[63] Ali H Nayfeh and Dean Mook. Nonlinear oscillations. John Wiley & Sons, 2024.
[64] Roussel Desmond Nzoyem, David AW Barton, and Tom Deakin. Towards foundational models for dynamical system reconstruction: Hierarchical meta-learning via mixture of experts. arXiv preprint arXiv:2502.085
2025.
[65] Samuel E. Otto and Clarence W. Rowley. Linearl--Recurennt Autoencoder Networks for Learning Dynamics, January 2019. URL arXiv:1712.01371 [es, math, stat].
[66] Matthijs Pals, A Erdem Sağtekin, Felix Pei. Manuel Gloeckler, and Jakob H Macke. In- ferring stochastic low-rank recurrent neural networks from neural data. arXiv preprint urXiv:2406. 16749, 2024.
[67] Dhruvit Patel and Edward Ott. Using Machine Learning to Anticipate Tipping Points and Extrapolate Post-Tipping Dynamics of Non-Stationary Dynamical Systems, July 2022. URL
http://arxiv. org/ abs/ /2007 00521. arXiv:2207.000 .002222 [physics].
[68] Dhruvit Patel and Edward Out. Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos (Woodbury, N.Y.). 33(2):023133, February 2023. ISSN 1089-7682. doi: 10.1063/5.013178)
[69] Jaideep Pathak, Zhixin Lu, Brian Hunt, Michelle Girvan, and Edward Ott. Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(12):12110, December 2017. ISSN
1500- 1089-7682. doi: 00.103//.5010300 URL http: htap::///aacxe /ttp///avvvv/./////////////..av/ org/ abs///b////// /1100. 07313. arXiv: 1710.07313.
[70] Jason A Platt, Adrian Wong, Randall Clark, Stephen G Penny, and Henry DI Abarbanel. Robust forecasting using predictive generalized synchronization in reservoir computing. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31()), 2021.
[71] Jason A. Platt, Stephen G. Penny, Timothy A. Smith, Tse-Chun Chen, and Henry D. I. Abarbanel. A Systematic Exploration of Reservoir Computing for Forecasting Complex Spatiotemporal Dynamics, January 2022. URL arXiv:2201 108910 [cs].
org/ abs/ 2201 08910.
[72] Jason A Platt, Stephen G Penny, Timothy A Smith, Tse-Chun Chen, and Henry DI Abarbanel. Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers.
Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(10), 2023.
[73] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Hassen, Anderson Schneider, et al. Lag-Ilama: Towards foundation models for time series forecasting. In RO-FoMo: Robustness of Few-shot and shro-shot Learning in Large Foundation Models, 2023.
[74] Michael T Rosenstein, James J Collins, and Carlo J De Luca. A practical method for calculating largest łyаpunov exponents from small data sets. Physica D: Nonlinear Phenomena, 65(1-2): 117-134, 1993.
[75] T KKnsstantin Rusch, Siddhartha Mishra, N Benjamin Erichson, and Michael W Mahoney. Long expressive memory for sequence modeling. In International Conference on Learning Representattinss 2022.
[76] Tim Sauer, James A Yorke, and Martin Casdagli. Embedology. Journal of statistical Physics, 65(3):579 616, 1991.
[77] Gerwin Schalk, Dennis J. McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R. Wolpaw. BCI2000: a general-purpose purpose brain-computer interface (BCI) system. IEEE transactions on bio-medical engineering, 1136)10334 2000. ISSN 0018-9294. doi: 10. 10.1100 1000/TTBEE.20044227700000
[78] Yair Schiff, Zhong Wan, Jeffrey B. Parker, Stephan Hoyer, Volodymyr Kuleshov, Fei Sha, and Leonardo Zepeda-Núñez. DySLIM: Dynamics stable learning by invariant measure for chaotic systems. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International
15

---

Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 43649 43684, PMLR. 21-27 Jul 2024.
[79] Evgeni E SEL'KOV. Self-oscillations in glycolysis 1. a simple kinetic model. European Journal of Biochemistry, 4(1):79-86, 1968.
[80] Steven H Strogatz. Nonlinear dynamics and chaos: with applications physics, biology, chemistry, and engineering. Chapman and Hall/CRC, 2024.
[81] Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. Test: Text prototype aligned embedding to activate I1m's 's ability for time series, arXiv preprint arXiv:2308.0827 2023.
[82] Floris Takens. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980, volume 898, pages 366-38) (866-88.355 Springer, 1981. ISBN 778-3-540-111711 978-3-
540-38945-3. 3. URL http: http::///innns..ingeerr.. com/ 10. 1 /BB0099994.
[83] Sachin S. Talathi and Aniket Vartak. Improving performance of recurrent neural network with relu nonlinearity. In Proceedings of the 4th International Conference on Learning Representationss 2016. URL http: //rxiv... . org/ 0too//151..0777
[84] Mingtian Tan, Mike Merrill, Vinayak Gupta, Tim Althoff, and Tom Hartvigsen. Are language models actually useful for time series forecasting? Advances in Neural Information Processing
Systems, 37:60162 00111226661-2-10 2024.
[85] Hua Tang, Chong Zhang, Mingyu Jin, Qinkai Yu, Zhenting Wang. Xiaobo Jin, Yongfeng Zhang, and Mengnan Du. Time series forecasting with Ilms: Understanding and enhancing model capabilities. ACM SIGKDD Explorations Newsletter, 26(2):109-118; 2025.
[86] William Toner, Thomas I. Lee, Artjom Joosen, Rajkarn Singh, and Martin Asenov. Per- formance of zero-shot time series foundation models cloud data. arXiv preprint arXiv:2502. 12944, 2025.
[87] Adam P. Trischler and Gabriele M.T. D'Eleuterio. Synthesis of recurrent neural networks for dynamical system simulation. Neural Networks, 80:67-78, 2016. ISSN 08936080. doi: 10. h0016/...emee.2016.000 pi 251/80993308016300031.
URL
[88] Eli Tziperman, Harvey Scher, Stephen E. Zebiak, and Mark A. Cane. Controlling spatiotem- poral chaos in a realistic el niño prediction model. Phys. Rev. Lett., 19:1034-1037, Aug 1997. doi: 10. 10w/PhysRewLLett.9.. PhysRevLett. 79. 1034.
URL https://tstt aps wttp:/k...pp...oooooo/ooooo 1103/
[89] Balth Van der Pol. Lxxxviii on Philosophical Magazine and Journal of Science, 2(11):978-8)) 992, 1926.
The London, Edinburgh, and Dublin
[90] Pantelis R. Vlachas, Wonmin Byeon, Zhong Y. Wan, Themistoklis P. Sapsis, and Petros Koumoutsakos. Data-driven forecasting of high-dimensional chaotic systems with long short- term memory networks. Proc. R. Soc. A., 474422133:201008444 2018. ISSN 1364-5021, 1471-2946. doi: 10. 10.1098/rspa.2011 2017.0844 URL http:://rroyalscccccpb....a doi/10. 100099//rpaaa wwtpp..007..... 2017 0844.
org/
[91] Eric Volkmann, Alena Brändle, Daniel Durstewitz, and Georgia Koppe. A scalable generative model for dynamical system reconstruction from neuroimaging data. Advances in Neural Information Processing Systems, 37:80328-80362, 2024.
[92] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pages 35151-35174. PMLR, 2023.
[93] Rui Wang. Yihe Dong, Sercan Ô Arik, and Rose Yu. Koopman Neural Forecaster for Time Series with Temporal Distribution Shifts, October 2022. URL http://arxiv.orgg abs/ 2210. 03675. arXiv:2210.0367 [cs, stat].
[94] Duncan J. Watts and Steven H. Strogatz. Collective dynamics of 'small-world" networks. Nature, 393(6684):440-442, 1998. doi: 10.1038/30918.
[95] Andrew Robert Williams, Arjun Ashok, Éitienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Cha- pados, et al. Context is key: benchmark for forecasting with essential textual information. arXiv preprint arXiv:
2024.
16

---

[96] Alan Wolf, Jack B. Swift, Harry L. Swinney, and John A. Vastano. Determining Iyapunov exponents from time series. Physica D: Nonlinear Phenomena, 16(3):285-317 317, 1985. ISSN 0167-2789. doi: 10. 10.10066006672227888339900011110
URL wwt.://www..ceeceiree....
[97] Simon N. Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310))1102-1104 August 2010. ISSN 1476-4687. doi: 10. 10.1038/naturc0919, URL
attps://wwer http://///w...aureee....m -wwurr...eeee.. com/ artuueeeeee// Number: 7310 Publisher: Nature
Publishing Group.
[98] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 01122419222430 2021.
[99] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. arXiv preprint arXiv:2111.0208) 2021.
[100] Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Advances Neural Information Processing Systems, 34.7561-7573, 2021.
[101] Rose Yu and Rui Wang. Learning dynamical systems from data: An introduction to physics- guided deep learning. Proceedings of the National Academy of Sciences, 121(27):e23118088 181881221118811 2024.
[102] Yuanzhao Zhang and William Gilpin. Zero-shot forecasting of chaotic systems. In The Thirteenth International Conference Learning Representations, 2025. URL https://
openreview.o net/ teae/fforum/ee
[103] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106-11115. AAAI Press, 2021.
[104] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pages 27268-27286 27286. PMLR, 2022.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, in the abstract and introduction only refer to results reported and supported in later sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We added Limitations pg. to the Conclusions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
There are no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The model architecture, along with the training and evaluation procedures, is described in the main text, with further details and hyper-parameter settings Appx.
A-1-A.3. We also included our codebase, ensuring the results can be replicated.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: A link to a repository is provided that contains the implementation of the architecture and its training method. All empirical data used are open-source with links provided in section A.2, where all simulation data are described as well.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
guudes//ooeeSumiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss... wwwppcccc/ppaaccccccccccccccceee gwwcc//guess..ee CCaeae/dddSSSbbiaea onnooiicy)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All settings provided in sections A.I and A.2, well in the accompany- ing codebase.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are included in all figures reporting numerical results, as well as statistical significance tests where needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on computational resources provided in sections A. and A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 'tttt make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https h/te://///eeeee /nerris.....p// ec/ cc/public Ceteiii////iieeeeeeeeeeeeeeeee csQuidel ines?
Answer: [Yes]
Justification: All datasets considered here are publicly available. This papers covers basic research to advance models for dynamical systems reconstruction, mainly of relevance in scientific and medical applications, which we believe carries primarily positive ethical implications. While we have not identified any specific ethical concerns, the broad range of potential applications also the time series domain means that the possibility of misuse cannot be entirely ruled out.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
21

---

Justification: This work constitutes basic, not applied, research. Hence, we do not anticipate any immediate negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We do not anticipate significant risk misuse beyond the potential of misuse of time series models more generally.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All novel code, models and results were produced by us. All other models tested against are publicly available, and the authors were properly credited. Likewise, all datasets used here are open-source with links and citations provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
22

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thooee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pan re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
fully documented version of the code is available.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects. While models were run data from human subjects, these are all from previously published experiments and openly available.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
23

---

Justification:
The paper does not involve research with human subjects, see above.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The methods developed in this paper did not involve the usage of LLMs. LLMs were solely used as support in editing and coding.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //tpur/essp.//eeeeee / for what should or should be described.
24

---

A Appendix
A.1 Methodological details
Training method For training our model, we used a variant of sparse teacher forcing (STF), a control-theoretic method for DSR, with the theoretical background and rationale developed in [60]. STF and related training methods [7, 9, 40, 60, 25] circumvent the exploding gradient problem when training on chaotic systems, while still enabling the DSR model to sufficiently forward-iterate trajectories into the future to capture long-term properties (see [60] for details), and produces state- of-the-art results for DSR [8, 10]. In STF, current latent states states 2, are replaced with states inferred from the data at fixed intervals by thus recalibrating the trajectory at times ideally chosen based on the system's 's Lyapunov spectrum [60] (or simply treating T as a hyper-parameter)) In our case of an identity mapping from latent subspace to the observations, ã; = I[Nx this inversion becomes trivial, where N is the number of observations (and thus readout neurons) and [ M)) N X M matrix with I[N xM), 1 for r S N and zeros else. Thus, during training, we force
of the decoder model, Σ, = g
Fo C) (Z C)
if TtET= = {lT + ThEN.
Ze+1 =
otherwise.
where Σ = (I,, NN 2N+1: Fo is the mixture of experts using the context C, and 7 = 10 here (see Fig. 26). Importantly, STF is only used for training the model, applied after calculating the loss, and is turned off at test time.
For the loss, we simply use the standard mean squared error (MSE) between model predictions X and ground truth observations X,
T
1 LMSE (X x) N. -T + Af)
Σ
[ - xel12
(8)
t= To A++1
To this we add regularization term meant to encourage the model to explore wider region of state space based on the context, by enhancing the variance Σ of exploration noise in the state attention:
Greg X
Σ exp (- /c)
where we chose x = 0.1 and c = 0.01. Rectified adaptive moment estimation (RADAM) [54] was employed as the optimizer, with = 50 batches of SB = 16 sequences per epoch, each of length T = 550, and 2000 epochs in total. We used a learning rate exponentially decaying from 'sstart = 5 10-3 to nend = 10 The context length used in training was set to Tc = 500, and the window of overlap with the model-generatee
time series to At 50, see Sect. 3.2. Training was performed on a single CPU (18-Core Xeon Gold 6254), and a single epoch took 30 seconds depending on sequence length and model size. At test time, no retraining fine-tuning is performed, but utilizing the context C the model just nerward-iteratee training were partly selected according previous results in [8. 7]. and partly few different parameter settings were tested, namely À = (0.0, 0.1, 0.2), At = {0, 803. and Te = (250, 500). extensive grid search was not necessary to obtain well perl-peeforming model.
from the last context time step. Hyperparameters for
Model details We use J 10 AL-RNN experts for our model. Each expert has a latent dimension of M 30, of which P = 2 are fiiffedd-linear units (ReLUs). As suggested in [8], only use the first N of the M - P linear units for the readout. We followed the initialization protocol in [7, 83], drawing W from a Gaussian with mean 0 and o 0.01 fI, setting h O, and A to the diagonal of normalized positive-definite random matrix. The initial latent state was estimated as
(10)
Lx
where L (MM-N) is jointly learned with other model parameters.
The gating network implemented using a single-layer CNN with three channels and a kernel size of 2, stride of 1, and zero padding, with the identity activation function. The MLP, which takes
25

---

as inputs the concatenated weighted CNN outputs and current latent state, consists of two layers with ReLU activation. We initialize the temperature weights Tau and Texp to 0.1, the covariance matrix Σ = 0.05 II, matrix D [(NxM): and draw all other CNN and MLP matrices from Gaussian with standard deviation 0.01. Model hyperparameters were selected according to previous results in [8] and by probing a few relevant parameter settings, namely J = {1.3,5, 5, 10, 20, 30) (Fig. 25), # #CNN-layers = {1, 2}. CNN-kereelsiie = {2, 3, 5). #MLP-layers = {1,2,3), and different activation functions {1, tanh, ReLUJ for the MLP/CNN. We found that an extensive grid search was not necessary to obtain a well-performing model.
Performance measures As in [49, 60, 40, 9, 66, 31, 102]. we assess the geometric (dis)similarity between true and model-generated attractors using a Kullback-Liible divergence (Dasp) defined across state space. Specifically, Dusp determines the overlap between the distributions of true trajectory points, (x), and of model-generated trajectories, Pgen (ar [2) given by
Datsp (Ptrue (x) ||pnen (z]))) - (true(x)) log Pirue (x)
(z]2z) dx.
(11)
Pgen
While especially for higher-dimensional state spaces these distributions may be approximated by Gaussian mixture models [49], here a discretized version was sufficient, with the state space parcel- lated into K m bins, where m is the number of bins per dimension and N the dimensionality of the system. We estimate the occupation probabilities pı of each bin via the relative frequencies ý; of trajectory visits and approximate sstsp as
K
Peruess Pgenz:
D sup
Ptruez
log
(12)
We set m = 30 bins per dimension, following [39]. For the 6d Lorenz-96 we use = 5 and for the empirical time series m = 20. To ensure a steady-state distribution is reached, long trajectories of T = 10, 000 time steps are sampled from the DS.
To evaluate the agreement in long-term temporal dynamics, we compute the Hellinger distance Dh between the power spectra of the true and model-generated time series [60, 40], defined as
H(F(w),
((w)) [0.
(13)
where F(w) and G(ww) are the power spectra of the true and generated time series, respectively. Power spectra are obtained via the dimension-wise Fast Fourier Transform (FFT), smoothed using a Gaussian kernel (with a = 20 for DS and a 2 for the short empirical time series), and normalized to ensure comparability. High-frequency tails dominated by noise are truncated as described in [40]. The aggregated Hellinger distance D H is then computed as the average across all dimension-wise spectral comparisons.
Note that both Dasp and Dir are assumed to assess Jairemaage in long-term properties in the limit T cc. Hence, they are only sensible on rather long-term horizons, with T 10, 000 used here.
For assessing short-term forecast quality aggregated across the whole test set of 54 DS, used normalized n-step ahead prediction error (with n = 10 in our evaluation) as recommended in [41], given by
N 1 Eletet
1
MASE
Σ n
(14)
N
1=1
To+T L1=Tc+1
[24, - Fi.--ll
T
For individual comparisons on single empirical times, the MAE for evaluating short term predictions [41] was used:
1 Totn
MAE
Σ 1r-0
(15)
71
t=Te+1
where we chose n according to the data's 's temporal scale and resolution (ETTThI,fMR)) = 40, traffic cloud requests, partially obs. DS}: a = 80, (human FEG, weather air pressure]:
120, wwather temperature): 200).
26

---

Lyapunov exponent For numerically estimating the maximum Lyapunov exponent Amax from one-dimensional empirical time series {ær), 1. T, one common upproach is the Rosenstein algorithm [74]. First, the time series is embedded into a state space using time-delay embedding as eq. 5 (e.g. using the PECUZAL algorithm [50]). For each embedded point Ti the nearest 4,. with 4 a threshold to remove purely temporal neighbors (living on the same piece trajectory). and a minimal initial state space separation ||e; 22,12 la. where we chose 150 (according to the mean periodicity [74]) and I 0.25. The local divergence between initially nearby trajectories then tracked over a range of time steps k 0... kmaz.
neighbor æ; is found under the constraints j
(k) = || - 2 ++
(16)
An estimate of the maximum Lyapunov exponent is given by
d.(k) = d, (0) Amnt
(17)
where At is the temporal resolution of the series. Taking the logarithm and averaging over all pairs leads to
(In d(k)) = Amt + C.
(18)
In chaotic systems, for a suitable choice of kmaa (here: 50), this quantity initially grows approximately linearly with time [45], with the slope given by Amax > O. In contrast, periodic (cyclic) systems have Amax O. As for Daup and DH. Lyapunov exponents can only be reasonably assessed for sufficiently long time series, with practical guidelines of T = 10d - -00 where d is the attractor's fractal dimension [96, 74].
A.2 Datasets
Training data DynaMix is trained on about 0.6 million simulated time series of length T = 550 sampled from 34 different 3d DS with cyclic or chaotic attractors, collected in [30]. Time series were standardized to ensure comparable scaling of all data, and Gaussian noise of 5% of the data standard deviation was added to all dimensions for the ground truth. Attractor dynamics were chosen reflect different types of behavior, see Fig. 10.
Test data Our test set for DS consists of simulated time series of length 10a sampled from 54 different 3d DS collected in [30], which
part of the training set.
Furthermore, we evaluated several 2d systems: The Selkov system [79] describing a kinetic model of an open monosubstrate enzyme reaction given by the equations
dr x + ay + x²
dt
(19)
dy
= ay x² y,
dl
where we chose a = 0.1 and = 0.5. The Van-der-Po system [89] describes self-sustaining oscillations in vacuum tubes by
dx
dt y,
(20)
dy
= p(1 - x²) x.
dt
where we chose 0.5. Similar to the Van-der-Pol system, the Rayleigh oscillator [63] describes a self-sustained nonlinear oscillator through
dr dt dy
y,
(21)
p(1
dt
where we chose μ = 1.0. As higher dimensional DS test case we use the Lorenz-96 system [56] defined by
dr;
(I141 21-2) Iz F.
(22)
dt
27

---

Figure 10: Training data (gray) and their reconstructions in red.
with system variables Iı, == I...,, and forcing term F' (here, F = 8 and N = 6, in the chaotic regime). Furthermore, cyclic boundary conditions are assumed with Φ- NN-1 To NN- DN+1 = I1, and the system was solved with integration step At = 0.08.
We further probed forecasting and DSR different types of real-world data: The traffic data are hourly recordings of the number of cars passing road junctions (https: : //www. kaggle com/ The cloud data, also used in [86] to evaluate TS foundation models, are publicly available from Huawei Cloud (https:/ / //gi thub. com/sir-labl aataadaaaaaaaa release). It consists of function requests from Huawei's serverless cloud platform. The weather data consists of daily sampled soil temperature and air pressure measured in the city of Mannheim, Germany (source: Deutscher Wetterdienst [German weather service]), and can be accessed via https::///// //www//..../ dwd. de/EN/eeeeeeeee htd//ousserviess cdc/ hte//d__uae celc_ueberblice wili-ikiee kl imadaten. en hhtml. The functional magnetic resonance imaging (fMRI) data comes from human subjects performing cognitive tasks and is publicly available on GitHub [51]. We followed Kramer et al. [51] and selected the first principal component of BOLD activity in each of the 20 brain regions. Kramer et al. [51] report positive maximum Lyapunov exponent for models reconstructed from these time series, indicating their chaotic nature (see also [91]). The E7Th| dataset is part of Electricity Transformer Temperature (ETT) benchmark, which is widely used for evaluating TS forecasting models. contains hourly data collected from a power transformer station [103] and can be accessed https: //gi //g/tttuu thub. com com/zhouhaoy/ zhouhaoyi/EEDataaaeeooooa data were taken from a study by Schalk et al. [77], comprising 64-channel data collected from human subjects performing various motor and imagery tasks. Following the approach of Brenner et al. [7], the signals were smoothed using a Hann window of length 15. As for the training data, all test data standardized for processing by the models.
datasets wwweet//fddssriieee aataannn///aiaaaeee tttaffffcccc predi
Electroencephalogria (EEG)
28

---

A.3 TS foundation models
Chronos Chronos is a recent TS foundation framework that adapts a transformer- LLM architecture for probabilistic time series forecasting [2]. Key to this approach is the tokenization of real-valued time series observations through scaling and quantization, transforming them into token sequences accessible to language models. The model is pretrained on extensive datasets, including both synthetic data generated via Gaussian processes, as well an extensive batch of empirical time series from [2, 33], including traffic, weather/ climate, electricity and web data, enabling it to achieve strong zero-shot performance across diverse datasets without task-specific fine-tuning. For our evaluation we used the standard pipeline as described in https: / het//gttttnt github. com/ mono/mmazo--oo amazon-science, chronos- forecasting.
TimesFM TimesFM is another transformer based TS foundation model, with decoder-only style architecture using input patching [21]. Its training corpus consists of synthetic as well as real-world time series data, and exhibits generalization across difffernnt time series domains and context lengths. Evaluation is performed as in https: //tgt/gttt/////////////////t com/ google- google-researchh research/ fimm/mmmm..mmmmmmmmm
Mamba4Cast Mamba4Cast is a zero-shot time series forecasting model based on the Mamba archi- tecture [36], a type of linear (" ("state-space) RNN with nonlinear imput & output gating, and inspired by Prior-data Fitted Networks (PFNs) [6]. Trained exclusively synthetic data, Mamba4Cast can generate zero-shot forecasts when provided with time series context information. For evaluation we thub. com/ haamm//utomm/aaaai Maattomm//Mmaaaaast.
follow https: //g
Tiny Time Mixers Like Mamba4Cast, Tiny Time Mixers is time series foundation model not founded on transformers [27]. It builds upon the TSMixer backbone [26] to which it adds several key innovations such as adaptive patching, diverse resolution sampling, and multi-resolution prefix tuning to improve generalization across datasets with varying temporal resolutions. Evaluation here performed as in https::/
/ // thub. com/ Nom/gla glehet/TTMi.
Panda Panda is recently proposed foundation model for short-term forecasting of DS [52]. is based on a transformer architecture, which relies on patching the DS-generated time series. 2 10a DS produced by combining base DS from the same database
The model is trained used to train and evaluate DynaMix in skew-product form [30]. Evaluation is performed as in abao / 199a//ada.......
https:/ //gs thub. com/abao
For comparability, all models were evaluated on the exact same CPU (18-Core Xeon Gold 6254) and GPU (Nvidia RTX 2080 Ti) using 512GB of RAM.
29

---

A.4
Further results
Figure 11: Zero-shot DSR (red) from 2000-step context of unseen DS not contained in the training corpus (ground truth in gray).
30

---

15-
MAA
AL
MALMAN
10-
10000
2250
1500
3750
100
200
500
12"4
MA WAL
13-1
C (
1014
230
300
750
1000
2250
1500
1750
2000
100
200
300
400
500
10a4
10m
Figure 12: Zero-shot DSR chaotic Lorenz-63 system (darkgray: context, lightgray: ground truth, red: model-generatedl Left: State space, center: time graphs, right: power spectrum.
1
11
10
400
590
1200
100
200
300
500
M
0.3
0.0
400
300
2000
1200
1400
200
200
300
502
Figure 13: sero-shot DSR 63renz-63 system in cyclic regime (darkgray: context, lightgray: ground truth, red: model-generated). Left: State space, center: time graphs, right: power spectrum.
15nd
10-3
200
503
10-3
MM
10
200
400
800
800
3000
1200
3AD0
300
200
300
400
500
10a4
10m
Time
Figure 14: Zero-shot DSR of chaotic finance system (darkgray: context, lightgray: ground truth, red: model-generated). Left: State space, center: time graphs, right: power spectrum.
NW 4404 200
1M 1200
10m
200
400
200
1000
I400
300
200
300
400
200
10-1
10m4
100
1200
100
200
100
500
2018
15
Figure 15: Zero-shot DSR of chaotic Genesio Tesi system (darkgray: context, lightgray: ground truth,
red: model-ggnerated). Left: State space, center: time graphs, right: power spectrum.
31

---

Lontext
Dyyaabiie
Chronos bolt Base
Panndaa
Finy iime adisers
Ground
Chronos
Chronos
Membbe
Timesh
(
Time Steps
Figure 16: Comparison of zero-shot forecasting of finance system for DynaMix vs. different TS foundation models.
Context
DynaMix
Chronos Bolt Base
Fonda
Tryy:Tme--iiorr
Ground truth
Chronos
Chronos
Merbon Caan:
TimesPN
Time Steps
Figure 17: Comparison zero-shot forecasting Sprott system for DynaMix vs. different TS foundation models.
32

---

Lontext
C.hronos bolt Base
Plandla
tiry- irme anloerr
Ground
Chronos
Chronos
Membbe ACCss
Timeshi
Now 1MF-1///0001111//000000000111110
WWWVV//VVVVV//00/////// Time Steps
Figure 18: Comparison of zero-shot forecasting of Sprott M system for DynaMix vs, different TS foundation models.
Context
DynaMiix Chronos Beee
Chronos Bolt Base
Fonda Merboon
Tiry.Time-Wisor
Ground truth
Chronos 2
Timese
WY WY M "
W "
MA MUA MI
AA
"
AN MAL
WWY WY YO A
MA MUA MA
AAA WI
MI MN AMM
AN
WULL
MI MN
Time Steps
Figure 19: Comparison of zero-shot forecasting Vallise El Nino system for DynaMix vs. different TS foundation models.
33

---

Context
Dyyaatii
Chronos boll Base
Handl
tiry iime adisers
Ground
Chronos Base
Chronos
Merboe
TimesrM
YuM MM wwwwy mwwny
w/ AM mwmmn
All AuM ( Luo
yna WM
w/ w/ momnt muy
11 Mmnyy"
4 wii.
womer "MM /M
w/ mome /M (M A Ammnn Al A. AM
w/ Nhh w Ww
-
A m4
w/ m ( M
AM w/
wwwl
wwnyy
wwyy
Time Steps
Figure 20: Comparison of zero-shot forecasting of Sprott C system for DynaMix vs. different TS foundation models.
Context
DynaMii
Chronos Bolt Base
Fonda
Tiry.Time-Wisrr
Ground Iruth
Chronos Beee
Chronos 2
Timesr
Time Steps
Figure 21: Comparison of zero-shot forecasting Sprott system for DynaMix vs. different TS
foundation models.
34

---

a
4
2
2
0
-2
-2
-2
0
2
-2
4
Figure 22: Zero-shot forecasts (red) for the a) Van-der-Pol system. b) Rayleigh oscillator (true vector field in lightgray) from different initial conditions outside the context range (darkgray).
a
Geometrical Disagreement
Temperal Disegreement
10-step ahoad Prediction Error
Creso loy.
Ormeed--
Dynatt
11.1
SyraMic
6322
<5
OprorSaam
Ank
0.S.
Oreezawe
Trom
Orewn sihee
0.51
15
Charmie
Figure 23: Zero-shot DSR performance across all 54 test set DS for DynaMix and various TS foundation models in Fig. 4, but for context length Tc = 512. Numerical details in Table 10. Statistical testing based on Wilcoxon signed-rank tests.
Low MAE
High MAE
2
1
0
-1
-2
100
200
300
Figure 24: Exponential divergence of two initially nearby trajectories (in blue and red) illustrated for the chaotic Lorenz-63 system. The prediction error is still sensible on a short time scale (MAE 0.09), but then rapidly increases and breaks down suitable metric in the longer- term (MAE 1.12), although both trajectories were drawn from the exact same system with the very same parameters.
35

---

15
23 30
10 25 20 25 30
10
15
25
30
Figure 25: DynaMix' performance as a function of the number of experts J. While a minimum number of experts is necessary, performance already plateaus after surprisingly small number of experts, suggesting there is lot of room for scaling the model to larger datasets. Error bands = STD
20
40
80
100
20
60
100
40
80
100
Interval
teacher forcing Interval
teacher forcing Interval
Figure 26: DSR performance for different teacher forcing intervals т, illustrating an optimal value is essential for successful training, see [60, 7]. Error bands STD
Context
Ground Truth
DynaMix
Chrones
Partially
M wy m
M/
mal V www AY
200
-100
-15
Time
Time (hours)
Traffic
d
Cloud Requests
MM MW Mww Weather (Temm..)
WW
Value
2000
2000
Weather (Press.)
2000
My
2000
Time idays)
Time (clyys
Human TMRI
-200
-100
100
150
200
-2000
.500
1009
2008
(seconds)
(seconds)
Figure 27: Comparison of DynaMix (red) to Chronos (blue) on zero-shot forecasts of various empirical time series: Forecasts of a) partially (1d) observed Lorenz-63 DS, b) electricity transformer temperature data, c) hourly car traffic data with weekly cycle, d) Huawei cloud request data, e) soil temperature development. f) air pressure data, g) human functional magnetic resonance imaging (fMRI) data, h) human electroencephalogram (EEG) data.
36

---

Context
Ground Truth
DynaMix
Mamba4Cast
Partially obs.
ETTh1
/n // /m M
MN MAW MVW
Trafficc
d
Cloud Requests
2000
2000
Time (hours)
Time
(Temp.)
Weather (Press.)
My
-1008
-750
-500
-20
200
750
1000
-200
200
660
Time (days]
(daass)
Human IMRI
h
Human EEG
Velue
(seconds)
(seconds)
Figure 28: Comparison of DynaMix (red) to Mamba4Cast (green) zero-shot forecasts of various empirical time series: Forecasts a) partially (1d) observed Lorenz-63 DS, b) electricity transformer temperature data, e) hourly traffic data with weekly cycle, d) Huawei cloud request data, soil temperature development, f) air pressure data, g) human functional magnetic resonance imaging (fMRI) data, h) human electroencephalogram (EEG) data.
Context
Ground Truth
DynaMix
TTM
Partially
b
ETTh1
M mm // /m/
Mmn
VM
NM WWW MOW TAYA
Traffic
dl
Cloud Requests
My Mh MM M1
Value
2000
Time (hours)
Time
(Temp.)
Weather (Press.)
My WWW
-1000
-700
-500
-008
300
750
1000
-200
20ee
800
Time (days]
(clays)
Human IMRI
h
Human EEG
volue
(seconds)
(seconds)
Figure 29: Comparison of DynaMix (red) to Tiny-Time Mixers (purple) zero-shot forecasts of various empirical time series: Forecasts of a) partially (1d) observed Lorenz-63 DS, b) electricity transformer temperature data, e) hourly car traffic data with weekly cycle, d) Huawei cloud request data, e) soil temperature development, f) air pressure data, g) human functional magnetic resonance imaging (fMRI) data, h) human electroencephalogram (EEG) data.
37

---

Partiallyy Context
Ground Truth
DynaMix
TimesFM
M wn mw V mn
M/
-100
Time
Time (hours)
Traffic
d
Cloud Requests
My Myy Myl
2000
Weather (Temp.)
Weather (Press.)
2000
My
2000
Time (days)
Time (cays)
Human
-200
-100
100
350
200
-1000
-008
1000
1208
Time (seconds)
Time (seconds)
Figure 30: Comparison of DynaMix (red) to TimesFM (orange) on zero-shot forecasts of various empirical time series: ground truth): Forecasts of a) partially (1d) observed Lorenz-63 DS, b) electricity transformer temperature data, c) hourly car traffic data with weekly cycle, d) Huawei cloud request data, e) soil temperature development, f) air pressure data, g) human functional magnetic resonance imaging (IMRI) data, h) human electroenceehalloramm (EEG) data.
Groid Tuth
Chrinis
POEALINN
ambedding
Certed
w whll, pa Mwwnaaaa
Dyrelher prices
Tyrolro Deton t1re33
18na
Figure 31: Failure of proper zero-shot DSR. a) DynaMix struggles with sharp peaks in an otherwise much more slowly evolving time series. b) As quantified in Fig. 3, selecting a too broad temporal resolution of the context time series can lead to improper reconstructions with, in this case, temporal disalignment w.r.t. the true signal. However, this is general problem for any foundation model, and Chronos fares even much worse in this example. e) Zero-shot DSR may fail (red) as well if the context time series is not properly embedded. In this example, the required embedding dimension exceeded the model capacity. However, shown in orange, this can be amended by using the positional encoding, eq. 6.
38

---

A.5 Using DynaMix for non-stationary time series
Similar as in FEDformers [104] and related TS models which use specialized decomposition and filtering blocks for handling von-stationary data, one could add simple preprocessing operations to the DynaMix pipeline separate out trend or other non-stationary components the context signal. For simple illustration, here we first apply Box- transformation to time series
if 0
2
(23)
log((r),
where À is estimated by maximizing the log-likelihood. Next, trend components of the form
f(t; 0) = 0,t + +
(24)
are inferred by least squares estimation and subtracted from the context signal. Standard DynaMix is then used to forecast the embedded residual context (see Sect. 3.3), after which the estimated trend model is added back on.
We tested this simple setup on the non-stationary Air Passengers dataset containing passenger counts of an airline (https: //www kaggle. com/ hat/astasss/////e// httpss/hhhag///// /air-passssggrr) The results in Fig. 32 illustrate that this is in principle a viable direction, although constituting just a proof-of-concept at this stage.
600
Ground Truit
Contesit
500
Dy
400
300
100
-150
Time
Figure 32: Example forecast for the non-stationary Air Passenger data using DynaMix with prepro- cessing pipeline.
A.6 Scaling up context dimension
For reconstructing DS sigher-dimensional than the context dimension defined by the architecture and used in training. one can use DynaMix without any modification by employing the delay-embedding theorems [82, 76]: Zero-shot infer the underlying DS from the observed TS and then delay-embed DynaMix output into sufficienlly high-dimensional space which assures diffeomorphism between original and reconstructed attractor. This idea is illustrated for the 6d Lorenz-96 system Fig. 33. Here, only the first two dimensions of the simulated Lorenz-96 were provided as context, for which DynaMix then produced long-term forecasts. The good geometrical and temporal agreement between delay-embeddings of both the two ground truth and the forcasted TS confirms that DynaMix has correctly inferred the underlying 6d DS despite its own 3d structure.
DynaMix Loren296 (Daup 3.48, Dy 0.19)
Gimansions
Steps
Figure 33: Zero-shot forecasting of partially observed (2d) Lorenz-96 system by DynaMix. Both context signal (ground truth) and DynaMix' forecast were subsequently delay-embedded into 6d, the system's original dimension. The black line indicates the transition between context and forecast.
Alternatively, if more dimensions are observed than the gating network can handle by design, N > N, and are all to be utilized for DSR and forecasting, then the gating network needs to be
39

---

modified architecturally such that it can receive N = N dimensional context signals. To allow for this without changing the 3d DS training corpus, observations æ E R were embedded into a higher dimensional N >> 3 space using a nonlinear transformation of the form
xx" tanh( {x)²" T RR
E = (z)
(25)
where the entries in A ER3 are chosen randomly from 0aj ~ (),,1) yielding embedding dimension of N = 3 + F.. DynaMix is then retrained on the embedded and standardized original 34 DS training set. Note that this embedding does not change the nature of the dynamics of the trained-on systems in any way, because femb is an instantaneous (time-independent) transform that is only applied posthoc (after the DS has been simulated). essentially places trajectories onto 3d manifold embedded within an N dimensional ambient space. Note that this yields general recipe for any desired N.
As shown in Fig. 34 and Table 2, using this embedding DynaMix is able to generalize to all 6d of the Lorenz-96 system, although only trained on 3d systems, while all time series foundation models fail.
DynaMix
Drmessore
P. N i
Chronos T5 Base
MambedCast
Diminiens
Tiny-Tima-Mixee
u N P
TimesFM
Dimansiens w 1
1006
2000
4000
Steps
Figure 34: Comparison of zero- shot forecasting of 6d Lorenz-96 system by DynaMix vs. different TS foundation models. The black line indicates the transition between context and forecast.
Table 2: Zero-shot DSR performance across 5 different Lorenz-96 trajectories for DynaMix and various TS foundation models for context length To 4000. Med of Datsp (geometrical disagreement), DH (temporal disagreement), and MASE (short-term prediction error).
Model
D stsp
D#
MASE
DynaMix
1.93 = 0.08
0.07 + 0.00
1.02 + 0.06
Chronos-(5-baad baase
12.74
0.076
2.44 : 1.30
Mamba4Cast
13.34 + 0.36
0.70
2.93 + 0.45
TTM
14.54 0.02
0.26
4.00
TimesFM
11.00 0.18
0.67 0
4.82 + 0.56
40

---

A.7 Comparison of DynaMix to custom-trained DSR models
To put DynaMix' zero-shot DSR performance further into context, we compared it to different SOTA DSR models explicitly trained on the respective context data. Note that for the custom-trand trained models unlike DynaMix this constitutes a form of in-domain generalization, as the context signal and the forecast come from the same ergodic distribution. For these reasons, it is also difficult to establish fair comparison, since nyper-parameten of the custom-trained models could be arbitrarily fine-tuned on the in-domain data (with often better results the model size is further increased, indicating a tendency toward over-fitting. For the comparisons performed here, we therefore decided to mainly rely previously reported evaluation on similar DS, without additional fine-tuning (which we did not do for DynaMix either).
settings, which often were already fine-tuned for
Our first comparison model is the same AL-RNN used for DynaMix' experts (with R 500 trainable parameters), i.e. given by eq. Training was performed as in Brenner et al. [8], and hyperparameter settings, as collected in Table 3, followed those in the original paper.
Table 3: Hyperparameter settings of custom AL-RNN.
Hyperparameter
Setting
M
20
P
8 (Lorenz-96: 14)
T
16
T
200
batch size
16
10-3
7/stur
10-5
Rend
epochs
2000
As another SOTA DSR model we trained Neural-ODEs [18], using the same type of MLP architecture. training setup, and same hyper-parameter settings as [38], see Table 4. With this, the total number of trainable parameters is already more than twice (!) than that used for DynaMix (=s 10k), and exceeded the number of data points (S 4k) manifold. Indeed, results became worse with less parameters, potentially indicating that Neural-ODEs particularly struggle on the short training segments.
Table Hyperparameter settings of custom Neural-ODEs.
Hyperparameter
Setting
hidden layer
[100, 100, 100)
activation
ReLU
T
30
batch size
32
ODE solver
Tsit5
10-3
10-5
Nend
epochs
100000
Finally, Reservoir Computers (RCs) are widely used for DSR. Here we employed the architecture provided in Patel and Ot [67], trained as in Göring et al. [38] and using the same hyper-parameter settings, summarized in Table 5. Note that this endows RCs with about 25x more parameters in total, of which < 3k were trainable, than DynaMix had at its disposal.
41

---

Table Hyperparameter settings custom Reservoir Computer.
Hyperparameter
Setting
M
500
p
1.0
0.7
0
0.2
B
0.5
With these settings, performance results Tables and 7 indicate that DynaMix within the same ballpark as the custom-trained DSR models, in contrast to all the TS foundation models. For the empirical time series, Table 8, DynaMix sometimes even seems to have an advantage over custom-trained models, potentially because it makes more efficient time series (for a fair comparison, custom-trained models were provided with the same positional embedding DynaMix; using standard delay embedding in 3d for the custom-trained models actually produced worse results). At the same time, DynaMix admits orders of magnitude faster inference times (see Fig. 5) and performs true oul-of-domain generalization without parameter fine-tuning.
of the relatively short context
Table 6: Performance of DynaMix and custom trained DSR models across all 54 test set DS. MAD of geometrical divergence (Dssp), long-term temporal distance (D#), and forecast error (MASE).
Model
Dasp
D#
MASE
DynaMix
3.8 + 1.4
0.16 + 0.06
0.35 + 0.17
AL-RNN
3.86 + 2.48
0.08
0.06
0.47 = 0.23
Neural-ODE
3.43 2.05
0.09
0.04
0.18 = 0.13
RC
1.33 + 0.99
0.08 + 0.07
0.86 = 0.37
Table Performance DynaMix and custom trained DSR models across 5 different Lorenz-96 trajectories. Median+ of geometrical divergence (Dasp), long-term temporal distance (D,r). and forecast error (MASE).
Model
Dasp
DH
MASE
DynaMix
1.93 + 0.08
0.07 + 0.00
1.02 = 0.06
AL-RNN
2.12 + 0.18
0.15 +115+00 0.03
0.83 + 0.29
Neural-ODE
0.76 0.06
0.06
0.34 0.16
RC
1.26 + 0.51
0.13 0.08
1.10 + 0.42
Table 8: Performance comparison on empirical time series in terms of geometrical divergence (Darp). long-term temporal distance (Dx), and forecast error (MAE), Best in red, second-best in blue.
System
DynaMix
AL-RNN
RC
Neural-ODE
Dus
Du
MAE
Dstsp
Dh
MAE
D
Das
DH
MAE
Dasp
Du
MAE
ETTh1
0.22 0.08
4.92
0.26
0.08
6.82
0.60
0.75
44.41
0.70
0.09
4.55
Traffic
0.81 0.21
6.93
0.95
0.13
7.29
0.19
0.70
7.08
0.85
0.17
5.03
Cloud Requests
0.27 0.14
161.58
0.19
0.20
529.91
0.24
0.19
587.35
0.37
0.30
172.61
Weather (Temp.)
0,66
0.09
2.24
0.53
0.11
2.08
1.20
0.11
2.50
0.65
0.11
2.23
Weather (Press.)
0.39
0.19
9.03
0.28
0.41
8.84
0.55
0.45
61.31
1.13
0.34
11.08
Human fMRI
0.17
0.09
0.45
0.10
0.11
1.33
0.16
0.19
0.95
0.23
0.06
0.78
Human EEG
0.79 0.23
1.07
0.23
0.44
0.70
1.80
0.55
0.89
1.23
0.26
0.91
42

---

D MAB
0.5
1,1
08
19
02
05
25
13
23
11
01
04
11
01
38
14
04
08
044
23
210
08
12
21
13
25
0.
13
14
10
02
16
24
0.1
02
31
10
32
25
0.4
21
00
12
50
17
TimesFM
0.4
033
04
02
03
02
02
06
03
05
02
02
02
06
04
03
63
01
05
055
0.5
02
63
32
03
07
06
02
05
06
04
01
04
055
01
04
02
05
05
05
05
02
05
622
05
0.4
Table 9: Performance comparison for ZCTO shot inference actoss all DS in the dataset from [30] using To = 2000. Best value in red, second best blue.
Doup
9.7
124
114
2.6
9.5
130
132
122
110
12A
142
20
125
126
128
131
124
1144
133
8.2
119
135
135
134
78
130
95
8.
81
35
11.
11.3
128
143
138
109
182
152
125
136
182
149
Dir MAE
1.5
56
21
4.6
40
35
60
1.7
42
17
32
22
27
61
82
28
22
27
21
76
16
30
12
37
21
63
40
28
19
88
37
102
75
25
102
48
31
11
12
38
110
47
34
51
TTM
0.4
0.3
04
02
04
02
04
0.4
03
02
03
8.3
0.5
04
02
03
05
03
0.3
011
066
01
04
04
03
0.2
03
02
02
03
0.4
03
0.
03
023
0.5
1.4
0.3
04
02
04
02
MAE Dasp 27 MAL Day
14.2
14.0
12
13.8
13.0
140
12
116
125
156
13.2
139
13.
130
134
14.0
1299
12
136
13
130
14.0
133
13
144
142
13
138
11.6
155
13.7
11.1
142
135
126
44
1.2
40
08
08
33
49
13
20
0.7
61
16
12
28
14
1.5
25
1.5
16
14
20
13
41
24
61
0.5
RX
300
16
3.3
34
MambuotC
0.3
0.7
0.7
06
0.4
08
08
0.5
0.6
0.9
89
0.6
0.
0.7
0.7
0.8
0.2
08
03
0.
08
0.8
08
107
0.8
02
0.8
038
11.2
11.4
12.
102
11
11.3
12.5
126
104
104
11-7
124
12.3
1E1
133
114
124
1044
102
93
13
102
11
141
122
108
116
10.9
11
11.8
126
102
12.0
13.5
12
12.5
106
06
0.4
0.7
0.3
02
07
09
0.3
055
0.5
0.7
0.4
06
18
04
024
02
03
0.5
04
04
055
0.2
0.7
Pardo
D
0.3
0.2
022
02
0.2
02
022
02
81
03
03
03
82
03
022
03
03
022
055
03
02
0.3
01
03
0.5
02
02
22 MAB Dap
13.0
140
130
12.3
12W
6.2
11.
124
128
132
101
91
119
70
122
124
125
11
134
11
126
86
122
122
128
132
02
75
110
128
100
11.5
80
128
03
13.1
121
12.5
131
11.
91
0.3
066
0.9
111
02
19
33
0.6
08
05
83
04
14
12
04
02
16
13
03
13
14
01
03
0.7
04
01
11
0.5
12
23
03
48
34
13
0.4
0.5
16
Chrooe-3
0.5
0.3
011
03
8.2
02
03
0.3
01
03
04
81
0-2
0.2
0.3
03
05
02
03
07
01
03
01
0.5
03
05
0.5
03
04
01
02
02
0.3
0.5
04
03
03
03
0.3
0.2
033
9.4
02
066
0.2
Daa
10.9
13.5
6,7
12.4
844
39
1111
100
12.8
11.5
68
130
37
011
11.
160
128
13
077
112
86
134
123
137
31
12.
80
86
88
93
10.
9.8
81
119
135
132
48
132
12.7
10.7
151
910
12.8
Chronne Seooon----orr -a-e-----rrpp Chromne-555 hase Chrmneet5- (-o------nnllll Bhronox-4--ooi S--iint Choomoe-ooe waamon--llehaae laae Cheomuee womeeeeboll-oo annll- Chrrmn bwit-mint Dos Dy MAE Daep D1r
D MAN
2.
0.5
14
06
37
20
04
07
06
17
08
11
08
10
22
18
09
1.
46
18
87
31
63
18
33
44
0.9
0.9
0.5
03
0.7
0.7
04
0.6
0.4
0.7
0.6
086
05
0.5
0.8
0.8
09
077
06
140
0.9
0.5
0.7
07
09
06
07
0.5
0.5
0.6
03
08
09
0.8
0.5
0.6
0.7
0.6
0X
08
0.9
0.7
0.7
0.6
0.5
08
0.6
0.4
Doop
12.3
119
12.3
11.
96
1.2
124
106
92
89
139
34
97
10.5
110
127
11.
9.5
10.1
52
77
134
125
10.
99
104
922
84
09
855
119
96
110
53
12.3
10.
10.5
12.8
135
MAE
311
0.9
13
09
1.5
0.8
400
1.4
110
20
22
0.4
4.6
21
0.5
83
27
1.6
11.
0.6
17
16
21
16
688
0.6
1.5
19
49
28
26
9.6
1.9
12
61
83
19
0.8
0.9
1.8
177
38
44
1.5
Deop Dyr
06
07
06
07
01
06
09
06
0.
03
8X
07
05
07
05
00
04
09
O8
06
07
06
05
06
07
06
09
08
09
05
07
0.5
08
06
0.5
8X
05
02
08
07
12.0
11.0
94
88
117
12.9
11.7
114
106
102
95
1099
115
113
129
106
10.
100
194
10.9
104
114
98
104
11.
104
83
126
13.4
1099
122
125
MAA
E4
23
L33
26
04
24
54
08
28
08
07
06
13
035
36
13
04
63
24
58
06
1.2
10
13
09
1-3
08
13
19
35
18
17
53
26
0.
122
42
34
32
1.0
13
35
37
0.
1.5
0.7
06
0.7
08
0.4
07
0.3
06
08
077
0.7
077
03
0.7
09
07
07
65
08
0.7
08
0.7
09
09
11.6
12.4
11.2
11.0
04
43
11.0
12
120
10.7
10.
108
79
922
1038
11.
109
11.5
106
92
10.2
F27
12.2
99
10.9
102
105
11.8
10.6
10.6
62
12.6
11.4
18.6
22.2
13.0
11
124
0.6
0-4
0.5
14
01
89
04
86
06
01
0.
015
08
013
11
08
0.5
04
23
0.2
18
06
06
0.4
06
08
06
13
08
0.2
07
0.3
0.2
022
04
0.
04
04
87
04
0.3
04
05
04
0.
01
0.7
0.4
04
83
0.5
0.4
01
03
03
02
0.5
0.4
86
05
0.5
0.4
05
05
03
13.3
169
13
11
122
33
135
11
129
122
116
132
125
910
128
11
138
120
122
13.5
12.5
188
111
125
11
135
132
126
222
110
118
120
125
11.6
125
126
145
139
125
11.4
122
11
13
124
132
MALE
0.5
0.3
13
0.3
06
01
04
89
1.5
05
03
03
05
0.
03
14
04
0.
04
05
05
1.2
03
30
13
06
0.4
03
Doop D1
6.1
02
02
0.2
0.3
011
84
0.4
033
022
02
0.3
0.3
02
05
0.3
0.3
81
02
0.5
0.2
06
0.3
0.5
05
03
120
13.3
116
81
128
29
133
118
134
72
11.
11.9
07
10.
75
128
100
11.
125
11.
11.5
75
114
64
11.7
129
125
160
90
82
124
127
738
11.
123
16.5
109
184
12.5
118
13.5
124
124
11.0
MAL
0.5
04
05
19
03
01
04
099
10
01
01
03
03
21
01
06
01
86
03
11
03
870
100
05
010
Dop Dy MAL Doop 111
0.4
0.4
0.2
61
03
63
04
055
0.4
06
01
02
06
05
03
05
04
02
63
04
03
06
04
03
05
84
044
05
05
03
04
03
01
05
06
87
08
63
0.5
05
05
05
03
066
9.3
12.1
56
9$
36
124
0.2
132
94
82
11.
93
84
79
844
97
10.
64
53
100
9.5
9.6
10.1
50
127
36
124
160
52
22
113
68
105
119
26
63
69
104
101
185
112
122
107
84
100
39
0.0
0.
64
05
16
60
06
22
03
03
02
R3
62
3D
13
13
61
04
22
02
63
01
05
63
12
03
23
03
62
81
07
03
39
63
61
23
04
60
0.4
037
05
61
06
62
100
05
03
01
05
62
05
05
84
01
04
05
01
61
86
34
01
02
04
06
05
05
34
05
04
04
46
05
86
05
05
01
01
02
88
09
07
05
05
05
03
95
01
07
05
03
9.2
12.1
7.0
80
96
422
85
89
104
108
27
37
10.0
10.1
11
64
6.5
10.1
35
99
855
56
129
45
98
96
74
97
98
10.8
10.1
25
10.1
125
25
84
132
11.4
9X
9.
11.1
Dany pt MAE
0.6
0.2
0.3
0.3
02
0.3
13
08
0.3
04
022
0.3
0.1
70
33
02
0.2
02
A3
011
01
07
03
0.3
0.4
0.4
07
0.
05
06
144
03
02
03
0.2
24
06
0.5
01
06
06
0,4
DinuMii
0.1
022
81
03
01
0.2
08
04
04
01
012
80
02
02
011
03
01
01
01
01
03
01
02
0.1
R00
0.5
0.3
0.1
388
288
855
31
98
7-4
877
3.2
88
95
92
38
45
42
4.9
8.3
Chukkahhiin ner Holmos
San Unn Stiuchh owgnng
Reginee
CV Taye Pls Quiipp
Thomas Labyrish
Witamm RetmmiRReeee Reduced
BeaRANN
Bosall
Bude Show
Chen Les
Thia
Coulier
Li
Fumnee
Cesrrio Tost
Hodiey
lindmana
Jolk Creait
Laaer
Lornz 64
Chen
Spirgel
Nove Nows Hoovr
Pohlivan Wei
Sakargg
Spennt A
Ngonn B
Spenn
Speil F
Spmin G
Speiil
Spont L
spenn P
Spent
Spents S
Spoul Torms
Thomas
Torus
vallise Et Nino
Worl Sun
yo Wong
Yo Wing 2
Zhon Chen
43

---

Dy MAB
0.
0.3
1.5
19
18
07
12
20
26
1.0
00
38
13
11
10
21
80
15
05
31
15
14
11
15
06
11
11
03
02
06
15
08
16
13
09
63
0.0
53
19
13
24
17
20
34
Times
0.8
04
03
04
03
04
02
04
03
84
04
01
033
055
03
63
04
01
02
055
6.5
01
04
02
05
04
01
04
05
05
02
01
03
04
055
64
03
01
01
0.5
04
05
04
03
04
0L5
Table 10: Performance comparison parisonn for zero-shor shot inference across all DS in the dataset from [30] using Tc 512. Best value in red, second best blue.
Doup
139
12.3
139
114
12.
136
108
115
133
75
134
132
138
131
74
129
128
31
138
128
131
135
132
152
105
97
11.7
96
91
124
8.9
11.3
128
125
134
152
10.5
138
134
14.5
Dir MAE
1.3
23
3.6
26
34
3.5
29
45
43
54
35
42
34
41
19
26
31
19
21
33
73
25
09
29
29
21
18
20
32
31
32
32
60
60
50
15
23
12
42
1.6
14
4.5
18
172
TTM
0.4
0.5
07
05
04
04
0.4
0.5
06
05
055
0.
R6
0.7
0.5
0.
05
06
07
0.1
0.5
04
05
02
86
0.5
06
0.5
06
03
02
R.5
0.4
86
0.5
0.5
04
0.6
0.5
0.6
05
03
Dasp 27 MAL Day
14.
11.4
13.7
13
61
13.
137
126
122
129
14
13
134
134
13
135
133
12.
140
126
13.
114
140
144
142
104
132
136
12.5
149
136
11
13
12.6
14
132
100
2.3
3.7
10
09
31
23
1.4
100
63
25
0.7
29
06
25
27
0.5
29
41
08
18
39
13
26
09
06
22
22
111
25
0.3
21
21
311
MambuetC
0.9
0.8
0.9
0.8
0.4
89
0.0
0.7
0.9
0.9
088
0.8
0.5
03
02
0.7
0.3
07
0.
03
02
0.9
03
02
0.5
099
0.7
0.7
0.0
0.7
11.3
13.0
12.
12.2
11
1100
11
10.7
11.1
150
102
11.
12.0
10.5
11.8
11
112
10.5
100
124
107
129
11
18.0
109
122
108
11
107
122
11.
12.5
11.5
12.6
13.5
11.4
120
11.6
7238
MAE
0.5
0.5
0.4
0.3
0.9
0.8
0.4
08
022
0.5
014
08
06
0.7
0.6
0.5
06
0.5
01
04
0.4
02
0.5
005
19
0.3
Pardo
D
0.3
0.4
04
03
0.
02
03
04
02
03
04
03
83
022
033
04
03
03
02
02
02
02
05
03
0.3
0.1
044
D1 MAB Dap
13.2
13.
13
13.2
133
6.2
122
120
13.2
135
12.5
16M
120
58
12.5
13.5
131
127
134
11.
131
12.
8.3
11.
120
115
120
13
105
822
120
11
130
98
120
8.5
136
126
1322
12.
13
134
102
11.0
1.3
0.5
19
05
02
02
1.3
19
12
85
47
04
03
077
48
88
12
08
22
05
22
11
12
05
05
02
0.6
13
03
06
43
28
1.0
155
0.4
41
13
09
Chrooe-3
0.3
0.4
04
04
03
03
03
0.5
0.4
03
044
01
02
0.3
0.4
03
04
04
0-2
8-3
0.1
04
02
0.5
02
03
0.3
055
02
03
02
0.2
0.1
0.4
03
03
04
0.1
0.4
03
0.5
02
0.7
0.4
Daa
18.2
7.6
11.1
78
49
98
922
11.5
110
74
82
129
6.5
988
166
24
11
128
129
131
81
10.8
10.5
104
115
16
132
6.6
86
67
76
7.6
11.7
74
115
10.1
129
73
0.5
10.
6.3
12.8
96
1300
Chronne Seoonn--5-arr 4a-e-----rpgg Chromnest5 Saae Chrmnnet5- -ennennnlll Bhrono--4--ooi ---niii Chromos- hoo--hllee hale-hase Cheomuee wowleeeoolloooe aanall Chromn bwit-mint Di MAL
D MAN
20
22
20
24
20
22
10
29
OX
1.7
30
20
6.5
05
35
31
12
34
110
0.7
31
08
22
300
21
0.9
0.9
0.7
0.7
06
0.7
088
0.8
0.4
ON
08
086
08
0.8
0.8
0.9
09
077
06
09
0.5
0.7
03
08
08
08
0.5
0.7
086
08
09
09
08
0.8
0.7
099
09
09
07
1.9
0.7
0.2
0.7
0.9
077
07
Doop
13.5
12.2
12.2
12.
80
966
134
103
94
27
129
10.8
119
129
130
132
11.
12.7
134
12.7
12.6
11.5
120
124
132
129
04
12.
87
12.5
106
112
79
137
755
135
112
119
12.8
8.6
135
MAE
24
11
1.4
38
28
1.7
11
25
47
20
388
0X
28
34
14
27
1.1
1.4
68
25
055
27
22
4.
1.3
40
25
17
11
57
24
34
11
52
0.7
30
1.3
19
44
18
5.
Dyr
09
077
06
08
07
04
08
02
07
03
08
00
08
09
05
07
09
01
09
06
09
07
08
06
6X
00
03
08
01
02
08
04
06
6X
08
07
08
08
Dap
966
12.1
116
79
113
103
105
96
119
11.
135
12.
130
122
126
113
120
12.7
130
113
12.0
123
12.4
119
123
88
10.2
121
130
12.5
11.9
113
1200
MAL
2.2
1.0
1.3
10
18
28
31
18
14
66
51
11
34
L8
20
28
133
20
1.5
29
56
05
11
41
27
08
122
26
13
28
19
28
08
11
44
41
28
1.3
13
22
29
26
Daop D1r
0.8
0.8
0.9
89
0.2
07
09
06
09
0.7
88
09
09
09
03
0.8
03
099
09
0.7
09
11.0
10
11.5
11.3
11.
11.5
11.6
1044
96
11.1
11.7
10.
12.5
12.6
12
11.
11.2
11.1
10.
11W
12.3
12.3
121
11.
10.2
70
11.5
11.6
93
11
103
19.2
MALE Dos Dy MAE
0.5
1.5
012
03
19
014
10
11
18
04
002
0.5
20
08
06
13
06
08
022
05
03
01
07
022
03
0.3
014
37
01
0.3
0.6
04
0.3
02
04
022
0.3
0.2
84
06
0.6
03
02
87
04
08
03
07
03
88
0.4
055
03
03
02
0.5
03
04
87
03
83
0.5
0.5
0.
06
11.1
1322
[3]
46
117
121
12
125
94
125
126
740
11.8
11.2
128
126
110
11.
130
129
102
102
130
11.7
129
130
37
101
11.
135
129
12.5
130
145
134
120
12
122
104
136
122
128
0.3
11
03
09
0.4
10
85
155
03
0.3
03
05
03
10
1.0
0.5
10
0.3
122
0.3
02
03
05
03
20
0.3
13
3.4
07
MAE Doop D1
0.1
0.6
04
8.3
0.3
044
02
05
0.3
03
0.5
063
0.2
84
0.6
0.3
03
04
0.5
022
03
0.2
0.3
05
0.3
0.5
0.4
0.0
10.5
9.3
86
130
92
48
28
21
77
124
125
96
83
98
185
11.
130
126
125
134
108
94
82
124
75
55
126
21
74
102
128
11.1
100
65
11.
130
134
120
160
744
1322
136
103
120
12.7
0.3
0.3
15
03
0.7
89
11
814
18
03
0.1
03
13
16
05
89
03
13
03
01
03
0.2
0.3
02
33
1.4
02
03
39
06
0.4
6.2
0.3
05
RJ
05
009
0.5
0.4
04
02
R1
05
03
86
04
03
03
04
03
86
07
04
83
08
86
05
02
01
04
0.4
05
03
83
05
03
84
02
02
04
03
06
03
Doop
11.1
3.5
7.3
9.0
R3
40
59
4.7
11.0
86
8.5
12
9.7
450
52
105
91
82
855
941
109
119
108
28
50
105
118
120
58
89
62
108
822
185
9.
62
10.2
120
121
119
18.
5.5
103
112
113
106
137
0.0
0.2
6J
12
R3
03
06
0.5
11
15
10
08
84
96
05
04
30
13
13
01
02
65
67
62
12
68
1.7
02
03
01
01
02
05
05
03
04
02
26
000
19
02
13
31
05
03
26
0.5
0.5
05
05
0]
03
03
06
03
86
04
01
05
R1
05
86
05
06
05
06
06
41
62
02
05
08
02
04
41
05
03
05
06
05
04
01
06
06
04
05
01
85
01
07
02
Dop
9.7
8.3
103
72
74
85
79
11.
27
25
25
82
88
755
9N
31
91
120
10.3
46
26
22
85
26
10.8
47
86
22
93
95
69
96
68
125
1099
10.1
11.
89
98
10.0
16.4
Dany yt MAE
0.5
0.5
0.7
03
0.2
21
06
0.
03
066
0.2
0.5
02
05
05
037
0.4
0.0
19
01
06
09
0.4
066
0.5
03
03
02
04
0.4
03
07
03
01
32
0.
1.5
0.2
0.5
18
0.3
0.6
0.1
0.2
01
0-4
0.5
011
0.2
81
04
03
02
02
0.5
02
03
01
0.3
00
02
03
00
02
0.2
0.3
0.3
011
41
2.3
12.7
83
06
11.3
4,3
455
11.4
48
31
24
81
36
5,3
5,0
Shukkahhiin ner Holmos
San Unn Stinch owgnng
CV Taye Pls Quaiip
Thomas Labyrish
Wirmmm RetmmiRReeee Reduced
BerRNN
Bosall
Bude Show
Chen Lee
Thia
Coalier
Li
Fumnee
Cenrrio Tost
Hodiey
dindmana
Jelk Creaat
Laaer
Lornz 64
Chen
Spirgel
Nove Hoovr
Pohlivan Wei
Sakarga
Spennt A
Ngonn B
Spmin G
Speiil
Spontt L
spenn P
Spent
Syentt X
Spents S
Spoul Torms
Thomas
Torus
vallise Et Nino
Worl Sun
yo Wong
Wine 2
Zhon Clen
44


Reviews:
[{"review_id": "AQo14lanN0", "reviewer": "Reviewer_beYK", "summary": "The paper introduces DynaMix a new type of zero-shot forecasting / dynamical system reconstruction approach.\nDynaMix uses a mixture of expert approach, where each expert is a multi-dimensional almost linear RNN model. \nGiven a contex window of a time series signal, an attention mechanism is used to gate the experts. The model is trained across a wide set of dynamical systems to reproduce dynamical system behavior.\n\nThe paper demonstrate in simulations that the model can forecast and re-construct the statistical properties of many dynamical systems and in particular for chaotic dynamics -- a setting where other foundational time series forecasting models fail. The paper also shows that DynaMix can give reasonable forecast accuracy on real world time series forecasting benchmarks, despite never being trained on such datasets.", "questions": "- Is there any kind of pre-processing of the time series done? For example scaling, normalization, or time scaling?\n- The model seems to be able to match the overall period of a signal well, but it essentially only does a local re-construction of the phase space dynamics. Do you have any insights on how the model is able to match the overall period? It would be interesting to see how the model predicts a simple limit cycle as you change the period.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Very novel and innovative approach\n- Strong numerical results for dynamical system and real world time series\n- Well written and easy to follow.\n- Comprehensive details about model setup, training procedure, parameter selection etc.\n\nWeaknesses:\n- On the forecasting task, MAE is not the best metric when evaluating accuracy across time series with different magnitude, because the metric can be dominated by some few time series in the dataset with larger magnitude. It would be better to use MASE or normalize MAE per metric and report geometric averages.\n- Somewhat complex training setup relying on not so standard methods (STF). But everything is well documented and the code is available, which mitigates this to some extend.", "was_revised": true, "final_justification": "The authors have addressed my open questions.", "rebuttal": "We thank the referee for the thorough reading, and the strong and very encouraging support of our work!\n\n**Weaknesses**\n\n**W1 (MASE instead of MAE):** Thanks for raising this point. We had standardized the time series for these comparisons, but in any case also computed the MASE now as suggested, with results reported in Table R1 below. As can be seen, using MASE, DynaMix even comes out top of the list, surpassing the best Chronos model even w.r.t. forecasting. \n\nTable R1: Forecast performance, same as Fig. 4C in paper, but for MASE (sorted from best to worst)\n| Rank | Model | MASE |\n| ------ | ------- | -------------- |\n| 1 | DynaMix | 0.36 ± 0.17 |\n| 2 | Chronos-t5-large | 0.37 ± 0.20 |\n| 3 | Chronos-t5-small | 0.42 ± 0.16 |\n| 4 | Chronos-t5-base | 0.42 ± 0.20 |\n| 5 | Chronos-t5-mini | 0.61 ± 0.26 |\n| 6 | Chronos-t5-tiny | 0.90 ± 0.47 |\n| 7 | Chronos-bolt-base | 1.46 ± 0.81 |\n| 8 | Chronos-bolt-small | 1.78 ± 0.84 |\n| 9 | Chronos-bolt-mini | 1.79 ± 0.85 |\n| 10 | Mamba4Cast | 2.00 ± 0.90 |\n| 11 | Chronos-bolt-tiny | 2.18 ± 0.80 |\n| 12 | TTM | 3.04 ± 1.25 |\n\n**W2 (complexity of training):** Yes the training is non-standard, but on the one hand, training techniques like STF are indeed important for achieving good dynamical systems reconstruction results, see [1,2] and Fig. 24. And on the other, our overall architecture is instead somewhat simpler we would argue than typical time series FMs.\n\n**Questions**\n\n**Q1)** Yes, all time series were standardized dimension-wise, see sect. 3.2 and A.2, for all models. We will make this more explicit in the revision!\n\n**Q2)** We are not 100% sure we understood the question correctly, but please note that Fig. 3b already demonstrates the capability of the model to forecast into regions of state space not even covered by the context information (in this sense non-local). We confirmed this now with two further examples (Van-der-Pol oscillator and Rayleigh oscillator) which we will include in the revision. We also confirmed on one example (cyclic Rössler DS) that DynaMix can, for instance, correctly forecast a limit cycle with half or twice the period of that experienced in training (Dstsp={0.48, 1.72}, DH={0.008, 0.026}, MASE={0.15, 0.36}).\n\n[1] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.\n\n[2] Mikhaeil, J., et al. (2022). On the difficulty of learning chaotic dynamics with RNNs. Advances in Neural Information Processing Systems, 35, 11297–11312."}, {"review_id": "M2lGqeyrXV", "reviewer": "Reviewer_qUES", "summary": "The paper proposes DynaMix, a lightweight AL-RNN–based mixture-of-experts architecture. The proposed DynaMix architecture leverages a mixture-of-experts (MoE) framework employing almost-linear RNNs (AL-RNNs) as experts. In this framework, the next hidden state is predicted as a weighted sum of the outputs from multiple experts, with the weights determined by a gating network which computes these weights using the context and the current hidden state as inputs. By using lightweight experts and gating network, DynaMix is computationally efficient compared to recent transformer-based time series (TS) foundation models. Experimental results demonstrate that the proposed method performs well on dynamical system reconstruction (DSR) tasks and even on short-term forecasting without any fine-tuning.", "questions": "Q1. Could the authors provide quantitative comparisons with the state-of-the-art DSR models in term of both DSR task and time series forecasting?\n\nQ2. Could the authors conduct ablation studies to clarify which specific architectural choices, beyond the general MoE framework, are responsible for the observed performance gains?\n\nQ3. How does the model scale and generalize as the number of experts increases, and are there strategies in place to control model size or improve robustness to highly novel dynamical systems?\n\nQ4. The rationale for selecting AL-RNNs as experts within the MoE framework is not sufficiently justified. Could you provide a more detailed explanation to explain why this choice is effective or necessary? \n\nQ5. Can you compare the test performances of the proposed scheme, benchmark SOTA DSR models and time series foundation models, for data set with more complicated dynamics (e.g., 6D) in terms of term of both DSR task and time series forecasting?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. DynaMix can reconstruct and forecast previously unseen dynamical systems from only a short context without need for retraining or fine-tuning\n\nS2. The model achieves high performance with a lightweight architecture—requiring orders of magnitude fewer parameters and delivering much faster inference compared to existing foundation models, which makes it suitable for practical deployment at scale\n\nS3. The effectiveness of the proposed method is validated through various experiments on both DSR datasets and general time series datasets.\n\nWeaknesses:\n\nW1. The paper lacks quantitative comparisons with the state-of-the-art DSR models in term of both DSR tasks and time series forecasting, which limits the fairness and practical context of its benchmarking. \n\nW2..The Mixture-of-Experts framework is not itself a novel concept, and given that strong performance is reported in a broad sense, it remains unclear to what extent the authors’ specific architecture choices contribute to the observed gains, making the source of novelty somewhat ambiguous\n\nW3. The scalability and generalization of the model remain uncertain, as increasing the number of experts to cover more diverse or fundamentally different dynamical systems could lead to substantial growth in model size and potential degradation in out-of-distribution performance", "was_revised": true, "final_justification": "The authors have satisfactorily addressed most of the concerns and questions raised in the previous review. Therefore, I recommend accepting this paper.", "rebuttal": "We thank the referee for taking the time to thoroughly scrutinize our work, and for pointing out various omissions in terms of model evaluation, which we have now all addressed and provided in the form of result tables below (as figures are not allowed for the rebuttal this time).\n\n**Weaknesses**\n\n**W1 (further comparisons):** Good point. Our comparison group here were other foundation models (to which we now have further added TimesFM [5]: Dstsp: 12.50 ± 0.93, DH: 0.36 ± 0.11, MASE: 1.11 ± 0.63), as ‘standard’ DSR models cannot perform zero-shot reconstruction and forecasting but need training on at least segments of the new data. However, we fully agree that such a comparison would provide practical context, and have now compared DynaMix’ zero-shot performance with that of three SOTA DSR models explicitly trained on the data, namely the AL-RNN trained with STF as in [1], reservoir computers (RC) for DSR [2], and neural ODEs (NODE) for DSR as in [3,4]. As shown in Table R2 below, the *zero-shot generalization* performance of DynaMix in terms of both long-term statistics and forecasting is completely within the ballpark of custom-trained SOTA DSR models, *even without any fine-tuning or retraining* (statistically largely indistinguishable across the 54 DS test set according to t-tests, and on average even slightly better on the empirical TS). \n\nTable R2: Performance for *custom trained* DSR models (ALRNN, RC, Neural ODE) across all 54 test set DS (as in Fig.4; median+/-MAD) and for the empirical time series (as in Fig. 8). Best in bold.\n| | DynaMix | | | ALRNN | | | RC | | | NODE | | |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Dataset | D_stsp | D_H | MASE | D_stsp | D_H | MASE | D_stsp | D_H | MASE | D_stsp | D_H | MASE |\n| DS testsets | 3.92 ± 1.47 | 0.19 ± 0.09 | 0.36 ± 0.17 | 3.65 ± 2.54 | **0.08 ± 0.06** | 0.27 ± 0.16 | 5.65 ± 4.77 | 0.23 ± 0.19 | 0.96 ± 0.50 | **3.51 ± 2.32** | 0.12 ± 0.07 | **0.23 ± 0.13** |\n| ETTh1 | 0.8 | **0.08** | 2.13 | **0.33** | 0.13 | **1.63** | 1.01 | 0.27 | 2.99 | 12.76 | 0.31 | 2.81 |\n| Traffic | 0.68 | 0.21 | **1.58** | **0.33** | **0.16** | 2.43 | 1.5 | 0.43 | 2.96 | NaN | 0.5 | 50.65 |\n| Cloud Requests | **0.17** | 0.14 | **1.05** | 0.52 | **0.13** | 4.62 | 3.99 | 0.31 | 2.4 | 11.98 | 0.3 | 6.51 |\n| Weather (Temp.) | **0.59** | **0.09** | **2.74** | 0.91 | 0.16 | 3.63 | 11.26 | 0.66 | 8.8 | 12.4 | 0.51 | 8.82 |\n| Weather (Press.) | 0.41 | 0.19 | 5.25 | **0.23** | 0.28 | 5.6 | 0.26 | 0.29 | 5.95 | 11.42 | **0.16** | **4.9** |\n| Human fMRI | 0.63 | 0.09 | **1.91** | 0.61 | **0.08** | 2.47 | **0.15** | 0.17 | 3.22 | 7.81 | 0.35 | 2.77 |\n| Human EEG | **0.78** | 0.23 | 8.2 | 2.33 | 0.54 | **3.29** | 6.55 | 0.2 | 4.8 | 6.26 | **0.16** | 5.04 |\n| Avg. real-world TS | **0.58 ± 0.22** | **0.14 ± 0.06** | **3.26 ± 2.57** | 0.75 ± 0.73 | 0.21 ± 0.15 | 3.38 ± 1.37 | 3.53 ± 4.11 | 0.33 ± 0.16 | 4.45 ± 2.29 | 10.44 ± 2.72 | 0.33 ± 0.14 | 11.64 ± 17.32 |\n\n**W2 (ablation studies):** Yes, the MoE concept itself is not novel of course, but what is very novel in our minds is the whole model design and training for DSR, making it to our knowledge the *first* foundation model with zero-shot DSR capabilities, incl. long-term properties. \nWe, however, fully agree with the referee that some more insight into which model components are crucial for its performance is useful, and have now performed a series of ablation experiments as summarized in Table R3 below. While the STF-based training itself (see also Fig. 24 in Appx.), the attention mechanism, a sufficient number of experts, and a sufficiently diverse training set with different chaos typologies, are all crucial components, the precise form of the context mapping (whether CNN or linear layer) and the exact expert model *within the class of PLRNNs* [4] has less of an influence on the overall architecture’s success (using as experts LSTMs, vanilla RNNs or reservoir computers instead of any of the PLRNN-variants, however, led to decay in performance). Hence, it is really the overall design and training procedure which make the framework so strong. On the side, we discovered that we can improve DynaMix’ performance even further by including noise into the ALRNN experts during training (see line “Expert: probabilistic ALRNN”). \n\nTable R3: Ablation experiments \\& different expert models: Mean across 5 different training runs of the median performance across the 54 test sets.\n| Component  | D_stsp | D_H | MASE |\n| --- | --- | --- | --- |\n| full model | 3.78 ± 0.25 | 0.17 ± 0.01 | 0.36 ± 0.04 |\n| no MLP | 4.02 ± 0.54 | 0.17 ± 0.02 | 0.37 ± 0.04 |\n| no CNN | 10.41 ± 0.60 | 0.47 ± 0.02 | 3.34 ± 0.24 |\n| no Attention | 11.63 ± 1.18 | 0.52 ± 0.04 | 3.37 ± 0.22 |\n| CNN to linear layer | 5.65 ± 0.22 | 0.22 ± 0.01 | 0.57 ± 0.06 |\n| no STF | 13.66 ± 0.14 | 0.49 ± 0.03 | 3.02 ± 0.41 |\n| #experts<5| 13.82 ± 0.14 | 0.49 ± 0.06 | 3.50 ± 0.09 |\n| Restricted training set: Lorenz-like DS only | 11.82 ± 0.75 | 0.55 ± 0.02 | 3.34 ± 0.25 |\n| Expert: clipped shallow PLRNN | 3.78 ± 0.26 | 0.16 ± 0.01 | 0.34 ± 0.03 |\n| Expert: probabilistic ALRNN | 3.07 ± 0.10 | 0.16 ± 0.01 | 0.40 ± 0.05 |\n| Expert: vanilla RNN | 5.72 ± 0.42 | 0.22 ± 0.01 | 0.61 ± 0.03 |\n| Expert: LSTM | 8.03 ± 4.31 | 0.32 ± 0.16 | 1.61 ± 1.64 |\n| Expert: Reservoir Computer | 13.93 ± 0.10 | 0.54 ± 0.01 | 3.86 ± 0.18 |\n\n**W3 (scalability):** We now explicitly tested model scaling with the number of AL-RNN experts. As can be appreciated from Table R4 below, a minimum number of experts ($\\geq5$) is necessary to achieve optimal performance, while after that performance rather plateaus. At the same time, the computational load (training times) scales only *linearly* with the number of experts $J$ (tested for $J=10:10:80, R^2=0.97$; note we are not allowed to provide figures this time, from which this may have been clearer; regarding inference times, these are all in the split-second range anyway). \n\nRegarding diversity, we would like to point out that in sect. 4.4 we did apply DynaMix to a very diverse range of empirical settings with dynamics fundamentally different from the training set. This, in our minds, demonstrates its generalization capabilities with regards to fundamentally different dynamical systems.\n\nTable R4: Dependence of model performance on \\# of experts ($J$): Mean across 5 different training runs of the median performance across the 54 DS test sets.\n| J | D_stsp | D_H | MASE |\n| --- | --- | --- | --- |\n| 1 | 13.82 ± 0.14 | 0.49 ± 0.06 | 3.50 ± 0.09 |\n| 3 | 12.81 ± 0.18 | 0.50 ± 0.02 | 2.19 ± 0.17 |\n| 5 | 3.91 ± 0.48 | 0.16 ± 0.02 | 0.48 ± 0.07 |\n| 10 | 4.17 ± 0.36 | 0.19 ± 0.01 | 0.38 ± 0.03 |\n| 20 | 4.86 ± 0.70 | 0.22 ± 0.02 | 0.44 ± 0.06 |\n| 30 | 5.09 ± 1.26 | 0.23 ± 0.06 | 0.48 ± 0.10 |\n\n**Questions**\n\n**Q1)** We have now done so, see response to W1 and Table R2 above.\n\n**Q2)** Yes, now done, please see our response to W2 and Table R3 above.\n\n**Q3)** We think that the applications of our model to the diverse set of empirical data in Fig. 8 and Table 1 of the paper already constitute test cases with highly novel and diverse dynamics, since no such data were part of the training set! We now further included a high-d hyper-chaotic example, see response to Q5 below. Finally, as our response to W3 and Table R4 above indicates, our system is very well behaved and highly robust with regards to scaling up model size. \n\n**Q4)** Yes, fair question. The original motivation was that the AL-RNN is a recent SOTA model proposed specifically for DSR, which combines a couple of favorable features from a dynamical systems perspective, like direct translation into symbolic dynamics and providing topologically minimally forms [1] (on which we will comment in our revision). As furthermore shown in Table R3 (see reply to W2 above), however, AL-RNNs also perform substantially better in this context than other typical RNN choices like LSTMs or reservoir computers often used for DSR.\n\n**Q5)** In our minds the empirical test case examples (sect. 4.4) already cover systems with very complicated and much higher dimensional underlying dynamics, human EEG for instance, or the air pressure and cloud request data. We now have further included a 6d Lorenz-96 system in its hyper-chaotic regime and compared the DSR and forecasting performance of all models in Table R5 below, verifying that the strong performance of DynaMix also holds for complex hyper-chaotic systems. \n\nTable R5: Reconstruction & forecast results on 6d Lorenz-96 across all foundation models\n| Model | D_stsp | D_H | MASE |\n|-------|--------|-----|-----|\n| DynaMix | 3.67 | 0.10 | 0.98 |\n| Chronos-t5-base | 12.08 | 0.34 | 1.59 |\n| Chronos-bolt-base | 13.55 | 0.69 | 2.07 |\n| Chronos-bolt-small | 13.93 | 0.63 | 2.48 |\n| Chronos-t5-tiny | 14.52 | 0.38 | 2.62 |\n| Chronos-t5-mini | 14.51 | 0.42 | 2.67 |\n| Chronos-bolt-mini | 14.07 | 0.61 | 2.70 |\n| Chronos-bolt-tiny | 13.11 | 0.62 | 2.71 |\n| Chronos-t5-small | 12.12 | 0.38 | 2.91 |\n| TTM | 14.65 | 0.37 | 3.52 |\n| Mamba4Cast | 13.59 | 0.70 | 3.55 |\n\n**References**\n\n[1] Brenner, M., et al. (2024). Almost-linear RNNs yield highly interpretable symbolic codes in dynamical systems reconstruction. Advances in Neural Information Processing Systems, 37, 36829–36868.\n\n[2] Patel, D., & Ott, E. (2023). Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(2).\n\n[3] Chen, R. T., et al. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31.\n\n[4] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.\n\n[5] Das, A., et al. (2024, July). A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning."}, {"review_id": "SltNdKx6Ta", "reviewer": "Reviewer_gq5P", "summary": "This paper proposes DynaMix, a foundation model for dynamical systems. The model is a mixture-of-experts based on an RNN and trained via sparse teacher forcing. The authors perform an extensive empirical evaluation of their method across a range of dynamical systems and time series forecasting tasks. DynaMix exhibits strong predictive performance while simultaneously requiring less inference time than existing well-known time series foundation models.", "questions": "- From my understanding, the context dimension $N$ is fixed and shared across tasks during model training. L153-165 describe techniques for applying the model to signals of smaller dimension. Is it possible to apply the model to signals of higher dimension, or is this a limitation of the model?\n- Is the model applicable in settings where your signal is not observed uniformly in time?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- The proposed architecture is clearly motivated and seems well-suited for the task at hand\n- The empirical evaluation in the paper is thorough and convincing. The proposed model architecture exhibits clear gains compared against well-known time series foundation models in terms of zero-shot DSR (Fig 4) and time series forecasting (Tab 1) while being significantly faster (Fig 5).\n- I especially appreciate that the authors clearly demonstrate various short-comings of existing methods (Section 4.2) rather than only reporting metrics. This provides significant insight. \n- Section 4.3, on MoE similarity, is an interesting qualitative evaluation of the model.\n- The paper is generally very clearly written and well-presented throughout\n\n# Weaknesses\n\n- The paper could potentially be strengthened by additional ablations. It wasn't clear to me which components of the model (the AL-RNN, the MoE, the specifics of the training setup, etc.) were most important \n- It also would have been interesting to see how the proposed method compares to DSR methods which are not based on foundation models -- while these certainly have drawbacks (e.g., requiring task-specific training or fine-tuning), it is unclear if this would result in better performance.\n\n# Minor Comments\n- I found Section 3.2 a bit dense; while things are explained in the appendix, it would be ideal to provide more details in the main paper given that this seems to be an integral part of the method\n- The curves in Figure 8 are difficult to see/distinguish from one another", "was_revised": true, "final_justification": "I think this is a solid, high-quality paper. The paper itself is very clear and insightful, and the proposed methodology seems highly performant. The authors clearly demonstrate the reasons for the failures of existing methods which provides a good deal of insight.\n\nThe reason I am not voting for a higher score of (6) is that, while this is a strong paper, I don't think that it quite constitutes \"groundbreaking impact\", as the methodology is largely within the existing paradigm.", "rebuttal": "We thank the referee for their thorough assessment of our work, and the very supportive and encouraging feedback!\n\n**Weaknesses**\n\n**W1 (ablations):** Thank you for bringing this up, we have now performed a series of ablation experiments, summarized in Table R3 below. While the STF-based training itself (see also Fig. 24 in Appx.), the attention mechanism, a sufficient number of experts ($\\geq 5$), and a sufficient diversity of the training set with different chaos typologies, are all crucial components, the precise form of the context mapping (whether CNN or linear layer) and the exact expert model *within the class of PLRNNs* has less of an influence on the overall architecture’s success. Using as experts LSTMs, vanilla RNNs or reservoir computers (RCs) instead of any of the PLRNN-variants, however, led to decay in performance. On the side, we discovered that we can improve DynaMix’ performance even further by including noise into the AL-RNN experts during training (see line “Expert: probabilistic ALRNN”). \n\nTable R3: Ablation experiments \\& different expert models: Mean across 5 different training runs of the median performance across the 54 DS test sets.\n| Component  | D_stsp | D_H | MASE |\n| --- | --- | --- | --- |\n| full model | 3.78 ± 0.25 | 0.17 ± 0.01 | 0.36 ± 0.04 |\n| no MLP | 4.02 ± 0.54 | 0.17 ± 0.02 | 0.37 ± 0.04 |\n| no CNN | 10.41 ± 0.60 | 0.47 ± 0.02 | 3.34 ± 0.24 |\n| no Attention | 11.63 ± 1.18 | 0.52 ± 0.04 | 3.37 ± 0.22 |\n| CNN to linear layer | 5.65 ± 0.22 | 0.22 ± 0.01 | 0.57 ± 0.06 |\n| no STF | 13.66 ± 0.14 | 0.49 ± 0.03 | 3.02 ± 0.41 |\n| #experts<5| 13.82 ± 0.14 | 0.49 ± 0.06 | 3.50 ± 0.09 |\n| Restricted training set: Lorenz-like DS only | 11.82 ± 0.75 | 0.55 ± 0.02 | 3.34 ± 0.25 |\n| Expert: clipped shallow PLRNN | 3.78 ± 0.26 | 0.16 ± 0.01 | 0.34 ± 0.03 |\n| Expert: probabilistic ALRNN | 3.07 ± 0.10 | 0.16 ± 0.01 | 0.40 ± 0.05 |\n| Expert: vanilla RNN | 5.72 ± 0.42 | 0.22 ± 0.01 | 0.61 ± 0.03 |\n| Expert: LSTM | 8.03 ± 4.31 | 0.32 ± 0.16 | 1.61 ± 1.64 |\n| Expert: Reservoir Computer | 13.93 ± 0.10 | 0.54 ± 0.01 | 3.86 ± 0.18 |\n\n**W2 (comparison to non-FMs):** Very good point. To put our model’s zero-shot performance into perspective, we have now trained various DSR models (AL-RNN trained by STF [1], reservoir computers for DSR [2], and neural ODEs for DSR [3,4]) on the same data used in the paper, with results shown below in Table R2. As evident from this table, DynaMix’ zero-shot *generalization* performance is completely on-par with that of SOTA DSR models specifically trained on these data, even *without any fine-tuning or retraining* (performance across the 54 DS test sets was statistically largely indistinguishable according to t-tests, while performance on the empirical TS was on average even slightly better).\n\nTable R2: Performance for *custom trained* DSR (ALRNN, RC, Neural ODE) models across all 54 test set DS (as in Fig.4; median+/-MAD) and for the empirical time series (as in Fig. 8). Best in bold.\n| | DynaMix | | | ALRNN | | | RC | | | NODE | | |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Dataset | D_stsp | D_H | MASE | D_stsp | D_H | MASE | D_stsp | D_H | MASE | D_stsp | D_H | MASE |\n| DS testsets | 3.92 ± 1.47 | 0.19 ± 0.09 | 0.36 ± 0.17 | 3.65 ± 2.54 | **0.08 ± 0.06** | 0.27 ± 0.16 | 5.65 ± 4.77 | 0.23 ± 0.19 | 0.96 ± 0.50 | **3.51 ± 2.32** | 0.12 ± 0.07 | **0.23 ± 0.13** |\n| ETTh1 | 0.8 | **0.08** | 2.13 | **0.33** | 0.13 | **1.63** | 1.01 | 0.27 | 2.99 | 12.76 | 0.31 | 2.81 |\n| Traffic | 0.68 | 0.21 | **1.58** | **0.33** | **0.16** | 2.43 | 1.5 | 0.43 | 2.96 | NaN | 0.5 | 50.65 |\n| Cloud Requests | **0.17** | 0.14 | **1.05** | 0.52 | **0.13** | 4.62 | 3.99 | 0.31 | 2.4 | 11.98 | 0.3 | 6.51 |\n| Weather (Temp.) | **0.59** | **0.09** | **2.74** | 0.91 | 0.16 | 3.63 | 11.26 | 0.66 | 8.8 | 12.4 | 0.51 | 8.82 |\n| Weather (Press.) | 0.41 | 0.19 | 5.25 | **0.23** | 0.28 | 5.6 | 0.26 | 0.29 | 5.95 | 11.42 | **0.16** | **4.9** |\n| Human fMRI | 0.63 | 0.09 | **1.91** | 0.61 | **0.08** | 2.47 | **0.15** | 0.17 | 3.22 | 7.81 | 0.35 | 2.77 |\n| Human EEG | **0.78** | 0.23 | 8.2 | 2.33 | 0.54 | **3.29** | 6.55 | 0.2 | 4.8 | 6.26 | **0.16** | 5.04 |\n| Avg. real-world TS | **0.58 ± 0.22** | **0.14 ± 0.06** | **3.26 ± 2.57** | 0.75 ± 0.73 | 0.21 ± 0.15 | 3.38 ± 1.37 | 3.53 ± 4.11 | 0.33 ± 0.16 | 4.45 ± 2.29 | 10.44 ± 2.72 | 0.33 ± 0.14 | 11.64 ± 17.32 |\n\n**Minor**\n\n**M1)** Agreed, this sect. is rather short, and we will move more details from Appx. A.2 back to the main text in the revision.\n\n**M2)** Yes, fair point. One solution here is to only show a subset of the empirical data in the main text (moving the others to the Appx.), and using the space instead to show predictions from each foundation model in separate panels (or, alternatively, to focus only on the comparison between DynaMix and its strongest competitor in each panel). \n\n**Questions**\n\n**Q1)** Yes, this is possible. First, please note that the empirical examples (incl. human EEG or the air pressure data) are inherently likely of much higher dimensionality, yet DynaMix is still able to reconstruct and forecast these systems well. Second, we have now included another, 6d DSR benchmark (Lorenz-96), see results in Table R5 below. \n\nTable R5: Reconstruction & forecast results on 6d Lorenz-96 across all foundation models\n| Model | D_stsp | D_H | MASE |\n|-------|--------|-----|-----|\n| DynaMix | 3.67 | 0.10 | 0.98 |\n| Chronos-t5-base | 12.08 | 0.34 | 1.59 |\n| Chronos-bolt-base | 13.55 | 0.69 | 2.07 |\n| Chronos-bolt-small | 13.93 | 0.63 | 2.48 |\n| Chronos-t5-tiny | 14.52 | 0.38 | 2.62 |\n| Chronos-t5-mini | 14.51 | 0.42 | 2.67 |\n| Chronos-bolt-mini | 14.07 | 0.61 | 2.70 |\n| Chronos-bolt-tiny | 13.11 | 0.62 | 2.71 |\n| Chronos-t5-small | 12.12 | 0.38 | 2.91 |\n| TTM | 14.65 | 0.37 | 3.52 |\n| Mamba4Cast | 13.59 | 0.70 | 3.55 |\n\n**Q2)** As it currently stands, when observations arrive at non-equal temporal intervals, this would need proper preprocessing (like interpolation) for *all* the models reported in the paper. So this is currently a limitation that we will add to our Limitations sect..  \n\n**References**\n\n[1] Brenner, M., et al. (2024). Almost-linear RNNs yield highly interpretable symbolic codes in dynamical systems reconstruction. Advances in Neural Information Processing Systems, 37, 36829–36868.\n\n[2] Patel, D., & Ott, E. (2023). Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(2).\n\n[3] Chen, R. T., et al. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31.\n\n[4] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR."}, {"review_id": "EYRmVnF5li", "reviewer": "Reviewer_H6rs", "summary": "This paper presents DynaMix, a foundation model for dynamical systems based on a mixture of experts with ALRNN models.\n\nThe model is pretrained on a large amount of dynamical systems (synthetic 3d data from chaotic systems), and can be used for zero-shot reconstruction of unseen dynamical systems.\n\nDynaMix is tested on both simulated and real-world datasets, and greatly outperforms existing time-series foundation models while also being very computationally efficient.", "questions": "See questions in the weaknesses section above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "__STRENGTHS__\n\n1. Novelty: The paper introduces a new class of foundation model for time series, specifically designed for dynamical systems reconstruction, which is a direction not previously explored to this extent as far as I know.\n\n2. Performance: DynaMix demonstrates superior long-term forecasting performance compared to state-of-the-art time series foundation models, with orders of magnitude faster inference.\n\n3. Motivation: The model is grounded on the hypothesis that many real-world time series exhibit local dynamics reflective of underlying dynamical systems.\n\n4. Clarity on Limitations: the paper clearly acknowledges that the model is not suited for more complex time series, setting realistic expectations.\n\n5. Thorough Literature Review: The paper provides a comprehensive overview of existing approaches in both time series modeling and dynamical systems.\n\n\n__WEAKNESSESS__\n\n\nThe paper lacks few analyses/clarifications to better understand the role of different components of the model\n\n1. Why did you choose the ALRNN? How does the model perform with other RNN architectures like LSTM/GRU?\n2. How does the model perform without the temporal delay embeddings? Can SOTA models be improved with similar ideas?\n\nFinally, the model is not suited for more complex time series that stationary ones, but this is a limitation that is acknowledged by the authors and I still find the paper valuable as is.", "was_revised": true, "final_justification": "This is a very solid paper, which is novel, well written and motivated. I believe could have a great impact in the dynamical systems community.", "rebuttal": "We thank the referee for the careful reading and very supportive feedback on our paper, and for raising a couple of great points which we now have addressed with further results (provided in tabular rather than graphical form, as new figures are not allowed this time).\n\n**Weaknesses**\n\n**W1 (further analyses/ other RNNs):** Fair question. We had initially chosen the AL-RNN as a recent SOTA model proposed specifically for DSR, which combines a couple of favorable features from a dynamical systems perspective, like direct translation into symbolic dynamics and providing topologically minimally forms [1] (which we are happy to comment on in our revision). However, we have now also tested other RNNs, like different variants of PLRNNs [2], LSTMs as suggested, vanilla RNNs, and reservoir computers which are also a popular choice in the DSR context [3]. As can be seen in Table R3 below, while AL-RNNs are on par with other PLRNN variants, other types of RNNs (vanilla, LSTM, RC) perform significantly worse, further supporting our specific choice. On the side, we discovered that we can improve DynaMix’ performance even further by including noise into the AL-RNN experts during training (see line “Expert: probabilistic ALRNN”). \n\nMore generally, we also performed a series of other ablation studies to clarify the role of different model components, also included in Table R3 below. While the STF-based training itself (see also Fig. 24 in Appx.), the attention mechanism, a sufficient number of experts ($\\geq 5$), and a sufficient diversity of the training set with different chaos typologies, are all crucial components, the precise form of the context mapping (whether CNN or linear layer) and the exact expert model *within the class of PLRNNs* has less of an influence on the overall architecture’s success. \n\nTable R3: Ablation experiments \\& different expert models: Mean across 5 different training runs of the median performance across the 54 DS test sets.\n| Component  | D_stsp | D_H | MASE |\n| --- | --- | --- | --- |\n| full model | 3.78 ± 0.25 | 0.17 ± 0.01 | 0.36 ± 0.04 |\n| no MLP | 4.02 ± 0.54 | 0.17 ± 0.02 | 0.37 ± 0.04 |\n| no CNN | 10.41 ± 0.60 | 0.47 ± 0.02 | 3.34 ± 0.24 |\n| no Attention | 11.63 ± 1.18 | 0.52 ± 0.04 | 3.37 ± 0.22 |\n| CNN to linear layer | 5.65 ± 0.22 | 0.22 ± 0.01 | 0.57 ± 0.06 |\n| no STF | 13.66 ± 0.14 | 0.49 ± 0.03 | 3.02 ± 0.41 |\n| #experts<5| 13.82 ± 0.14 | 0.49 ± 0.06 | 3.50 ± 0.09 |\n| Restricted training set: Lorenz-like DS only | 11.82 ± 0.75 | 0.55 ± 0.02 | 3.34 ± 0.25 |\n| Expert: clipped shallow PLRNN | 3.78 ± 0.26 | 0.16 ± 0.01 | 0.34 ± 0.03 |\n| Expert: probabilistic ALRNN | 3.07 ± 0.10 | 0.16 ± 0.01 | 0.40 ± 0.05 |\n| Expert: vanilla RNN | 5.72 ± 0.42 | 0.22 ± 0.01 | 0.61 ± 0.03 |\n| Expert: LSTM | 8.03 ± 4.31 | 0.32 ± 0.16 | 1.61 ± 1.64 |\n| Expert: Reservoir Computer | 13.93 ± 0.10 | 0.54 ± 0.01 | 3.86 ± 0.18 |\n\n**W2 (delay embedding):** Some form of embedding is indeed necessary to achieve strong DSR results (cf. sect. 3.3), but this does not necessarily need to be a delay embedding, but could also be a positional encoding (eq. 6) as similarly employed in Transformer-based models like the different versions of Chronos. Indeed, for all empirical examples in Fig. 8 we had used positional encoding for DynaMix, i.e. DynaMix outperformed the other FMs in terms of DSR (and often forecasting) even though all models received the same types of inputs (scalar time series and positional embedding). However, we have also tested TTM with delay embedding now (the only one of the FMs which actually allows for this type of multivariate embedding), hardly improving the results as provided below.\n\nTable R6: Reconstruction results for TTM with delay embedding\n| Model | D_stsp | D_H | MASE |\n|-------|--------|-----|-----|\n| original TTM | 13.30 ± 0.68 | 0.32 ± 0.07 | 3.04 ± 1.25 |\n| TTM with delay embedding | 13.03 ± 0.61 | 0.29 ± 0.06 | 3.94 ± 1.77 |\n\n**W3 (non-stationary TS):** Thank you. Non-stationary time series are indeed still an open issue, with which other models struggle as well. As a proof of concept, however, we have now designed a pre-processing pipeline, which (similar as in FEDformers [4]) first estimates trend components from the context signal and adds them back on for inference. We have tested this idea on the non-stationary Air Passengers TS benchmark (cf. [5]), and compare performance of DynaMix with this enhancement to the best competitor (Chronos-t5-base; see Fig. 4c \\& Table 1) in Table R7 below. While we are not allowed to provide figures in the rebuttal which would illustrate the approach and results more clearly, we will include these in our revision. This is just a proof-of-concept at this stage, but it illustrates the approach in principle works and could be extended and potentially optimized.\n\nTable R7: Forecasting results for *non-stationary* Air Passenger TS benchmark\n| Model | D_stsp | D_H | MASE |\n|-------|--------|-----|-----|\n| DynaMix | 2.25 | 0.04 | 0.54 |\n| Chronos-t5-base | 3.49 | 0.06 | 1.16 |\n\n**References**\n\n[1] Brenner, M., et al. (2024). Almost-linear RNNs yield highly interpretable symbolic codes in dynamical systems reconstruction. Advances in Neural Information Processing Systems, 37, 36829–36868.\n\n[2] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.\n\n[3] Patel, D., & Ott, E. (2023). Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(2).\n\n[4] Zhou et al. (2022) FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. Proceedings of the 39th International Conference on Machine Learning, PMLR 162:27268-27286\n\n[5] Ansari, A. F., et al. (2024). Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815."}]

Author rebuttal:

Meta-review: We thank the authors for their submission. 

This work present a mixture of experts-based architecture for forecasting dynamical systems.  The model is trained on a wide range of dynamical systems, and is shown to be effective for zero-shot reconstruction of unseen dynamical systems.  The reviewers agreed that the proposed approach is novel and well-suited for the stated task.  They also agreed that the paper is clearly written, well-motivated, with thorough empirical evaluations. 

Empirical results show that the architecture is efficient compared to baselines, while still achieving good performance.  Multiple reviewers initially called out the lack of ablation study as a weakness.  But a thorough back and forth with reviewer qUES and gq5P yielded more performance results as well as an informative ablation study.
Decision: Accept (poster)