Zero-shot forecasting of chaotic systems
Authors: Yuanzhao Zhang
William Gilpin
Venue: ICLR 2025 Poster (2025)
Keywords: chaos
nonlinear dynamics
forecasting
physics
scientific machine learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Large language models zero-shot forecast chaotic dynamics

Abstract: Time-series forecasting is a challenging problem that traditionally requires specialized models custom-trained for the specific task at hand. Recently, inspired by the success of large language models, foundation models pre-trained on vast amounts of time-series data from diverse domains have emerged as a promising candidate for general-purpose time-series forecasting. The defining characteristic of these foundation models is their ability to perform zero-shot learning, that is, forecasting a new system from limited context data without explicit re-training or fine-tuning. Here, we evaluate whether the zero-shot learning paradigm extends to the challenging task of forecasting chaotic systems. Across 135 distinct chaotic dynamical systems and $10^8$ timepoints, we find that foundation models produce competitive forecasts compared to custom-trained models (including NBEATS, TiDE, etc.), particularly when training data is limited. Interestingly, even after point forecasts fail, large foundation models are able to preserve the geometric and statistical properties of the chaotic attractors.
We attribute this success to foundation models' ability to perform in-context learning and identify context parroting as a simple mechanism used by these models to capture the long-term behavior of chaotic dynamical systems. Our results highlight the potential of foundation models as a tool for probing nonlinear and complex systems.

Published as a conference paper at ICLR 2025
ZERO-SHOT FORECASTING OF CHAOTIC SYSTEMS
Yuanzhao Zhang
William Gilpin"
Santa Fe Institute
Department of Physics
Santa Fe, NM, USA
University of Texas at Austin Austin, TX, USA
ABSTRACT
Time-series forecasting is challenging problem that traditionally requires spe- cialized models custom-trained for the specific task at hand. Recently, inspired by the success of large language models, foundation models pre-trained on vast amounts of time-series data from diverse domains have emerged as a promising candidate for general-purpose time-series forecasting. The defining characteristic of these foundation models is their ability to perform zero-shot learning, that is, forecasting a new system from limited context data without explicit re-training or fine-tuning. Here, we evaluate whether the zero-shot learning paradigm extends to the challenging task of forecasting chaotic systems. Across 135 distinct chaotic dynamical systems and 10 timepoints, find that foundation models produce competitive forecasts compared to custom- custom-trained models (including NBEATS, TiDE, etc.), particularly when training data is limited. Interestingly, even after point forecasts fail, large foundation models are able to preserve the geometric and statistical properties of the chaotic attractors. We attribute this success founda- tion models' ability to perform in-context learning and identify context parroting as simple mechanism used by these models to capture the long-term behavior of chaotic dynamical systems. Our results highlight the potential of foundation models as tool for probing nonlinear and complex systems.
INTRODUCTION
Classical paradigms in machine learning (ML) require the model to be trained on data specific to the intended task. For example, forecast the weather in Singapore, a model would need be trained on past weather data from Singapore. However, recent work in statistical learning has highlighted the power of generative pre-trained models, which use probabilistic approaches and vast amounts of training data to build foundation models that can excel at diverse tasks without the need for separate retraining. In time-series forecasting, this paradigm shift has ignited an intense race to build general- purpose pre-trained models that can make zero-shot forecasts for any time series (Oreshkin et al., 2021; Garza & Mergenthaler-Canseco, 2023; Rasul et al., 2023; Jin et al., 2023; Gruver et al., 2024; Dooley et al., 2024; Liu al., 2024b; Woo et al., 2024; Ansari et al., 2024; Goswami et al., 2024). Such models have seen some initial success in forecasting real-world time series (Liang et al., 2024) but they have not been systematically tested chaotic dynamical systems, especially in terms of their performance in long-term forecasting over an extended time horizon.
There are several reasons why such tests are interesting. First, to train foundation models for time series, the amount of high-quality time-series data needed is the single most crucial bottleneck. For this reason, significant percentage of -vvilalav time-series data has been used to train these models. It is thus difficult to verify that the test set is not contaminated by time series related to those in the training set. In contrast, as far as we know, no trajectories generated by classical chaotic systems (c.g., Lorenz equations) have been used to train foundation models. Thus, time series from chaotic systems constitute an independent test set that can be used quantify the generaliza- tion ability of foundation models. Second, chaotic dynamical systems have well-defined attractors that exhibit invariant statistical and geometric properties (fractal dimensions, Lyapunov exponents, power spectra, etc.). This allows us to quantify ML models' ability to capture the long-term behavior of the system even after point forecasts inevitably fail (Pathak et al., 2018; Hess et al., 2023). Such
"Correspondence 1 wwpppineeee echaos utexas.edu

---

Published as a conference paper at ICLR 2025
135 chaotic systems
different initial conditions
Measure performance
short-ton foott-term eccuracy
long-term invariant properties
Input context
Pre-trained LLM
A
wan' of Mo
Forecast
Baselines
Fully train
(sMAPE)
Correlation dimension
Valid prediction time (VPT)
Divergence
Figure 1: Chaos benchmark for zero-shot forecasting of time series. We use 135 distinet chaotic systems to generate chaotic trajectories from 20 different initial conditions each. Each tra- jectory is used train the baseline deep-learning models (NBEATS, TiDE, etc.) and also provided as context to the pre-trained LLM (we use Chronos, best-in-class foundation model for time se- ries). Both the trained baseline models and Chronos are then asked to predict the trajectory into the future. We measure the quality of the predictions in terms of both short-term accuracy and long-term attractor reconstruction. Across 10d distinct trajectories and 10 data points, we find that zero-shot forecasts can be competitive in both short-term predictions and in capturing the long-ten "climate" of the dynamics.
tests usually possible for general time series. Third, the past few years have seen growing activities at the interface of physics and MI (Yu & Wang, 2024; Levine & Tu, 2024; Gilpin, 2024), with the fertilzetiiinationn between ML and dynamical systems yielding advances in both directions (Weinan, 2017; Chen et al., 2018; Pathak et al., 2018; Li et al., 2020; Chen & Tao, 2021; Jordan al., 2021; Gauthier al., 2021: Levine & Stuart, 2022; Mikhaeil al., 2022: Krishnapriyan al., 2023; Yang et al., 2024). Benchmarking foundation models on chaotic systems introduces the pos- sibility of applying dynamical systems techniques (e.g., Takens embedding theorem (Huke, 2006)) to understand the inner workings of these models and the origin of their generalization abilities.
In this paper, we set out to perform the first systematic evaluation of the zero-shot learning paradigm in the context of forecasting chaotic systems. A schematic summarizing our benchmark pipeline is presented in Fig. I. We also show another schematic illustrating the difference between classical deep learning models and foundation models when making time series predictions (sce Fig. 7 in the appendix).
Our study is also of intrinsic interest to scientific machine learning (SciML) and nonlinear dynamics communities. So far, the data-driven modeling approaches developed in these communities (c.g., reservoir computing (Pathak al., 201 2018), PINN (Karniadakis al., 2021), SINDy (Brunton al., 2016), Koopman operators (Brunton et al., 2022), neural operators (Azizzadenesheli et al., 2024), etc.) follow classical train/test dichotomy. That is, to forecast the dynamics of the Lorenz os- cillator, SciML model is first trained on data generated by the Lorenz equations. The model learns chaotic dynamics by extracting the underlying vector field (or flow map) from time-series data during training. At first glance, seems ludicrous that a model can effectively forecast chaotic dynamical systems without first explicitly learning the flow. A convincing demonstration of the pos- sibility of zero-shot learning in a SciMI. context could lead to new forecasting tools and generate novel insights into chaotic systems.
From a theoretical standpoint, an emerging direction in SciML is to understand the out-of- distribution generalization ability different data-driven modeling frameworks (Wang et al., 2020; Kong al., 2021; 2023; Göring et al., 2024). This parallels a long line of research that investigates the generalization ability of neural networks (Neyshabur et al., 2018; Belkin et al., 2019; Baldassi et al., 2020; Xu et al., 2020; Feng & Tu, 2021; Nakkiran et al., 2021; Power et al., 2022; Liu et al., 2022c). For example, if a model was only trained on trajectories from limited number initial conditions, it effectively extrapolate the learned dynamics different part of the phase space
2

---

Published as a conference paper at ICLR 2025
and forecast from previously unseen initial condition (that is, far from any of the training initial conditions) (Zhang & Cornelius, 2023)? Foundation models that not only generalize to new initial conditions but also to new systems could introduce novel ideas and insights into this endeavor.
Our main contributions are:
1. A large-scale evaluation of the ability of time series foundation models model physical systems outside of their training domain.
2. Discovery that foundation models produce zero-shot forecasts competitive with models custom-trained forecast chaotic attractors, Moreover, larger foundation models produce better forecasts.
3. Observation of scaling of a foundation model's zero-shot prediction ability with context lengths far exceeding typical correlation timescales of chaos, indicating in-context learning of chaotic dynamics.
4. Observation that foundation models retain long-term statistical properties of chaotic attrac- tors, after pointwise predictions fail.
2 RELATED WORK
Several works train transformers perform long-horizon forecasting tasks (Li et al., 2019; Zhou et al., 2021; 2022; Liu et al., 2022b; Wen et al., 2022), obtaining leading results in long- long-horizon fore- casting. However, recent works question their consistency and utility compared to properly-tuned simpler models (Lara-Beníted et al., 2021; Zeng et al., 2023; Das et al., 2023; Tan et al., 2024). De- spite these debates, unique property of large models like transformers is zero-shot generalization, in which they learn to perform a novel task without training the model weights on task-specific data (Brown, 2020). The resulting in-context learning strongly differs from prior approaches to fore- casting chaotic systems, which focus on training the weights of models based on the past observed history of system (Pathak et al., 2018; Gauthier al., 2021; Vlachas al., 2020). In-context learn- ing has motivated the development of foundation models: large models pre-trained on vast amounts of data, which perform few-shot inference via prompting (Bommasani et al., 2021).
Several recent zero-shot forecasting models are modifications of large language models, which en- code time series as tokens (Xue & Salim, 2023; Ansari et al., 2024; Gruver et al., 2024; Miller et al., 2024; Liu et al., 2024b; Ekambaram et al., 2024). Several of these models have been shown to exhibit in-context learning at test time (Lu et al., 2024; Gao et al., 2024; Liang et al., 2024).
Foundation models have recently been introduced for other scientific machine-learning tasks (Miller et al., 2024). These include models for partial differential equations (Yang et al., 2023; Rahman et al., 2024: Subramanian et al., 2024; Herde et al., 2024; Takamoto et al., 2022), numerical integra- tion (Song et al., 2024), fluid flow prediction (Herde et al., 2024), molecular dynamics (Allen et al., 2024), weather forecasting (Nguyen al., 2023; Bodnar et al., 2024), material discovery (Takeda et al., 2023), astrophysics (Parker et al., 2024), and electrocardiogram (ECG) analysis (McKeen al., 2024). A recent study used an open-source language model evaluate zero-shot forecasting performance on stochastic dynamical systems (like Markov chains) as well as the Lorenz attractor (Liu et al., 2024a), finding evidence of a neural scaling law relating context length and prediction accuracy, consistent with prior works (Gruver et al., 2024; Jin et al., 2023). However, to the best of our knowledge, our work is the first large-scale evaluation of the zero-shot learning ability of foundation models on over 100 chaotic systems, both in terms of short-term forecast accuracy and long-term attractor reconstruction performance.
A MOTIVATING EXAMPLE
Here, we chose Chronos (Ansari et al., 2024) to represent pre-trained models because it has been shown to outperform earlier foundation models for time series, such as TimeGPT and Moirai (Garza & Mergenthaler Sergenthaler-Cansecoo, 2023; Woo et al., 2024). Chronos internally uses a large language model based the text-to-text T5 transformer model family (Raffel et al., 2020). It introduces scaling and quantization layer, which converts continuous -valued univariate time series into a set of discrete tokens, with vocabulary size acting model hyperparameter. The model was trained diverse
3

---

Published as a conference paper at ICLR 2025
time series spanning ~ 10" observations drawn from 42 synthetic and real-world settings, but the training data does not contain any dynamical systems. We evaluate five pre-trained variants of Chronos, denoted by the sizes of the underlying T5 architecture: 8M, 20M, 46M, 200M, and 710M parameters.
Figure 2 shows zero-shot forecasting of the Lorenz os- cillator (defined in the appendix), a well-studied chaotic dynamical system, using ChrOnos-200M. The only data available to Chronos are 512 data points that serve as the context for the prediction (gray in the Figure). Because Chronos is a univariate forecast model, we separately forecast each coordinate of the attractor and reset the model state between each forecast. Forecasting chaotic systems based on partial observations (e.g., only having access to the y coordinate of the Lorenz oscillator) is an extremely difficult task in nonlinear dynamics (Ratas & Pyragas, 2024). Despite this challenge, the prediction closely tracks the ground truth for over 3 Lyapunov times and, even after diverging from it due chaos, remains in the vicinity of the strange attractor.
Zero-shot prediction Ground Truth
Interestingly, although Chronos predicts x and y sepa- rately, it maintains a positive correlation between x and y so they have the same sign most of the time (which is nec- essary for accurately reconstructing the attractor). This suggests that Chronos internally models y when forecast- ing z and vice-versa. In nonlinear dynamics, this process possible due to Takens' theorem, which states that low- dimensional measurements can reveal unmeasured dy- namical variables using delay embedding (Huke, 2006).
sime
Zero-shot forecasts of chaotic systems. We use Chronos to predict the r(t) and y(t) components of the Lorenz oscillator. The zero-shot forecasts match remarkably well with the ground truth for both short-term pre- diction and long-term attractor recon- struction.
Figure 2:
However, the performance of Chronos, while impressive, can also be fragile. Keeping everything unchanged, sim- ply starting the context trajectory from different initial condition on the attractor can significantly degrade the accuracy of Chronos's 's prediction (see Figs. 5, 8 and 9). So how good is Chronos forecasting chaotic systems, truly? More generally, is zero-shot forecasting from foundation models a promising alternative to custom-trained models when it comes predicting chaotic systems? To answer these questions, we next perform systematic benchmarks that average over a diverse set of chaotic systems and different initial conditions.
METHODS
chaotic systems forecasting benchmark. The dysts dataset represents a standardized bench- mark of 135 low-dimensional chaotic systems, described by ordinary differential equations that have been aligned with respect to their dominant timescales and integration steps (Gilpin, 2021; 2023). Each system is annotated with its largest Lyapunov exponent an invariant property associated with every set of differential equations that quantifies the rate at which small errors accumulate. Systems that approach a periodic orbit or an equilibrium exhibit zero or negative Lyapunov exponents be- cause different initial conditions converge to the same state. In contrast, chaotic systems exhibit positive Lyapunov exponents, implying that small changes the initial conditions or the model parameters lead to trajectories that (at least initially) diverge exponentially. When modeling such systems, a small error will compound over a characteristic timescale, the Lyapunov time, T = --1, making highly-chaotic systems (those with small 7) difficult to forecast.
The dynamical attractor of each system in dysts also annotated with mathematical properties such as entropy fractal dimension. Here, in order to match the typical granularity of the real- world time series used to train Chronos, we re-integrate all systems using an implicit Runge-Kun Kutta integration scheme. We downsample the resulting time series to uniform coarse granularity of 30 timepoints per Lyapunov time 7. We find that our forecast results depend only weakly on the data granularity (Appendix).
4

---

Published as a conference paper at ICLR 2025
Baseline experiments. Our baseline experiment design matches prior works (Gilpin, 2021; 2023; Godahewa et al.; Schötz et al., 2024). For each of the 135 chaotic dynamical systems, 20 trajectories of length 812 are generated, each originating from random initial condition on the attractor. This produces a set of 2700 (135 x 20) multivariate time series, which have dimensionalities between 3 and depending on the particular dynamical system. All time series then split into training sets consisting of the first 512 points of each time series, with the last 300 timepoints set aside to determine final test scores. For experiments with varying context lengths, trajectories are extended backwards in time, so that the 300 test points remain the same.
For the baseline models, hyperparameter tuning performed separately for each of the 135 dynam- ical systems. For given dynamical system, each of the 20 training trajectories is divided into a true training set comprising the first 435 timepoints, and a validation set of the last. timepoints. For each set of hyperparameters, model is trained on the true training set and then evaluated on the vali- dation set. The validation scores are averaged over the 20 trajectories, and the hyperparameters from the best-performing model are selected. A model is then initialized with those hyperparameters, and it is trained on the full 512 timepoints. The model is then tasked with autonomously generating a forecast of the next 300 timepoints (around 10 Lyapunov times), which compared against the ground-trune trunt trajectories to generate overall model scores. The testing dataset is therefore causally disconnected from the training data at all times.
To match the design of Chronos, for multivariate dynamical systems, each baseline model is sep- arately trained and tested along each dimension, and the results averaged. This channel- independent forecasting task is intrinsically harder than providing full state information, because the models cannot leverage the mutual information between different dimensions (Ratas & Pyragas, 2024). However, recent works on large-scale forecast models actually obtain stronger results by isolating input channels, because the resulting model class is more expressive (Nie al., 2023). We thus do not expect Chronos's 's performance to improve if it were instead trained to produce multivari- ate forecasts (i.e., one which æ.y.= are jointly embedded and tokenized).
The experiments yield 2700 distinct forecasts of 300 timepoints each along 3 - 6 dimensions de- pending on the underlying chaotic system, all generated by separately-traine forecast models. Our large-scale experiments thus span 5.5 x 10" training points, 3.2 x 107 test points, and 3.2 x 10 generated forecasts across all models. The experiments require 10a walltime compute hours on an Nvidia A100 GPU.
Our baseline models include NBEATS (Oreshkin et al., 2019), a hicrarchical neural network model that has been shown to perform particularly well on dynamical systems forecasting tasks (Gilpin, 2021; 2023). TiDE (Das et al., 2023), a recent model that addresses several known computational limitations of Transformer class models forecasting time series. A next-generation reservoir computer (NVAR) (Gauthier et al., 2021), which has a strong inductive bias for learning dynamical systems and which has previously been found to perform well on chaotic systems (Gilpin, 2023). We also include a small encoder-decoder Transformer with 0.5M trainable parameters, as well as an LSTM (Vaswani al., 2017; Hochreiter, 1997).
In principle, the baseline models have a wide variety of additional hyperparameters available to tune, such as optimizer settings, reservoir or recurrent layer size, etc. Here, we focus on the lookback window, which is a common hyperparameter across all forecast models. It is also analogous to the context window in Chronos, for which we tune no other hyperparameters in the zero-shot setting.
Metrics. Following prior studies (Hyndman & Koehler, 2006; Makridakis et al., 2022; Gilpin, 2021; 2023), we use four metrics to evaluate forecast quality, including Symmetric Mean Absolute Percentage Error (sMAPE).
T X
100 T
X X1
SMAPE(X... x) =2 2
X
where X1, ..., Xy correspond to the true test values of a time series up maximum forecast horizon 7', and X1, X2,... XT are the predictions of a forecast model at those same timepoints.
Valid Prediction Time (VPT). The first forecast horizon at which the sMAPE exceeds fixed thresh- old e Aade&Vlaann (Vlachas et al., 2020).
VPT argmax, (ty (sMAPE(X,, &.) <,, VI t.).
5

---

Published as a conference paper at ICLR 2025
We set 30, in prior studies (Vlachas et al., 2020; Gilpin, 2023).
Correlation Dimension (dfac). For chaotic dynamical systems, the long-term distribution of ob- served data points approximates fractal object known as the strange attractor. Fractals have space- filling properties that are intermediate between integer dimensionalities, and every strange attractor has unique and invariant fractal dimension. The correlation dimension non-parametrically esti- mates the fractal dimension from a time series, by calculating the scaling of the number other attractor points that fall within a given radius of each point (Grassberger & Procaccia, 1983). We compute the correlation dimension using all data points from a model's 's forecasts and report the root mean square error between the inferred correlation dimension and the ground truth.
Kullback-Leibie Divergence between attractors (Detsp). We compute the KL Divergence between the original and reconstructed attractors, following previous works (Hess al., 2023; Göring al., 2024). To perform the computation, we center Gaussian distribution at each point from the true and reconstructed trajectories. We then use a sampling- -ased approach to estimate the KL Divergence between these Gaussian mixtures (Hershey & Olsen, 2007). This metric measures whether two attractors have matching distributions, and it largely agrees with the correlation dimension. We thus report the KL Divergence results in the Appendix.
5 RESULTS
5.1 ZERO-SHIOT MODELS ARE COMPETITIVE WITH FULLY-TRAINED MODELS IN SHORT-TERM ACCURACY.
To evaluate the effectiveness of zero-shot forecasting for chaotic systems, we evaluate the perfor- mance of Chronos and the baseline models on the dysts benchmark (Fig. 3). Across the 135 systems, the median VPT of the three largest zero-shot Chronos models is statistically indistin- guishable, while the smaller models exhibit significantly smaller VPT (p < 10 non-parametric Friedman test, N = 135). Scaling of performance with model size indicates that the larger models exhibit better generalization properties, because the chaotic systems dataset strongly differs from their training data. This finding supports the premise of the foundation model paradigm for chaotic systems, because it shows that the sheer scale of a domain-agnostic model, when matched with suf- ficient training, improves forecasts. Compared to the fully-trained baseline models, the three largest zero-shot forecast models outperform all except for NBEATS (Friedman, p < 10-3 N 135). While recurrent neural networks and next generation reservoir computers have previously shown promising forecast results for dynamical systems (Vlachas et al., 2020; Gilpin, 2021; Gauthier et al., 2021), they underperform zero-shot models in the data-limited setting investigated here. However, when given enough training data, it has been shown that these models can achieve longer prediction horizons (Gauthier et al., 2021; Gilpin, 2023; Pathak et al., 2018). In contrast, the context length of Chronos and other attention-baa based forecast models is limited, and they are most effective when data is scarce.
We emphasize that the similarity of the error curves in Fig. 3 does not arise from a lack of sensitivity in the forecast metrics. When the baseline models instead given full state information (multi- variate forecasting), the prediction task becomes easier, resulting in lower sMAPE and higher VPT across all systems (see Appendix). These results underscore that partial observability, which char- acterizes most practical forecasting tasks (Ratas & Pyragas, 2024), is quantifiably harder for current forecasting models. The zero-shot models perform nearly well as state-of-the-arr, fully-trained models in this setting, reaching a VPT as high as Lyapunov time.
Historically prediction time of Lyapunov timescale has been considered prohibitive even for fully-trained forecast models. This is because both observational and modeling error compound over this timescale (Palmer, 2000; Medio & Lines, 2001). Chronos's 's ability to consistently forecast up to Lyapunov time, without prior training on dynamical systems, suggests the advantages of its large-scale training on diverse time series. This scale allows it to extract generic predictive features from time series, which also prove to effectively represent nonlinear dynamics. A similar concept occurs in computer vision, in which convolutional neural networks tend to learn generic Gabor- like feature extractors in early convolutional layers (Zeiler & Fergus, 2014). The ability of Chronos to generate meaningful forecasts suggests that these learned nonlinear features, coupled with high di- mensionality both in the input feature space (context length) and internal model dynamics, mitigate

---

Published as a conference paper at ICLR 2025
100
Veldd Predcteo
Error (sMAPE)
4EM
203
whan
10
50
10-1
10P
10
10-1 10" 10'
Forecast Horizon (Lyapunov time)
Figure 3: Zero-shot models of chaotic systems competitive with custom-trained models. Zero-shot forecasts from Chronos for five different model sizes (left), compared to other forecast models directly trained the points given Chronos context (right). Inset plots show the valid prediction times (VPT), the first time each forecast exceeds an error limit. All error bars are over 135 chaotic systems, each with 20 distinct initial conditions.
the intrinsic chaoticity of the underlying systems. In dynamical systems theory, recent works Koopman operators show that appropriately-selettee nonlinear transformations make chaotic sys- tems appear more linear (and thus predictable) in higher-dimensions spaces (Mezić, 2013; Brunton et al., 2022). As Chronos contains tens of millions of internal weights, it has an intrinsic advan- tage due its scale, which counteracts its low inductive bias when compared to forecasting models specialized for dynamical systems, such as next-generation reservoir computers.
5.2 LARGE ZERO-SHOT MODELS EXCEL AT LONG-TERM ATTRACTOR RECONSTRUCTION
Next, we quantify Chronos and the baseline models' ability to capture the long-term behavior of chaotic systems after point forecasts inevitably fail. This corresponds global measure of fore- cast quality: how well does model capture the shape of the strange attractor and reproduce the statistics of major dynamic events, even if not necessarily their particular timing? In forecasting, this problem is known as predicting the climate, rather than the weather (Patel et al., 2021; Bram- burger & Fantuzzi, 2024).
A
B
NBEATS
Transfommer
Pozta Cimenston Kooutayy
base
mm
Transtomer
NAR
TIDE
US3M
0.4
0.2
Fully
tiny
mini
0.0
02
0.6
1.0
12
14
Vorld Predition Time
Figure 4: Zero-shot forecast models effectively capture attractor geometry. (A) Example fore- casts produced by the zero-shot and trained models, for 20 initial conditions from the Lorenz chaotic attractor. (B) The correlation between the fractal dimension of the predicted attractor and the true attractor (Spearman's rank-order coefficient, N 2420 points, p 10-3 for all cases), versus the VPT of the corresponding model. The red markers represent variants of Chronos with different model sizes: tiny (8M parameters), mini (20M)) small (46M)), base (200M), and large (710M). The blue markers represent the baseline models. Models closer to the top capture the attractor ge- ometry better and models closer to the right make accurate point forecasts for longer. Error bars are standard errors over 135 dynamical systems, each with 20 different initial conditions.

---

Published as a conference paper at ICLR 2025
Figure 4 shows the correlation dimension accuracy (long-term attractor reconstruction quality) against the VPT (short-term forecast quality) for different models. NBEATS performs the best in both metrics, likely because very high pointwise aceuracy necessarily leads to high global attractor quality. Generally, this trend holds across both zero-shot models and baseline models. However, within each model class a few surprises emerge: the fully-trained small Transformer, which pro- duced relatively weak forecasts, captures the attractor shape as accurately as the zero-shot models. This observation suggests that attention-based models, which process their entire context simultane- ously, have an innate advantage in capturing the long-term structure of attractors attractrs-mmrroring similar results for language models (Brown, 2020). Consistent with this interpretation, we observe weak at- tractor reconstruction accuracy from the LSTM and NVAR models, which both operate sequentially and downweight earlier parts of their context. To ensure that these results are not a consequence of our choice of metric, we also evaluated attractor reconstruction quality using the KI. Divergence between the true and forecasted attractors, and we found the same trends (see Appendix).
5.3 ZERO-SHOT FORECASTS PARROT MOTIFS FROM THEIR CONTEXT.
We next identify a simple mechanism for zero-shot forecasting. Because Chronos a generative time series model that learns conditional dependencies among points in its context, we directly quan- tify the similarity between the timepoints immediately preceding a forecast and previous intervals seen in the context. We use the highest-correlating subsequence of duration greater than 30 time- points (1 Lyapunov time in our units) as a measure of context overlap. We find that the zero-shot model's forecasts strongly correlate with this metric over all dynamical systems, and that this depen- dence is more pronounced than in the best rest-performing fully-trained model (Fig. 5). This suggests that much of Chronos's performance arises from its ability parrot context sequences, underscoring our earlier observation that Chronos primarily models conditional dependencies among timepoints.
In Appendix E, we further probe this effect by showing that zero-shot performance continuously degrades as nonstationarity is introduced into the time series. Nonstationarity represents distribu- tion shift for time series, and disrupts the effectiveness of context-parroting as a forecast strategy because the dynamical attractor continuously and irreversibly changes. In Appendix C, we also identify a weak correlation between forecast accuracy and the first forecast point's 's natural mea- sure density (the local density of points a dynamical system's 's attractor), underscoring how rarer dynamics challenge zero-shot predictions.
A
W MAA
B
AA Y
WMMI
Con with Context
PAMAN Pomn M
Chooos
NBEATS
Figure 5: Context parroting as a mechanism for zero-shot forecasting. (A) Better zero-shot forecasts often have initial stages that overlap with the context. The context overlap quantifies the similarity between the last 30 points of the context and the prior points. (B) Comparison of
context overlap of the zero-shot forecasts (Chronos base) with the best performing fully- fully-trained model (NBEATS). The zero-shot model correlates with context significantly more than the trained models across the chaotic systems dataset (matched t-test, N = 135, p < 10-3).
5.4 CHRRONOS PERFORMS EFFECTIVE IN-CONTEXT LEARNING EVEN WITH SHUFFFEE CONTEXT.
Chronos's 's forecasting performance stems from its ability to perform in-context learning, in which early context points on attractor act analogously prompts in language models (Brown, 2020: Li et al., 2023). This mechanism underlies our earlier observation that the model's 's generalization ability improves with its size. While early points in long context are decorrelated with the predictions,
8

---

Published as a conference paper at ICLR 2025
they drawn from the same underlying distribution, and we thus hypothesize that longer contexts provide information about the distribution of attractor states, as occurs in language models (Xie et al., 2022). We test this hypothesis by randomly shuffling all length-k sequences of successive timepoints in the model's context, and then repeating our zero-shot experiments as increases (Fig. 6A). For example, if the context X1, X2, X3, X4, then 1-gram shuffle would be X1, X4, X2, X3 while 2- gram shuffle would be X3, X4, X1, X2. We keep the last & context timepoints the same as the original training dataset, but we ensure that the penultimate & sequence differ from the unshuffled context. As baseline, we also directly perform zero-shot forecasts using only the last A context timepoints.
We find that the model's forecast accuracy increases with the context length, but that, for sufficiently long contexts, random shuffles provide better forecasts than shorter context baselines. Earlier context points thus provide statistical information about the distribution of single timepoint values, as well as conditional probabilities of certain pairs, triplets, et cetera (Xie et al., 2022). The ergodicity of chaotic attractors implies that they have well- stationary distribution of expected states p(x,), known the natural measure (Ott, 2002). Long contexts (even when shuffled), beyond the timescale over which the states of a system become decorrelated, facilitate in-context learning of this measure. Consistent with this observation, in Appendix E, show that non-stationary time series (in which this measure irreversibly deforms) generally lead to worse zero-shot forecasts. This process resembles the warm-up time in reservoir computers, type of recurrent neural network used for dynamical systems forecasting (Jaeger & Haas, 2004; Pathak et al., 2018). In this setting, extended context allows the reservoir to gradually synchronize with the dynamical system being learned (Lu & Bassett, 2020).
5.5 ZERO-SHOT FORECASTING IS COMPUTATIONALLY EFFICIENT COMPARED TO FULLY-TRAINING MODELS.
A
B
NBFATS
Chrons-innlr
NREATS-inler
VPT (Lyap. times]
yPI (Lyap. times]
10"
Time (s)
0.1
Long Context
Length Context
0.0
0
15
10?
K gram arder
Context Length
Context Length
Figure 6: Scaling laws with context length. (A) The forecast accuracy (VPT) of Chronos-base when given a context of length versus when given a full context (length 512) but with all k-grams shuffled. (B) Comparing Chronos-base zero-shot forecasts with NBEATS fully trained the same context. Both models are trained in a channel-indeeendeen manner (C) The single-node walltime for zero-shot forecasts (Chronos-baai base), compared to the training and inference costs of NBEATS (including hyperparameter tuning). All curves show medians and standard errors over 20 different initial conditions from each of 135 dynamical systems.
We next evaluate how variation in context length affects the performance of Chronos. We vary the context length of the base Chronos model between 5 and its maximum value of 512 and repeat our zero-shot forecast experiments. We also select the best cest-performing traditional model, NBEATS, and fully train it (including cooss-valiation to set the lookback window) over the same points given to Chronos context. We find that the VPT of Chronos increases monotonically with context length, even as the context reaches over Lyapunov times (Fig. 6B). This timescale extends well-beyond the ~1 Lyapunov timescale over which chaotic systems typically become decorrelated ((x(t)x(t + T))r = 0) (Shaw, 1981). This regime also exceeds the typical range of Takens' embedding theorem, Chronos's
because time series usually lifted using delay embeddings over timescales performance therefore arises from more than just featurization and extrapolation from recent points in the context.
We next consider the practical question of whether the foundation model caaudigm-pretraiin in domain-agnostic settings, and then specializing task-confers computational advantages over

---

Published as a conference paper at ICLR 2025
directly training a smaller model from scratch. We measure the walltime of training and inference on a single A100 GPU node. We find that the computational cost of Chronos can be favorable at long context lengths when compared to NBEATS (Fig. 6C).
The inference time of Chronos is bounded by the quadratic scaling of attention layers with the con- text length. This limitation motivates newer architectures like Mamba (for language) and TiDE (for time series), which exhibit linear scaling. However, despite the relatively slow inference at small context windows, we find that Chronos can be very efficient when working with long context, mak- ing it a viable choice for many practical applications. In terms of the prediction horizon, Chronos exhibits the same linear scaling of cost as auto-regressive models (RC, LSTM, NVAR, etc.).
6 CONCLUSION AND FUTURE DIRECTIONS
We have performed the first large-scale evaluation of zero-shot forecasting models on the classi- cal problem of predicting chaos. Our most striking finding is that a large pre-trained model can successfully forecast chaotic systems for up to one Lyapunov time, beyond the expected degree of predictability, even though it was not directly trained on chaotic systems. The resource re- quirements of inference-only zero-shot forecasting are negligible compared to fully training deep- learning models such NBEATS, particularly when long context lengths and lookback windows are used. Moreover, zero-shot models perform well without hyperparameter tuning. All in all, the success of Chronos indicates that many aspects of chaotic dynamics can be captured by generic high-dimensional transformations, suggesting that the internal representations used by Chronos to learn dynamical systems may provide insight into other time series tasks, such as system identifi- cation or bifurcation detection. It also supports the hypothesis that there is a common "language" for time series-uuiversal features and structures shared by time series across different domains that make transfer learning possible.
On a conceptual level, unpredictability in chaotic systems arises from the rapid growth of the gap between the true trajectory and its approximation by a forecast model-motivaan Lyapunov time bounds predictability. The long lookback of models like Chronos allows them to leverage information from multiple past timepoints, and thus stabilize accumulation of error relative to forecast models that only consider the most recent timepoints (Viswanath, 2001). In this sense, long-context forecasting resembles multistep integration (Zhang & Cornelius, 2023; 2024). Recent work on dynamic mode decomposition and Koopman operator inference take this idea even further, by showing that time delays can lift dynamical systems into spaces where the dynamics linear (Brunton et al., 2017). We therefore broadly interpret the zero-shot capabilities of Chronos, which improve with model size, as illustrating the intrinsic inductive bias that comes from lifting nonlinear time series to very high dimensions. However, this does not fully explain our observation that long context windows, spanning multiple Lyapunov times, improve zero-shot forecasts. Instead, we attribute this phenomenon to the recent discovery of in-context learning in pre-trained forecast models, which only recently starting to be explored in SciML (Yang et al., 2023; Subramanian et al., 2024).
the intuition that
nearly
Our study therefore affirms the suitability of the foundation model paradigm for SciML tasks. An important future direction for our investigation is task-specific tuning, in which the weights of large pre-trained models like Chronos are fine-tuned on small number of example chaotic time series. This differs from the zero-shot in-context learning that we discuss above, and recent foundation mod- els for partial diffcrential equations have found that in-weights tuning can improve generalization (Subramanian et al., 2024). In initial experiments, we found that at least two orders of magnitude more data were required to stably update the weights and validation scores of Chronos. However, this came at the expense of worse performance the original Chronos training dataset, implying that our dynamical systems dataset differs from a typical time series corpus. This underscores the need for large-scale retraining or low-rank adaptation to further tune Chronos to our task. This mir- rors results for large language models, where in-context learning has been shown to be preferable when few examples of the target task are available (Liu al., 2022a).
Walltime imperfectly measures computational costs, as different models are specialized for different hard- ware (e.g. paralleization GPU acceleration). Nonetheless, walltime within a given model class provides proxy for model's practical performance.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
YZ acknowledges support from the Omidyar Fellowship and NSF DMS 2436231. WG was sup- ported by NSF DMS 2436233 and NSF CMMI 2440490. This project has been made possible in part by Grant No. DAF2023-32959 from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. Computational resources for this study were provided by the Texas Advanced Computing Center (TACC) at The University of Texas at Austin.
8 REPRODUCIBILITY STATEMENT
All zero-shot benchmark forecast results and scripts are available online at https: //ggt//gtttttt com/w hom/wwllin williamgiipine hitpilpsss/i/siiiieeeesessssssss ddttssdattssaaaa._a_aad data. The dynamical systems benchmark dataset is available online at https: /ttttpps////tubb.......
thub. coom/wwll wwwwwwlllmmme... amggi pin/dysts
REFERENCES
Alice EA Allen, Nicholas Lubbers, Sakib Matin, Justin Smith, Richard Messerly, Sergei Tretiak, and Kipton Barros. Learning together: Towards foundation models for machine learning interatomic potentials with meta-learning. upj Computational Materials, 10( 1051): 2024.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, al. Chronos: Learning the language time series. arXiv:2403.0788 1,, 2024.
Alexis Asseman, Tomasz Kornuta, and Ahmet Ozcan. Learning beyond simulated physics. In Modeling and Decision-making in the Spatiotemporal Domain Workshop, 2018. URL https:
/ /ppeneevieww httpiiiwwwee/peee.. nett http:e/wwwwweeeeeee.e...i.ddie
jWsRF7.
Kamyar Azizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design. Nat.
Rev. Phys., pp. 2024.
Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecchina. Shaping the learning landscape in neural networks around wide flat minima. Proc. Natl. Acad. Sci. U.S.A., 117(1):161-70,, 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proc. Natl. Acad. Sci. U.S.A., 116 132):15849-15554, 2019.
Cristian Bodnar, Wessel P Bruinsma, Ana Lucie, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: A foundation model of the atmosphere. arXiv:2405. 30663, 2024.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv:2108.07258 2021.
Jason J Bramburger and Giovanni Fantuzzi. Data-driven discovery of invariant measures. Proceed- ings of the Royal Society A, 22002226) 2024.
Manuel Brenner, Florian Hess, Jonas Mikhaeil, Leonard F Bereska, Zahra Monfared, Po-Chen Kuo, and Daniel Durstewitz. Tractable dendritic mns for reconstructing nonlinear dynamical systems. In International conference on machine learning, pp. 2292-2320. Pmlr, 2022.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Steven L. Brunton, Joshua I. Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. U.S.A., 113(15): 3932-3937, 2016.
Steven L Brunton, Bingni Brunton, Joshua Proctor, Eurika Kaiser, and J Nathan Kutz. Chaos intermittently forced linear system. Nature communications, 8(1):19, 2017.

---

Published as a conference paper at ICLR 2025
Steven L Brumton, Marko Budisić, Eurika Kaiser, and J Nathan Kutz. Modern Koopman theory for dynamical systems. SIAM Rev., (4(2))229-344) 2022.
Cristian Challu, Kin Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecast- ing. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 6989-6997, 2023.
Renyi Chen and Molei Tao. Data-driven prediction of general hamiltonian dynamics via learning exactly-symplectic maps. In International Conference on Machine Learning, pp. 1717-1727. PMLR, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. NeurlPS, 31, 2018.
Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. (xXiv:2304.08422 2023.
Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Naidu, and Colin White. Forecastpfn: Synthetically-traine zero-shot forecasting. Advances in Neural Information Pro- cexsing Systems, 36, 2024.
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M Gifford, and Jayant Kalagnanam. Tims: Fast multi-level tiny time mixers for improved zero-shot and few-shot forecasting of multivariate time series. arXiv preprint arXiv:2401.0333 003955, 2024.
Yu Feng and Yuhai Tu. The inverse variance flatness relation in stochastic gradient descent is critical for finding flat minima. Proc. Natl Acad. Sci. U.S.A., 11800):e0016617118 2021.
variance-flatness
Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitikk. Units: Building a unified time series model. arXiv preprint 1rXiv:2403.00131, 2024.
Azul Garza and Max Mergenthaler-Canseco. Timegpt-l, arXiv:2310.03588 2023.
Daniel Gauthier, Erik Bollt, Aaron Griffith, and Wendson AS Barbosa. Next generation reservoir computing. Nat. Commun., 12:5564, 2021.
William Gilpin. Chaos as an interpretable benchmark for forecasting and data-driven modelling. NeurIPS, 34, 2021.
William Gilpin. Model scale versus domain knowledge in statistical forecasting of chaotic systems. Phys. Rev. Research, 5(4):043252, 2023.
William Gilpin. Generative learning for nonlinear dynamics. Nat. Rev. Phys., ((3):194-200 2024.
Rakshitha Wathsadini Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Niclas Göring, Florian Hess, Manuel Brenner, Zahra Monfared, and Daniel Durstewitz. Out-of- domain generalization dynamical systems reconstruction. arXiv:2402. 18377, 2024.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv:2402.03885. 2024.
Peter Grassberger and Itamar Procaccia. Measuring the strangeness of strange attractors. Physica D: nonlinear phenomena, (11-)):1892200 1983.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Divij Gupta, Anubhav Bhatti, Suraj Parmar, Chen Dan, Yuwei Liu, Bingjie Shen, and San Lee. Low-rank adaptation of time series foundational models for of modality forecasting. arXiv preprint 240 2024.
12

---

Published as a conference paper at ICLR 2025
Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli,, Roberto Molinaro, Em- manuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes. arXiv:2405.19101, 2024.
John Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaus- sian mixture models. In 2007 IEEE International Conference Acoustics, Speech and Signal Processing-ICASSPP00) volume 4, pp. IV-317. IEEE, 2007.
Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, al. Darts: User- Uisen-friendly modern machine learning for time series. Journal of Machine Learning Research, 23(124):1 2022.
Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learning chaotic dynamics. In International Conference on Machine Learning, pp. 13017- 13049. PMLR, 2023.
S Hochreiter. Long short-term memory. Neural Computation MIT-Prees, 1997.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2021.
Jeremy Huke. Embedding nonlinear dynamical systems: A guide to takens' theorem. 2006.
Rob J Hyndman and Anne B Koehler. Another look at measures of forecast accuracy. International journal of forecasting, 22(4):679 688, 2006.
Herbert Jaeger and Harald Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy wireless communication. Science, 304(5667):78-8) 2004.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li. Shirui Pan, et al. Time-LLM: Time series forecasting by repro- gramming large language models. arXiv:2310.01728, 2023.
Ian D Jordan, Piotr Aleksander Sokół, and Il Memming Park. Gated recurrent units viewed through the lens continuous time dynamical systems. Frontiers in computational neuroscience, 15: 678158, 2021.
George Em Karniadakis, Joannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang- Physics-informed machine learning. Nat. Rev. Phys., 3(6)):22-444)) 2021.
Ling-Wet Wei Kong, Hua-We Wei Fan, Celso Grebogi, and Ying-Cheng Lai. Machine learning prediction of critical transition and system collapse. Phys. Rev. Res.. 3(1:0013090, 2021.
Ling-Wer Kong, Yang Weng, Bryan Glaz, Mulugeta Haile, and ing as digital twins for nonlinear dynamical systems. Chaos, 33(3), 2023.
Lai. Reservoir comput-
Aditi S Krishnapriyan, Alejandro F Queiruga, N Benjamin Erichson, and Michael W Mahoney. Learning continuous models for continuous physics. Communications Physics, 6(1):319, 2023.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform didt----iitiittion.. arXiv preprint arXiv:2202.10054, 2022.
Pedro Lara-Benítez, Luis Gallego-Ledesma, Manuel Carranza-Gareía and José M Luna-Romera. Evaluation of the transformer architecture for univariate time series forecasting. In Advances in Artificial Intelligence: 19th Conference of the Spanish Association for Artificill Intelligence, CAEPIA 2020/2021, Málaga, Spain, September 22-24, 2021, Proceedings 19, pp. 106-115. Springer, 2021.
Herbert Levine and Yuhai Tu. Machine learning meets physics: two-way street. Proc. Natl. Acad. Sci. U.S.A., 221277):2403580121 2024.
13

---

Published as a conference paper at ICLR 2025
Matthew Levine and Andrew Stuart. A framework for machine learning of model error in dynamical systems. Commun. Am. Math. Soc., 2(07):283- -344, 2022.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopoolos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp. 19565-19594 PMLR, 2023.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial diffcrential equations. ar arXiv:2010.0882 2020.
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. vrXiv:2403.14735 2024.
Toni JB Liu, Nicolas Boullé. Raphaël Sarfati, and Christopher Earls. Lims learn governing principles of dynamical systems, revealing an in-context neural scaling law. arXiv preprint arXiv:2402.00795, 2024a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang. and Tang. P-tuning: Prompt tuming can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Short Papers). Association for Computational Linguistics, 2022a.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-9899 2022b.
Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes: Autore- gressive time series forecasters via large language models. 0rXiv:2402.02370. 2024b.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. To- wards understanding grokking: An effective theory of representation learning. Advances in Neu- ral Information Processing Systems, 35:34651-3466 2022c.
Jiecheng Lu, Yan Sun, and Shihao Yang. arXiv:2405. 14982, 2024.
In-context time series predictor. arXiv preprint
Zhixin Lu and Danielle S Bassett. Invertible generalized synchronization: A putative mechanism for implicit learning in neural systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(6), 2020.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m5 competition: Back- ground, organization, and implementation. International Journal of Forecasting, 38(4):3325- 1336, 2022.
Kaden McKeen, Laura Oliva, Sameer Masood, Augustin Toma, Barry Rubin, and Bo Wang. Eeg-fin fim: An open electrocardiogram foundation model. arXiv preprint arXiv:2408.05177, 2024.
Alfredo Medio and Marji Lines. Nonlinear dynamics: A primer. Cambridge University Press, 2001.
Igor Mezić. Analysis of fluid flows via spectral properties of the koopman operator. Annual review fluid mechanics, 45(1):357-377 2013.
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dy- namics with runs. Advances in Neural Information Processing Systems, 35: 1297-1111131. 2022.
John A Miller, Mohammed Aldosari, Farah Saced, Nasid Habib Barna, Subas Rana, I Budak Arpinar, and Ninghao Liu. A survey of deep learning and foundation models for time series forecasting, wwXiv:2401.133/2. 2024.
14

---

Published as a conference paper at ICLR 2025
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Stat. Mech., 202((12) 2021.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of over-paramettization in generalization of neural networks. arXiv: 2018.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv:2301.1034) 2023.
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:/ http:///ppeeeiiiieee /tttts:///ppeeeeeeies wttp::/weweeeoueeeeeeeee forum?id- JbdcOvTOcol.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv: 10437, 2019.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zero-shot time-series forecasting. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 9242-922299 2222,, 2021.
Edward Ott. Chuos in dynamical systems. Cambridge university press, 2002.
Tim N Palmer. Predicting uncertainty in forecasts of weather and climate. Reports progress in Physics, 63(2):71 2000.
Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Rudy Morel, al. Astroclip: cross- modal foundation model for galaxies. Monthly Notices of the Royal Astronomical Society, 531 (4):4990-5011, 2024.
Dhruvit Patel, Daniel Canaday, Michelle Girvan, Andrew Pomerance, and Edward Ott. Using ma- chine learning to predict statistical properties of non-stationary dynamical processes: System climate, regime transitions, and the effect of stochasticity. Chaos, 31(3), 2021.
Jaidcep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott. Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach. Phys. Rev. Lett., 120(2))024102, 2018.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gener- alization beyond overfitting on small algorithmic datasets. arXiv:2201.0211 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67 2020.
Md Ashiqur Rahman, Robert Joseph George, Mogab Elleithy, Daniel Leibovici, Zongyi Li, Boris Bonev, Colin White, Julius Berner, Raymond A Yeh, Jean Kossaifi, et al. Pretraining codomain attention neural operators for solving multiphysics pdes. arXiv:2403.12553 2024.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Bilos, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, et al. Lag-llama: Towards foundation models for time series forecasting. arXiv:2310.08278, 2023.
Irmantas Ratas and Kestutis Pyragas. Application of next-generation reservoir computing for pre- dicting chaotic systems from partial observations. Phys. Rev. E, 109(6):062215, 2024.
Christof Schötz, Alistair White, Maximilian Gelbrecht, and Niklas Boers. Machine learning for predicting chaotic systems. arXiv:2407.20158 2024.
Robert Shaw. Strange attractors, chaotic behavior, and information flow. Zeitschrift für Natur- forschung A, 36(1):80-112, 1981.
15

---

Published as a conference paper at ICLR 2025
Zezheng Song, Jiaxin Yuan, and Haizhao Yang. Fmint: Bridging human designed and data pre- trained models for differential equation foundation model. arXiv:2404.14688 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Process- ing Systems, 36, 2024.
Makoto Takamoto, Timothy Praditia, Ruphael Leiteritz. Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances Neural Information Processing Systems, 15:1596-611, 2022.
Seiji Takeda, Akihiro Kishimoto, Lisa Hamada, Daiju Nakano, and John R Smith. Foundation model for material science. In Proceedings the the AAAI Conference on Artificial Intelligence, volume 37, pp. 15376-15383, 2023.
Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? arXiv:2406. 16964, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964
Divakar Viswanath. Global errors of numerical ode solvers and lyapunov's theory of stability. IMA journal of numerical analysis, 21(1)3887-406, 2001.
Pantelis Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis P Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural. Netw,, 126: 191-217, 2020.
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.0306/ 2020.
Weinan. A proposal on machine learning via dynamical systems. Commun. Math. Stat., 1(5): 1-11,2017
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey, arXiv:2202.07125, 2022.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv:2402.02592, 2024.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL https: / openrevi ew net. / forum? d=RdJVFCH JUMT.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv:2009.11844 2020.
Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting, IEEE Transactions on Knowledge and Data Engineering, 2023.
Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning with data prompts for differential equation problems. Proc. Natl. Acad. Sei. U.S.A., 122(39)e23101422222 2023.
Lu Yang, Xiuwen Sun. Boumediene Hamzi, Houman Owhadi, and Naiming Xie. Learning dynami- cal systems from data: A simple cross-validatie perspective, part v: Sparse kernel flows for 132 chaotic dynamical systems. Physica D: Nonlinear Phenomena, 460:134070, 2024.
16

---

Published as a conference paper at ICLR 2025
Rose Yu and Rui Wang. Learning dynamical systems from data: An introduction to physics-guided deep learning. Proc. Natl. Acad. Sci. U.S.A., (3(27))e231188812,, 1808121, 2024.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12. 2014, Proceedings, Part I 13, pp. 818-833. Springer, 2014.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp- 11121-11128, 2023.
Yuanzhao Zhang and Sean P Cornelius. Catch-22s of reservoir computing. Phys. Rev. Research, (3):033213, 2023.
Yuanzhao Zhang and Sean P Coreeliss. How more data can hurt: Instability and regularization in next-generation reservoir computing. arXiv:2407.00 08641, 2024.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-tern tengtermm series forecasting. In International conference on machine learning, pp. 77268-27286. PMLR, 2022.
17

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL SCHEMATICS
Data from Lorenz
Train
Baseline models
Forccast
MM mm
w/
(e.g. NBEATS)
www
WW
may
(learn the flow)
Prediction for Lorenz
Contexi
mmh
Data from Lorenz
Data from diverse domains (not including Lorenz)
Context
Prodiction Lorenz
Pre-train
Foundation models
Forecast
www M
ww
(e.g.. Chronos)
(zcro shot)
Figure 7: Difference between baseline models and foundation models in forecasting chaotic systems. Classical deep-learning models (i.e., baseline models) forecast chaotic system by learn- ing the underlying vector field or flow map. To achieve this, the model adjusts its weights based on data from the same chaotic system. In contrast, foundation models (e.g., Chronos) do train directly on the system they want to predict. Instead, they aim to "learn the language of time series" (Ansari al., 2024) by training vast amounts of time series data from diverse domains. After that, foundation models can make zero-shot forecasts on any (previously unseen) chaotic system based on short context trajectory.
B ADDITIONAL METHODS
B. LORENZ EQUATIONS.
Lorenz oscillator is one of the most studied chaotic systems and described by the following equa- tions:
ă aly x), ý = x(p 2) - y: ry A2z
where the default parameter values are = 10, p 28, and B 8/3..
B.2 POINTWISE ERROR METRICS
We quantify point-wise accuracy of forecasts using the Symmetric Mean Absolute Percentage Error (sMAPE).
T
MMAPE(x,X) = 100
X X1
T
Xx
where X1, X2, .... XT correspond to the true test values of a time series up maximum forecast horizon T', and &1, &2...., &T are the predictions of a forecast model at those same timepoints.
Prior studies have evaluated the suitability of various enror metrics in evaluating forecast accuracy (Hyndman & Koehler, 2006; Makridakis et al., 2022), including specifically on dynamical sys- tems prediction (Gilpin, 2021: 2023), and found that sMAPE strongly correlates with other metrics (c.g. RMSE, NRMSE, MASE, Spearman correlation) while exhibiting favorable properties like a bounded range.
B.3 MEASURING ATTRACTOR SIMILARITY.
We measure attractor similarity using an approach introduced in previous works (Hess et al., 2023; Brenner et al., 2022). The state space divergence between the true and generated attractors is given
18

---

Published as a conference paper at ICLR 2025
by the Bailbacck-Leibler divergence between the distributions p(x) and g(x),
p(xx)
Datep (x((p(x) ((x))
p(x) log
dx.
XERN
((x)
In high-dimensional spaces, a Gaussian Mixture Model (GMM) is created from the true and gener- ated trajectories in order to approximate these distributions,
T
p(x) = (1/T)
N(x: Xx, Ex)
(2)
and
= (0/)-(17))) Σ Nix: & Σ₁).
While prior works set the covariance matrix equal the scaled identity matrix Σ, of with 0t for all I, we instead set 0; = ||x x1-1ll in order to adjust for uneven spacing among data points. We next perform Monte Carlo sampling and estimate the KI. divergence as
1 EEOO log
p(x
Dstsp
((x()))
where (+) are samples drawn from the true orbit (Hershey & Olsen, 2007).
C DEPENDENCE OF FORECAST ACCURACY ON INITIAL CONDITIONS
We investigate the degree which zero-shot forecasting performance depends the initial con- dition. As an illustrative example, in the right panel of Figure 8, we repeat the experiment shown in Figure 2, but for different initial condition. We use the base Chronos model with maximum context of 512 points, but we choose a trajectory emanating from a different point on the chaotic attractor. We see that the performance of Chronos is noticeably worse for this trajectory, indicating that the particular choice of initial conditions can influence zero-shot performance.
For both initial conditions, Chronos attempted perform pattern matching by looking for snippets in the context trajectory that most closely resemble the history immediately preceding the prediction and simply repeating that motif. The difference is that there is a very good repeating pattern in the context trajectory on the left but not the right, which directly leads to worse prediction from the second initial condition. From the perspective of Takens' embedding theorem, this context-matching strategy is trying to find the closest context data point to the initial condition in the delay embedding space and repeating the context trajectory from that point.
To further quantify variability in forecast accuracy caused by initial conditions, we sample set of 200 trajectories originating from different points on the attractor, and generated zero-shot forecasts that we evaluated using the VPT (Eq. 1). We define the initial condition for each trajectory as the final point of the context given to the model before a forecast. We observe wide variation in predic- tion performance with the initial condition (Fig. 9, with a nearly exponential distribution of VPT across initial conditions. Thus while the median VPT of Chronos is relatively high (approaching Lyapunov time for the largest models), occasionally initial condition will result in a forecast that remains accurate for several Lyapunov times.
In order identify the origin of these anomalously long forecasts, calculate the relative denstity of the attractor at each initial condition. Chaotic dynamical systems approach a steady-state distri- bution of points, the strange attractor, with continuous density μ(x) known as the natural measure of the system. We estimate (xx) using Eq. 2 for each initial condition, and compare the (PT(x) of a forecast originating from each initial condition x to the estimated measure at that point (a(x). We perform this procedure for 20 distinct initial conditions from each of the 135 chaotic dynamical sys- tems in our dataset (Fig 9B). In the figure, we highlight the initial conditions for the Lorenz attractor in blue. We find a weak but robust correlation between measure and forecast accuracy (Spearman's rank order coefficient, p = 0.26 = 0.03, p < 10-3 N = 2700. This consistent with the idea that zero-shot forecast models perform better at forecasting denser, more common regions of the attrac- tor, because those points are more common in the context. Conversely, rarer points (i.e., extremal points closer to out-of of-distiibtttinnn dynamics relative to the context points) lead worse forecasts.
19

---

Published as a conference paper at ICLR 2025
Predicted Ground Truth
x(0
MM wp Yimmn
mmn' ppmm
Figure 8: Zero-shot forecasting performance depends on initial conditions. Zero-shot forecasts of the Lorenz attractor using Chronos- for two different initial conditions the Lorenz attractor. Both forecasts use the same context length of 512 timepoints; their performance difference arises only from their starting point.
A
B 10
VPT (Lyapunov times)
10"
Ldh
Proportion of IC
10-1
Lorenz
systems
10-7
10-6
10-5
10
10-3
10-2
10-1
VPT (Lyapunov times)
Initial Condition Density
Quantification of the dependence of zero-shot forecasts on initial conditions. (A) A set of points on the Lorenz chaotic attractor, colored by the forecast accuracy (VPT) of zero-shot forecasts in which they were the final context point. A histogram of the accuracy values underlaid. (B) The forecast accuracy (VPT) versus the relative density of the region of the attractor in which the last context point appears. Black circles indicate 20 initial conditions from each of 135 chaotic dynamical systems, and the 20 initial conditions from the Lorenz attractor are highlighted in blue.
Figure 9:
20

---

Published as a conference paper at ICLR 2025
D APPLICATION TO REAL-WORLD CHAOTIC SYSTEMS
We next compare our results for our large-scale dynamical systems benchmark dataset to real-world multivariate time series from chaotic systems. Unlike simulated differential equations, real mea- surements exhibit measurement еттоr, stochasticity, and non-stationarity.
Our experimental dataset consists of 400 fps video of an oscillating double pendulum, as recorded on a high-speed Phantom Miro EX2 camera (Asseman et al., 2018). The video is paired with a time series of centroid coordinates for each pendulum hinge and joint, as extracted by the original authors using object tracking. This time series consists of positions of the pivot attachment to the wall, the hinge connecting the first and second pendula, and the second pendulum's 's tip. We transform the dataset into new sequences that represent the angles each pendulum forms with the vertical axis, denoted as (01, 02). We then numerically differentiate these angle measurements to obtain the angu- lar velocities (01, 82). In an ideal double pendulum system, the set of four variables (61, 82, 01, 82) uniquely parameterizes the Hamiltonian, thereby defining the system's 's attractor. However, for the experimental data, the time-averaged kinetic energy T x of + 63 gradually decreases course of the experiment. As a result, the pendulum dataset non-stationary, with an attractor that gradually changes over time. We downsample this time series by factor of 3.
the
We use Chronos (base model) to forecast this dataset for 7 non-overlapping contiguous time intervals spanning the full experiment. Each window is split into context window of length 512 and testing dataset of length 300, for total of 8 x (512 + 300) ~ 6500 total timepoints. We find that the error exhibits similar scaling as we observe for ergodic dynamical systems the main text (Fig. 10). This indicates that experimental variation and measurement errors do not preclude the application of zero-shot forecasting to chaotic time series generated by real-world systems. Additionally, the pendulum dataset exhibits non-stationarity due to gradual loss of energy from the system. As result, this dataset exhibits weak distribution shift between the training (context) and testing (zero- shot forecasting) settings. Because we observe the same general scaling of error in the 135 ergodic and stationary systems, we conclude weak distribution shift does not preclude effective zero-shot forecasting. Thus, in this example, Chronos exhibits out-of-domain generalization, because the underlying chaotic attractor (and thus distribution testing points) changes relative to the context.
A
B
Error (sMAPE)
25
x.
x,0
True Predicted
1
1
10-1
Forecast Horizon (Lyapunov time)
Figure 10: Zero-shot forecasting of a chaotic pendulum experiment. (A) Zero-shot forecasts along the first angular coordinate of a double pendulum for the base Chronos model, for 7 different initial conditions. (B) Scaling forecast error with forecast horizon. Curve corresponds to means and standard errors across 7 initial conditions and 4 coordinates each.
21

---

Published as a conference paper at ICLR 2025
E PROBING OUT-OF-DIITRIBUTIION DYNAMICS AS TRAJECTORIES LEAVE THE ATTRACTOR
0.8
VPT (Lyapunov times)
0.4
0.2
0.2
0.4
0.6
0.8
1.0
Nonstationarity
Figure 11: Zero-shot forecasts degrade with distribution shift. Forecast accuracy (VPT) of zero- shot forecasts with Chronos-basse as the degree of nonstationarity in the time series varies via Eq. 3. Curve and error bars are median and standard error over 20 initial conditions for each of 135 chaotic dynamical systems.
We next evaluate the degree to which non-stationarity affects zero-shot forecasting performance. For each trajectory considered in the main text, we apply an exponential modulation along the time dimension. For time series of length T given by X1, X2; ..,. XT, the exponentially-decayii modulation has the form,
Juin
X₁
X,e
By decreasing fmi from 1 to 0, we increase the degree to which the dynamics appear non-stationary. When f 1, then the damping term becomes constant and the dynamics are unaffected. How- ever, when fin -> O, the dynamics resemble damped oscillations that monotonically approach fixed point. We thus consider experiments forecasting time series with fmin < quantitative probe of the degree to which zero-shot forecasts are applicable to real-world systems, in which the chaotic attractor irreversibly deforms due to processes like dissipation. In machine learning context, this setting corresponds to dfittiibbittttinnn or -f--a-- generalization, in which the forecast points describe different dynamical regime than the context (Göring et al., 2024).
We find that, across all 135 systems, the performance of Chronos degrades as the degree of nonsta- tionarity 1 fmin increases (Fig 11). This observation matches intuition, based vation the main text that Chronos performs in-context learning the distribution of points (and pairwise, k-wise conditional dependencies among successive timepoints). We also find in the main text that Chronos performs more strongly on trajectories resembling its training data. Nonstationar- ity undermines all of these mechanisms, leading the degradation in performance as the forecast regime more strongly differs from the context.
obser-
Because context-parrotinn is particularly effective strategy for stationary systems like ergodic chaotic attractors, time series models like NBEATS, which can directly identify and model mono- tonic trends, have an advantage on simple out-of-distr forecasting tasks like the one we con- sider here. NBEATS and its variants have successfully been applied to several types of time series with predominant trends, underscoring their advantage this setting Challu et al. (2023), Based on this observation, we anticipate that several modifications could make foundation models like Chronos more robust to weak nonstationarity: (1) Chronos currently uses an encoder-decoder lan- guage model Raffel et al. (2020). Using Chronos's 's tokenizer in tandem with modern language model with an explicit positional encoding scheme, like rotary positional embedding, would pro- vide the model with explicit time information that would allow it to capture longer-term trends in
22

---

Published as a conference paper at ICLR 2025
time series Su al. (2024). (2) Pretraining with short time series. While Chronos's original training dataset includes many nonstationary processes, shorter time series generally exhibit greater nonstationarity, and so their inclusion represents simple mechanism to improve model robustness. (3) Biasing generative forecasting towards rarer states. As generative model, Chronos generates forecasts prohabilistically by sampling multiple potential future trajectories. Modifications of this scheme that encourage oversampling of rarer states could help the model better account for irre- versible processes, though potentially at the expense of lower performance on ergodic processes.
F BASELINE MODELS
F. BASELINE MODEL IYPERPARAMETERS
Our baseline models follow the experiment design and hyperparameter tuning procedure used in prior works on the chaotic systems dataset (Gilpin, 2021; 2023). Those works contain qualitative descriptions of the different models, and the performance results obtained in those works motivate our particular baseline model choices. newly introduced linear state space model that can achieve nearly-optimal error rates for linear dynamical systems (Das et al., 2023). For many models, we use reference implementations and hy- perparameters found in the Darts forecasting library (Herzen al., 2022). For the gext--generation reservoir computer (NVAR). we use the default settings used in the original work (Gauthier et al., 2021). However, in order fairly tune hyperparameters across models, for each model select one hyperparameter to tune that corresponds to the lookback window, or context, that sets the number of past timepoints that the model simultaneously processes when generating forecast.
also include the Time-series Dense Encoder (TiDE),
N-BEATS Model (Oreshkin et al., 2019)
Key Hyperparameters:
Input Length: Tuned for each system among {0.067, 0.167, 0.333, 0.5, 0.833, 1} Lyapunov times
Number of Stacks: 30
Number of Blocks: 1
Number of Layers:
Layer Widths: 256
Expansion Coefficient Dimension: 5
Degree of Trend Polynomial: 2
Dropout Fraction: 0.0
Activation Function: ReLU
Transformer Model
(Vaswani et al., 2017)
Key Hyperparameters:
Input Length: Tuned for each system among (0.067, 0.167, 0.333, 0.5, 0.833, 1}
Lyapunov times
Number Attention Heads: 4
Number Encoder Layers: 3
Number Decoder Layers: 3
Dimension Feedforward: 512
Dropout Fraction: 0.1
Activation Function: ReLU
TiDE (Das al., 2023)
Key Hyperparameters:
Input Length: Tuned for each system among (0.067, 0.167, 0.333, 0.5, 0.833, 1} Lyapunov times
23

---

Published as a conference paper at ICLR 2025
Number of Encoder Layers: 1
Number of Decoder Layers:
Decoder Output Dimension: 16
Hidden Dimension Size: 128
Past Temporal Width: 4
Future Temporal Width: 4
Past Temporal Hidden: None
Future Temporal Hidden: None
Temporal Decoder Hidden: 32
Dropout Fraction: 0.1
NVAR
(Gauthier et al., 2021)
Key Hyperparameters:
Number Input Lags: Tuned for each system among {0.067, 0.167, 0.333, 0.5, 0.833,
} Lyapunov times
Maximum Order: 2
Regularization: 10-4
Stride: 1.0
LSTM (Hochreiter, 1997)
Key Hyperparameters:
Input Length: Tuned for each system among {0.067, 0.167, 0.333, 0.5, 0.833, 1} Lyapunov times
Hidden Dimensionality: 25
Number of Recurrent Layers: 2
Dropout Fraction: 0.0
Training Length: 24
F.2 FINE-TUNING CHRONOS
As an informative additional baseline, we attempted fine-tune Chronos-base on the chaotic sys- tems dataset. From the zero-shot experiments, we compiled a collection of 1.3 x 106 observations, corresponding to trajectories of length 512 timepoints originating from 20 initial conditions for each of 135 chaotic dynamical systems. We fine-tuned Chronos-base using the authors' original training scripts, with all hyperparameters matching those used in original Chronos training run Ansari et al. (2024). On zero-shot dataset, did not observe strong improvement in Chronos's 's validation scores on held-out trajectories. Instead, the loss plateaued early during training, and the qualitative appearance of forecasts did not improve only fine-tuning on single system, the Lorenz attractor, we observed similar results. Moreover, we observe weak reduction in forecast accuracy on datasets randomly drawn from Chronos's 's training corpus. Across the 135 chaotic dynamical systems in our dataset, we did not observe a general re- lationship between fine-tuning performance and invariant properties of the underlying system, such as dimensionality or Lyapunov exponents.
the zero-shot case. When instead tried
Based on these observations, we conclude that the training behavior of Chronos is decoupled from properties of the underlying datasets in the training regime we reach in our fine-tuning experiments. We thus conjecture that the chaotic systems time series dataset strongly differs from the large time series corpus on which Chronos was originally trained, leading to fine-tuning failing due to strong task shift Kumar et al. (2022). This phenomenon represents a variant out-of-distribution gen- eralization error, manifesting tuning Chronos for chaotic systems will require full retraining on dataset comparable size to the Chronos training corpus "10 -10" observations),
slow convergence new datasets. We therefore expect that fine-
well potential customizations of the tokenizer and language model to better handle dynamical systems datasets. For example, recent works note that multivariate time series often exhibit weak coupling among channels, motivating
24

---

Published as a conference paper at ICLR 2025
the general use of channel-independerr training schemes Nie et al. (2023). We also expect that hyperparameters, particularly training schedule and optimization rates, will need to be selected in or- der to obtain noticeable improvements. This level of tuning and data scale exceeds that used for the other baseline models, and so we defer further investigation of fine-tuning and few-shot learning to future work. Additionally, in order avoid fully retraining Chronos for our task, alternative strate- gies such as low-rank adaptation Hu et al. (2021), and its generalizations for time series forecasting Gupta al. (2024), may be applied in future work.
G ADDITIONAL EXPERIMENTS AND ANALYSES
1STM
Divergence from True Attractor
NVAR
109
TIDE
NBEATS
Transformer
mmni
base
arge
0.0
0.2
0.6
1.2
Valid Prediction Time
Figure 12: Zero-shot forecast models capture attractor geometry well, as measured by the KL Divergence. The state space divergence Detep between the predicted attractor and the true attractor, versus the VPT of the corresponding model. The red markers represent variants of Chronos with different model sizes: tiny (8M parameters), mini (20M parameters), small (46M parameters), base (200M parameters), and large (710M parameters). The blue markers represent the baseline models. Models closer to the bottom capture the attractor geometry better, and models closer to the right make accurate point forecasts for longer.
25

---

Published as a conference paper at ICLR 2025
100
Vold Predtction mmn
Vald Predectio
MOEATS
Error (sMAPE)
LSTM
WDAR
71DE
ASEATS
70
Trane
WWAR
50
100
10
10-1
Forecast Horizon (Lyapunov time)
Figure 13: Comparison of univariate versus multivariate baseline forecasts chaotic systems. Because Chronos is a univariate forecast model that predicts each time series channel independently. baseline experiments we present the main text (left panel here) involve channel-independent training, in which each baseline model is separately trained and tested each dimension of the input time series. We repeat these experiments in a multivariate setting, by retraining the baseline models simultancously on all dimensions (right panel). All error bars are over 20 distinct initial conditions for each of the 135 chaotic systems.
Correlation With Lyapunov
V000 Presidin Tiee
10
10-1
10
10a
101
Forecast Horizon (Lyapunov time)
Figure 14: Correlation between forecasts and invariant properties. The correlation between the Lyapunov exponent of each of the 135 chaotic systems, and the sMAPE error of forecast model, as function of the prediction horizon.
26

---

Published as a conference paper at ICLR 2025
1.0
Accuracy of Fractal Dimension
0.8
0.6
0.4
0.2
0.0
10
102
Model Parameters (Millions)
Figure 15: Zero-shot aftractor reconstruction accuracy scales with model size. The Spearman correlation between the fractal dimension of Chronos's predictions. and the true fractal dimension of the underlying system, compared to the number of trainable parameters in the Chronos model.
100
Error (sMAPE)
50
10
10"
10!
Time (Lyapunov times)
Naive forecasts underperform all models evaluated. The growth of the sMAPE error for naive constant forecast, in which the most recent training point is carried forward as the prediction for all future values. The shaded region corresponds standard error across 135 dynamical systems, with 20 initial conditions each..
Figure 16:
27


Reviews:
[{"review_id": "Gxu08W61BT", "reviewer": "Reviewer_1G8R", "summary": "The article explores the application of foundation models, particularly zero-shot learning, to forecast chaotic systems. Using the foundation model Chronos, the study investigates whether these models can generate accurate short-term forecasts and capture the long-term statistical properties of chaotic attractors, which are characteristic of chaotic systems. Through an extensive evaluation on 135 chaotic systems, the authors found that, even without specific training on chaotic dynamics, Chronos provides competitive forecasts, preserving geometric and statistical attributes of chaotic attractors over the long term. The study presents these findings as a benchmark for the feasibility and limitations of using foundation models in scientific machine learning, especially in physics-informed tasks.", "questions": "1. Impact of Initial Conditions: Can the authors elaborate on how initial conditions affect Chronos’s zero-shot forecasting stability? Is there a significant dependency on starting points, especially when far from the attractor’s typical state?\n2. Effectiveness of Positional Encodings: How critical is the choice of positional encodings for Chronos when applied to chaotic systems? Have other encoding methods been tested, and if so, how did they affect both short-term accuracy and long-term attractor preservation?\n3. Model Complexity and Forecasting Horizon: Given that larger models perform better in long-term forecasting, could the authors provide insights into the mechanisms by which increased model size aids in managing chaotic variability?\n4. Comparison with In-Weights Fine-Tuning: Since the study focuses on zero-shot learning, have the authors considered comparing this approach with in-weights fine-tuning on a few chaotic trajectories to assess the performance improvement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive Benchmarking: The authors conduct a thorough and extensive evaluation, with a dataset that includes 135 chaotic systems, providing statistically robust results.\n2. Foundation Model Scalability: By demonstrating the scalability of Chronos across varying chaotic systems, the study establishes the potential of foundation models for scientific machine learning in challenging domains.\n3. Zero-Shot Learning Feasibility: The findings validate that foundation models can make meaningful forecasts in chaotic systems with minimal domain-specific adjustments, a promising step for generalizing AI across diverse scientific domains.\n4. Insight into Long-Term Statistical Consistency: The study’s focus on long-term attractor behavior offers a novel way to assess model performance, moving beyond conventional short-term forecast accuracy.", "weaknesses": "1. The article mentions that Chronos’s forecast accuracy is sensitive to initial conditions, but the specifics of this dependency aren’t deeply explored. Could the authors provide a more systematic investigation into how initial condition variability impacts model robustness? For example, have you considered evaluating Chronos’s performance across a range of initial conditions sampled from different regions of the attractor, particularly comparing accuracy in central versus peripheral areas? Additionally, could you quantify how forecast accuracy shifts as the initial conditions deviate from those in the training data? Such analyses would offer valuable insights into Chronos’s generalization capabilities and help identify any specific sensitivities related to initial condition variability.\n2. Positional Encoding and Temporal Structure: Given the chaotic nature of the systems studied, the choice of positional encodings (e.g., rotary embeddings) could significantly impact model performance, especially in maintaining temporal coherence over long horizons. However, the article lacks an in-depth discussion or ablation study regarding the choice of these encodings.\n3. Limited Explanation of Performance in Extreme Cases: The authors mention that larger models tend to perform better, yet details on why model size improves stability in chaotic systems are sparse. A theoretical or empirical justification of this scaling behavior, such as through model capacity for capturing non-linear dynamics, would improve understanding.\n4. Scalability of Model Parameters and Computational Requirements: Although Chronos shows promise in forecasting chaotic systems, the article could discuss practical limitations regarding computational costs, especially for larger model sizes, and the trade-offs compared to specialized models like NBEATS or TiDE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KxPK61iXRk", "reviewer": "Reviewer_yZ6s", "summary": "The paper presents a large-scale evaluation of the zero-shot learning capabilities of foundation models for chaotic systems. The authors investigate both short-term and long-term forecasting scenarios. They evaluate the foundation models against over 100 chaotic systems and observe several interesting findings. The key point is that foundation models not only generalize to new initial conditions but also to entirely new systems. This suggests the models have the capacity to capture the broader dynamical structures of chaotic systems.", "questions": "The authors aim to evaluate the proposed approach on real-world data? Interesting characteristics can be observed. \n\nHow does foundations models forecast accuracy vary with different initial conditions within chaotic systems? This sensibility limits its generalization? \n\nI suggest the authors to state limitations of the zero-shot long-term prediction problem. Another possible limitation is how the foundation model zero-shot approach will behave when there is a drastic different dynamical behaviour between training and testing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents strong findings on the power of foundation models for zero-shot learning in chaotic systems. Firstly, it demonstrates that foundation models can perform as well as fully trained models across multiple scenarios, especially in long-term forecasting.\n\nAnother strength of the paper is its study and evaluation of the suitability of foundation models in the field of scientific machine learning. This proof of concept can definitely lead to new developments in the study of chaotic systems.", "weaknesses": "Further experiments are needed to assess sensitivity to initial conditions and real-world chaotic systems, as these factors currently weaken the proposed evaluation and findings.\n\nWhile the application and findings of the paper are important, I believe the work in its current state lacks novelty due to limited experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cRxxK05Sn4", "reviewer": "Reviewer_MDoC", "summary": "The present paper proposes to use time-series generated from chaotic systems as a new benchmark in time-series forecasting.\nIt makes the case for time-series foundation models (Chronos) which were pretrained on diverse synthetic and real-world data which it finds to yield better results than customly trained algorithms. This is surprising given that Chronos is a univariate prediction model and therefore sees the x, y, z components of the chaotic systems independently.", "questions": "-", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The core contribution is a benchmark of foundation models on the evolution of different chaotic systems. I think this is a good contribution which goes into a similar direction as Chronos and ForecastPFN which use synthetic data (Chronos not exclusively) to pretrain their models and still obtain good performance. I agree with a reference in the conclusion stating that there appears to be common language shared by time-series as it's surprising that the existing synthetic data generators would result in such well performing models.\n- I see potential for this benchmark to be used itself for more expressive synthetic data in time-series foundation models.\n- I think the finding that chaotic systems can be learned better by foundation models from context rather than by a specific model is valuable and interesting.", "weaknesses": "- It would be good to give more details on the LSTM and transformer models shown in the paper in order to allow for a fair comparison. I couldn't find details on the number of layers used e.g. for LSTM after hyperparameter optimization like the number of layers or the state size of the LSTM.\n- Could you clarify your rational on choosing the models you evaluated in the paper? Given the recent interest in linear RNNs like Mamba 1 / Mamba 2 / Gated Linear Attention, could you provide some results in this model class? Maybe Mamba 2, as it's fast to train.\n- I think the paper in its current state is quite verbose and could benefit from a clearer structure. For example sections 5.1 and 5.2 could have individual paragraphs with titles indicating the finding which the authors would like to emphasize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)