Temporal horizons in forecasting: a performance-learnability trade-off
Authors: Pau Vilimelis Aceituno
Jack William Miller
Noah Marti
Youssef Farag
Victor Boussange
Venue: Accepted by TMLR ()

Abstract: When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, we address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.

Published in Transactions on Machine Learning Research (10/2025)
Temporal horizons in forecasting: a performance- trade-off
Pau Vilimelis Aceituno",", Jack William Miller?, Noah Marti', Youssef Farag', and Victor Boussange"
Institute of Neuroinformatiis, ETH Zürich and University of Zürich, Winterhurerstrasse 190, Zürich 8057, Switzerland
SSchoo of Computing, National Australian University, 108 North Rd, Acton ACT 2601, Australia Unit of Land Change Science, Swiss Federal Research Institute for Forest, Snow and Landscape Zürcherstrasse
111, Birmensdorf 8903, Switzerland "Corresponding author: Pau Vilimelis Aceituno, pau@ini.ush.ch
Reviewed on OpenReview:
Abstract
When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-hoo horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.
1 Introduction
Forecasting the future state of dynamical systems is fundamental challenge in scientific and engineering disciplinos, from climate modcling to robotics. central objective is to generate accurate prodictions as far into the future as possible. Autoregressive (AR) models, which iteratively feed their own predictions back
as inputs, are a standard tool for this task (Petropoulos al. 2022 Lai & Lu 2017 Azencot et al. 2020 Van den Oord et al., 2016 Lam et al. 2023).
Traditional AR models were linear and limited to oow-dineniional time series (Petropoulos et al.
2022 Lai &
Lu 2017). Modern AR models based on neural networks and trained via gradient descent, have emerged as powerful alternatives due to their ability to capture nonlinear dynamics in high-dimensional systems. Such nonlincarity is ubiquitous across scientific domains, including climate science (Houghton et al. 2001), ecology
2009 Huisman Weissing, 1999), physics (Levien & Tan 1993), and robotics (Igbal al. 2014). Consequently, AR models have demonstrated success c.g. weather and climate forccasting (Lam) et al. 2023; Kochkov et al.] 2024), ecological forecast. (Boussange et. al. 2024), physical system forecast.
Benincà et al.
Azzencot al. 2020 Erichson et al. 2019 Li et al.
2020
Miller al. 2022). robotic control (Sasagawal Beyond forecasting, AR models can be used for 2020
ct al. 2021 Williams et al. 2017 Heetmeyer et al.
2023
generative tasks like text or code generation (Brown et al.
A critical yet understudied challenge in training AR is selecting an appropriate temporal forecast horizon during training. Common approaches either use a single step prodiction scheme (Van den Oord et al. 2016 Azencot et al. 20200 Miller al. 2022; Brown et al. 2020) or adopt longer horizons without rigorous
justification (Lam et. al.] 2023: Kochkov et al.] 2024
Price et al. 2025 Cornille et. al. 2024). Currently,

---

Published in Transactions on Machine Learning Research (10/2025)
theoretical framework guides this choice. This gap extends to mechanistic modeling with differential equations or discrete maps, where methods like piecewise regression [iisarenko & Sornette 2004 Doya et al.
1992. Boussange et al.) 2024) and multiple shooting (Bock) 1981 England 1983
Aydogmus & Tor 2021
which similarly lack principled criteria for horizon selection.
In this work, we rigorously analyze how the temporal forecast horizon during training affects AR model performance in the context of dynamical system prediction. We derive practical guidelines for selecting this horizon based on the underlying system dynamics. Our contributions include:
theoretical analysis linking the temporal forecast horizon during training to the geometry of the loss landscape,
an empirical validation of these relationships across different dynamical systems,
practical recommendations for horizon selection in both data-driven and mechanistic modeling.
The remainder of this paper organized as follows. We first situate our work within the existing literature. Nexi, we introduce necessary background and notation. We then present our theoretical analysis of how temporal horizons shape the loss landscape, that we illustrate with numerical experiments. We examine the practical implications of our thcoretical findings in the presence of noise, and how it connects with the task of fitting mechanistic models. Finally, we conclude and suggest future research directions.
2 Related works
Most of the theoretical literature machine learning for dynamical system forecasting has focused on the vanishing or exploding gradient problem (VEGP), a phenomenon where the gradients over the parameters of a neural network scale with the temporal distance between the current and past state of the work
Pascanu 2013 Hochreiter et al. 2001 Bengio et al. 1994.. Common strategics that have been proposed to
mitigate VEGP include architectural modifications (e.g., gated mechanisms (Hochreiter 1997)), specialized
weight initializations (Narkhede et al., 2022), gradient clipping (Goodfellow 2016; Zhang et al. 2020). and
constrained dynamics (e.g., unitary recurrent neural networks (Arjovsky et al. 2016; Chang et al. 2019
Erichson al.) 20200). However, these techniques limit the expressivity of the models introduce biases
(Orhan & Pitkow| 2019 Schmidt et al. 2021 Mikhaeil et al. 2022). Recent works have derived connection
between the VEGP and system properties like Lyapunov exponents (Mikhaeil et al. 2022),, which provides an
intuition of how fast gradients explode, suggesting that it is possible to analyze mathematically the connection
between machine learning for forecasting and the dynamics of the system being learned.
On the practical side, tuning the forecast horizon is an empirical technique that has been used in a variety of context to optimize the performance of AR models. In particular, this strategy is also referred to multiple shooting or piecewise inference, and has been proposed for fitting classical discrete maps (Pisarenko & Sornette 2004). differential equation models Boussange et al. (2024), transformer models applied to language Glloeckle et al. 2024; Cornille et al. 2024) and in the context of reinforcement learning Zhang & Sutton 2017)) learning/Zhanng Similar strategies implicitly tune the forecast horizon via discount factors, effectively reducing the relevance of long time horizons. Concrete examples can be found for example weather forecasting (Lam et al. 2023
financial time scries modeling (Bernaciak & Griffin 2024), or in reinforcement learning, where the effective horizon influences model success, for example through discount factors in techniques like X-returns (Sutton 1995 Anand & Precup 2021). The relevance of temporal horizons for forecasting also appears in other fields. Model predictive control also requires careful horizon selection (Hewing et al.] 2020). Similarly, neural PDE solvers often prioritize high-frequency components, which are inherently tied to shorter temporal
dependencies (Wiener 1930 Lippe et al. 2023 Kurth et al. 2023).
Thus, although is well documented that the training horizon has an effoct on the convergence of AR models, we do not know of theoretical work that directly investigates the effect of the forecast horizon on the geometry of the loss landscape.
2

---

Published in Transactions on Machine Learning Research (10/2025)
3 Background and Notation
3.1 Dynamical systems
Dynamical systems consist of a state space X' in which the system has one position I, and a mathematical function that describes how I changes with respect to t. As the state of the system evolves in time, defines a trajectory. Trajectories may show variety of patterns. For example, they can converge to an attractor, diverge from a point of manifold, remain within a manifold indefinitely. Some trajectorics are inherently easy fixed point); some cannot be predicted (e.g., unstable systems,
to predict (e.g.; trajectories converging which are constantly expanding into new states).
Two particularly interesting families of trajectories, limit cycles and chaotic attractors, appear exclusively in nonlinear systems and are neither trivially predictable nor impossible to predict, and will be the focus of this work.
Limit cycles are orbits that repeat themselves, and can be characterized by the a closed line that defines their orbit and by their period p or their frequency 2r which represents the time it lakes for them to come back to a previous point; such that Tttp = Iz. Brin & Stuck 2002).
Chaotic trajectories the other hand never repeat any point in the trajectory, and can be defined by their chaotic attractor (the set of points towards which the system will evolve). In chaotic trajectories, the Lyapunov exponent À measures how two points that are initially close in the attractor diverge with time by the relationship |xa x'i = ea a - z/d, where zy is infinitesimally close to vi (Brin & Stuck, 2002).
Crucially, both chaotic or limit cycle systems are practically impossible to forecast very far into the future, as small differences between the model and the system being modeled will make long term predictions essentially random (sce A.5
3.2
Examples of dynamical systems
In order to develop and illustrate our theory, we will use the following examples of dynamical systems: A Lorenz attractor, double pendulum, simple limit eycle, and chaotic food web model Hastings (1991) The Lorenz altractor, food web and the Double pendulum are classical examples of chaotic attractors. All the systems are described in detail in appendix
3.3 Autoregressive models
Trajectories in discrete time dynamical systems, or continuous dynamical systems whose trajectories are sampled at fixed intervals, can generally be characterized by function that expresses how the system evolves in a single timestep,
(mm) = 6(x(m - 1))
(1)
where is the temporal index. Under this formulation, the dynamics of the system can be captured by function f, such that for a given state x(m), [xx((())) 0) gives a subsequent state x(m++ 1) approximation thereof.
Since f is constructed to output vectors with the same dimensionality as the input, we can generate a future stale a(m + k) by simple composition of the model
z + k) f fo... of (x(m), 0) 8 (x(m),
(2)
where & corresponds to the number of compositions. This composition constitutes the simplest definition of AR (to use a model's own predictions to generate further predictions), and we will thus refer to the model implementing as autoregressive. Note we can extend the output voctors include hidden states or context. windows (see A.20), thus covering RNNs and Transformers. However, doing so would require significantly more mathematical machinery leave it for future work (see remark A.21.
3

---

Published in Transactions on Machine Learning Research (10/2025)
To simplify our theory and experiments we will focus on models where the function is implemented by feedforward neural network which we will refer to autoregressive neural network (ARNN), and implement it with multi-layer perceptrons for our experiments (MLP, see appendix F for details about the models).
3.4 Loss functions for ARNNs
Consider time scries of discrete states x = [z(0), ((1),.. x(M) sampled at times 1,, M). We can define a quadratic ARNNs loss as
Cx(0,T)
|a(m + k) fht (c(m).0)]²
K
where K indexes the number of autoregressive steps (k: in eq. required to forecast the state of the system in a temporal horizon T which uses the temporal units of the dynamical system question. Note that the temporal units for depend on the underlying system being studied and can be continuous, while the autoregressive steps K depends on the choice of sampling rate and are natural numbers, and we will usually refer to T in our paper. We will use the Euclidean norm squared derive bounds on errors and losses for the rest of the paper, but other norms could also be used. Note that it is possible to have different temporal horizons (Ts) for training and testing. Unless explicitly stated, we will focus our study on the temporal horizon during training (not testing).
3.5 Theoretical assumptions and model set-up
To make our derivations we make three main assumptions:
We assume that the loss function is smooth, which implies differentiability and we can therefore use gradient descent.
We assume that we have enough data during training. In terms of dynamical systems, this means that we have sample trajectorics that are long enough to evaluate various temporal horizons, and that the data provide comprehensive coverage of the stationary distribution of the dynamical system.
We assume that our dynamical systems are ergodic, mcaning that a long enough trajectory will converge to a stationary distribution in which every point is visited with a fixed probability. We also assume that our systems are fully observable, and deterministic, meaning that every observation corresponds to a unique point the state space of the system from which is possible to predict the next.
Those assumptions allow us to derive our main results, but might not be realized in the real world. In section 4.4 we discuss how and when those assumptions can be relaxed and how to extend our results to other architectures such as RNNs and Transformers.
4 Temporal horizons and loss landscapes
In this section, we derive connection between the loss gradient and the forecast. horizon during training, and exploit it to understand the geometry of the loss landscape. Full proofs of our theorems and their corollaries are presented the appendix A in the main text we only provide an outline. We provide simulations to illustrate all our theorems.
4.1 The building blocks to link dynamics and training loss
To build our theory we need to characterize the region on the parameter space where the model f(; 0) is a reasonable approximation of the true dynamics é. We do so by limiting how far the model can deviate from the true dynamics, which we characterize by the Jacobian matrix the matrix of partial derivatives of the system that captures how the system evolves locally.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Definition 4.1 (e-bounded region). For O, an e-bounded region of the parameter space O for model trained to forecast a dynamical system with a bounded stationary distribution is convex set within the space of parameters of the model such that
I oF,0) (f(,,0) + Fo))) <
vr Sx
where Jo the Jacobian matrix of Φ(-), F a unitary random vector and Sx is the region of the state space where the stationary distribution has non-zero probability.
Remark 4.2. Notice that the definition of e-bounded region is closely related the notion of a ball around a point as it is often used in geometric treatment of dynamical systems Jost (2005). However, the metric is defined through the error of the model, which is, to the best of our knowledge, a new approach. This connection will allow us to leverage the theory of dynamical systems to infer properties of the loss function in the rest of the paper.
We note that e-bounded regions exist for arbitrarily small (lemma A.1 and have bounded loss (lemma A.7 Importantly, a small can only be achieved in models that have been partially trained, so that they are already a good approximation of the system dynamics. Thus, our theory applies to models that have been at least partially trained, not to models at initialization.
Now we can connect the dynamics of the system to the loss of model trained to forecast the system. Consider the function
(0,T()
( (o,
(4)
which represents the relative scaling of the gradient with respect to T. The following proposition and its validation in fig. g shows how g evolves with T. Note that we used 1 as the reference for the gradient magnitude, but any finite value would work.
Proposition 4.3 (Loss gradient growth). Consider dynamical system which is either chaotic, limit cycles, and a corresponding model with an e-bounded region of the parameler space O (m where O are all the balls of radius p < e around the minima of the loss. For a sufficiently small e, when the forecasting horizon T is large, g(T) scales with T as
9(7) = (O(e))) for chaolic/unslable
(5)
((wT), for limit cycles.
Proof sketch. The Jacobian of the system over a time window T will scale exponentially or lincarly with or w. Furthermore within definition 4.1 the Jacobian of the model remains close to the Jacobian of the true system (lemma A.6). The Jacobian of the model is always contained in the expression for VoLx(0, T",, and thus it scales with T. See A.8 for the full proof. Note that we removed the minima in (Or } to avoid zeros,
which would lead to a divergence of g. We will treat them in the next corollary.
An important limitation of proposition 4.3 is that it does not inform us about the minima of the loss. Yet, we can use it to gain insights into the Hessian of the loss around those same minima.
Corollary 4.4. Consider dynamical system which either chaotic or a limit cycle, and a corresponding model with with mēnima an e-bounded region of the parameter space Oc. For a sufficiently small e, when the forecasting horizon T is large, the Hessian oround ménima scales with T as
|H(0,T) T) (O(e)), for chaotic systems || (0,1) 1 (WWL), for limit cycles
(6)
where [[H((,))]] is the sum of the eigenvalues of the Hessian (which in a minima is equivalent to the so-called Nuclear norm).
Proof sketch. As the gradient grows with T, the second derivative must necessarily grow. To make this formal, we consider small balls around the minima, and use the divergence theorem, where the gradient the vector field and the trace of the Hessian emerges.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Food web
Double Pendulum
Limit Cycle
Lorenz attractor
0.03
200
100
0:100
50
0.02
100
0.01
0
4
x
x
x 0.005
30
0.005
0.01 0.015 0.02 0.025
0.5
1.5
3.0
0.015 0.02 0.025
T
T
T
T
Figure 1: Gradient scaling: We measured the relative Lz-norm of the gradients (g(T)) as a function of T for MLPs g(T) during training. As expected from proposition 4.3 we observe an exponential increase of g(T) for the lorenz system and the double pendulum, and linear one for the limit cycle and the food web. Notice that we used the temporal units of the system dynamics as units for the temporal horizon to make our results independent of the choice of timestep size, and used a single prediction the default timestep.
Remark 4.5. Note that proposition 1.3 are a direct consequence of dynamical systems theory, and they have appeared previously in the study of VEGP (see for example Mikhaeil et al. (2022), Thm. and 3). However, link them the geometry of the loss landscape through the Hessian, which we later will extend by leveraging definition 4.1
Corollary 4.4 lends itself to two interpretations:
Classical statistics suggests that having minimum with high Fisher information quantity that scales with the Hessian means that the model extracts a lot of information from the data Ly et al.) (2017). This would suggest that it is better to pick high T as it would make any minima found very precise.
Flat minima seem to better at generalization, because any difference parameters (for example due to differences between training and testing data) will have little effect on the loss Hochreiter & Schmidhuber (1997); Keskar et al. (2016). This would suggest that it is better to pick low T to improve generalization.
Clearly, both intuitions are at odds with each other. To gain further insights we will study the geometry of the loss landscape.
4.2 Generalization across temporal horizons
A key problem in comparing models trained with high and low T is that different forecast horizons correspond to different forccasting probloms, and therefore we should not directly compare their minima in terms of their loss or forecasting performance. Instead, we need a metric that directly relates the performance a model trained one temporal horizon
performance in another temporal horizon.
We thus start by asking whether minima found for given time horizon will work for another time horizon. We formalize this notion as the ratio between two minima for different time horizons,
Lx(0m Th) Lx (O0inn Th) (x Ti) Lx (8 T))
(Th, TI)
where the parameter values gmin gman are two minima found with losses evaluated at Ti and Th respectively with Ti < Tí (l stands for low and & for high).
Note that different temporal horizons could have different numbers of minima and those might be clustered in different regions of parameter space, and therefore we need to restrict to comparable minima. Thus, will

---

Published in Transactions on Machine Learning Research (10/2025)
focus specific pairs (f _mmin), where if the model is initialized with parameters f and trained with the temporal horizon Ti then it would converge to O and conversely, if the model starts with parameters 0miin, it would converge to 0g if trained at Th.
Theorem 4.6 (Minima with long forccasting horizons generalize to lower horizons). Consider model f(., ()) with 0 in an e-bounded region. We assume the eristence of lwo minima Ogin and O for the losses (0, Ti) and Cx (0, Th) respectively with Th > Ti such that t would converge to 0gin if trained on Th, and versa. Then the difference in the change in losses follows
(T Tι) - (O(ATT-T)), for chaotic/ unstable (OwW(T)-1) Ti)), for limit cycles
(8)
Proof Sketch. We can follow the gradient of the loss for Th or Ti between the two minima. The change in loss is the integral of that gradient, which scales as proposition 4.3 Taking the ratio of the aforementioned change loss yields our result. See A.10 for the full proof.
Food web
Double Pendulum
Limit Cycle
Lorenz attractor
20
10
200
(T..TT)
10
5.0
100
*
8
2.5
X
2
8
14
20
28
0.005 0.01 0.015 0.02
0.5
1.0
1.5
2.0
2.5
0.005 0.01 0.015 0.02
T, T
T,
T, TT
T, TT
Figure 2: Performance ratios for longer temporal horizons: We measured the ratio of the difference between losses on connected minima found at different temporal horizons r(T,. T)), and plotted it against the difference between those temporal horizons. We computed the loss by training a single model for Ti corresponding to one forecasting step, and then trained that model further with higher Th. As expected from theorem 4.6 we observe an upward trend for all the systems, with the limit cycle seemingly linear, the double pendulum and food web being similar to an exponential and the Lorenz attractor being somewhat inconclusive. Notice that here the difference in losses is implicitly linked to the accuracy of the minima we find, which depends on the optimizer and the randomness of the system, thus is expected that we do not find a perfect match to our theory.
Intuitively, the Theorem rellects the fact that long temporal predictions rely on shorter predictions, implying that models that are able to make predictions over long temporal horizons must also make good predictions over short time horizons. However, the converse is not necessarily true, and valid short-term predictions might diverge in the long run, suggesting that 0min might. belter capture the global dynamics of the system than gmin.
4.3 Loss landscape roughness and the temporal horizon
Although theorem 4.6 would suggest that we should train models with a long T, we need to beware that such training might be extremely hard. We quantify this through a notion of roughness, which we take as the number of maxima and minima of the loss zTT,0,, found over line in parameter space 0,02, and prove that it scales with the temporal horizon in thcorem 4.7 which is validated empirically fig. 3
Theorem 4.7 (Loss landscape roughness). For any two points 0,, O2 in an e-bounded region of the parameler space that are not in a connected region of zero loss the number of minima and marima along the line
1 such scenario not covered in the assumptions on proposition 4.3 and in this case there learning

---

Published in Transactions on Machine Learning Research (10/2025)
Food web
Double Pendulum
Limit Cycle
Lorenz attractor
50
100
200
20
100
25
0
0
0 x
X
0.005 0.01 0.015
0.025
0.5
1.5
2.5
3.5
0.005
0.01 0.015 0.025
T
T
T
T
Figure 3: Loss landscape roughness: We counted the number of minima and maxima on cross-section between two parameter sets 0,, 0g achieving low losses, for different values, denoted as =(T). To find 0,, 02, trained single model with one forecasting step (k = 1), and we took late training stages which had low loss but with training epochs in-between ensure that there is a distance between the two. As expected from theorem 4.7
clear increase in the number of zeros found. Note the number of minima and maxima that we can detect is limited by how finely we can discretize the line between 01, and 02, and since we have to evaluate the loss at each point in that line, our results underestimate the number of cross-section minima. This explains the saturation seen the Double Pendulum and the Lorenz attractor which is not. expected from our theory.
segment that connects them will grow as
[0(e7 02ll), for chaotic/ /unstable -TT,01.0) - -
(9)
02ll), for limit cycles
Proof Sketch. Consider two arbitrary elements 02, 02 in an e-bounded region. We consider the variation in the loss function through a line segment / connecting 02 and 02, which takes the form an integral dependent on 1x, T)). This derivative is the projection of Vx (1, onto ū the unit vector the line segment, and thus grows with as shown in proposition 1.3 As T grows, the loss variation grows beyond the maximum possible loss Lmax, thus it must go up and down, and hence have minimum maximum. By letting T grow further, it will have more and more minima and maxima. See A.12 for the full proof.
]
Remark 4.8. Notice that the minima and maxima in this theorem are defined as local minima in the line between 81 and 82, not local minima of the loss function in its full dimensionality. It would possible to have the same number of local minima as 7 grows. For example, in valley with single minima winding around the line segment. The theorem does refer to local minima in single parameter training (see section 5.3).
A curious corollary from this thcorem shows that as the temporal horizon grows towards T 0o, the loss landscape becomes fractal (corollary A.13).. Since fractal is non-differentiable this result implies that gradient descent methods would inevitably fail. In the more realistic case of T < co, theorem 1.7 simply states that the loss landscape becomes harder to navigate as T grows.
4.4 Limitations, generalizations and implications
To develop theory assumed that forecasting is applied to fully observable systems that are deterministic and where we have data covering all the relevant state space. Such assumptions make our theory concise and amenable to experimental verification, but are often invalid in realistic scenarios. In this section we discuss how and when our assumptions can be relaxed.
First, we acknowledge that some assumptions such as ergodicity, stationarity and broad state coverage are fundamentally linked to statistical learning, and thus cannot be avoided in mathematical treatment of forecasting (see appendix B
8

---

Published in Transactions on Machine Learning Research (10/2025)
In partially observable dynamical systems, it might be impossible to make forecasts from single observations (ex: we cannot. infer the speed of an object from a single observation of its position). However, multiple delayed observations can contain enough information to recover the full state of the system Takens (1981b a), as often done for Transformers in forecasting tasks (Nic et al., 2022) (sce lemma A.16 but also in classical deep neural nelworks Deco & Schürmann (2012). Another approach is to use hidden states, which can keep information from past obscrvations recover the state, as done in RNNs (sce lemma A.18) Extending our theory those architectures would thus relax this assumption (see proposition A.20
A more fundamental problem is that assume deterministic dynamics, but real data are often stochastic. Yet, stochasticity is not an all-or-nothing phenomena, as there are systems that are very stochastic, and systems that are almost deterministie. Our theory should partially hold for systemss with sufficiently small noise, but not for systems that are too noisy to be predicted even with T = 1 in the first place. More specifically, our theory remains approximately valid within a time window given by the noise level and the system dynamics (sec proposition A.14), after which the system is too random to be forecast. We verify this in the ecology model, where we observe that indeed the noise limits the optimal forecasting horizon (see appendix Gb.
Finally, we discuss the practical implications of our theory. Our loss surface analysis suggests that incrcasing T will lead to better minima by theorem 4.6 However, as a result. of theorem 4.7 we would expect that a large value of T will render learning impractical. In the next section we explore in more depth this trade-off, how applies to real data, and other related implications.
5 Practical insights
To connect our theory with practical problems, we (1) investigate the effects of the temporal forecasting horizon on the performance of ARNNs for forecasting the synthetic dynamical systems presented in the previous section and on three empirical datasets, (2) discuss challenges in choosing the training temporal horizon T and the factors that affect it, and (3) place our thoory the context of fitting the parameters of mechanistic models.
5.1 The temporal horizon as an hyper-parameter
We trained residual MLPs with different training temporal horizons for the four dynamical systems presented in appendix C until they appeared to reach convergence or large total wall time cutoff. We evaluated their performance in forecasting using the mean square error (MSF), which corresponds to the testing loss in eq.
Based on our theory, we would expect to observe the following set of behaviors: (1) an initial increase in the performance of models with larger prediction horizons (theorem 4.6b but (2) worse performance for models trained on time horizons sufficiently long such that the loss landscape becomes harder to navigate (theorem 4.7). We find the expected U-shaped curve all the dynamical systems considered, as seen fig. 4 Importantly, the optimal training horizon neither one step nor the time horizon that we wish to use for forecasting.
To evaluate her our theory applies beyond the original synthetic dynamical systems, we trained similar residual MLPs on three real-world datasets:
The ClimSIM dataset (sce appendix D for details), which is complex deterministic simulation of weather patterns.
The National Oceanographic Atmospheric Administration Sea Surface Temperature dataset (NOAA
SST, Huang et al. (2021) )) which comes from real world measurements and thus contains noise.
The AMZN (Amazon) stocks from 1997 to 2017 (Kouroupetroglon, 2019 which contains a short dataset. that is noisy and non-stationary.
9

---

Published in Transactions on Machine Learning Research (10/2025)
Food web
Double Pendulum
Limit cycle
Lorenz attractor
10-2
3 10
2.00
10
tm0.01
k=0.5
t-0.04
t=4.00
t=0.01
t=1.00
t=0.08
10-3
6.00
t=0.01
4=1.5
t=0.12
8-0.0
t-0.02
2-000
10
t--0.16
MSE
k= 1=10.00
5m0.03
10-4
2.5.0
1.20.00
10-3
10-3
10-5
10-6
6x 10
10-1
10
20
0.01
0.02
0.03
2
4
6
0.05 0.10
0.15
0.20
Horizon in training (T)
Horizon in training (T)
Horizon in training (T)
Horizon training (T)
Figure 4: Performance of residual MLPs trained predict four dynamical systems with MSE the y-axis and the temporal horizon T used in training on the z-axis. For each model, we look the minimum validation MSE. Each line in the figure is associated with given evaluation horizon marked by Ti. Each point range error bars) trained with an increasing training horizon T when evaluated Ti time into the future. That is, if we have a model f(x, (r) trained on T, the points above T on the graph are ([)) - f (ax()) 0r) where ny is the number of auto-regressive steps to get to 7i. As expected, the loss grows with Ti, the loss is convex, and the optimal predictive horizon for training rarely coincides with T.
goes along the line represents the median performance of model (with interquartile
ClimSim
NOAA SST
AMZN stock
1.3e
10x100
1.2e
1.7
MSE
8e"
1.1e*
6e
15x
1.0e
14x10
20
20
40
60
80
100
5
Horizon in training in minutes
Horizon in training (T) in days
Horizon in training (T) days
Figure 5: Performance of various architectures trained with different training horizons on the ClimSim, NOAA SST, and AMZN stocks datasets. We evaluated the average performance time steps into the future for both tasks, which correspond to 50 days in the case of SST and 100 minutes in the case of ClimSim. We plotted the median of the MSE with upper and lower quartiles. The loss is convex with respect to the temporal horizon in training, and the optimal training horizon is not the evaluation horizon. Note that the AMZN stock shows high forecasting variability, expected from noisy, non-stationary time series
Together, the three datasets consist of examples where dynamics fulfill all of our original assumptions but are very complex (the ClimSIM dataset), dataset that well behaved (stationary, ergodic, full coverage) but noisy (the NOAA SST datasot), and dataset where our core assumptions are not valid (the AMZN stock dataset; financial data considered non-stationary, and the range of stock values goes beyond those in the training data). In all cases we find that the optimum training horizon is longer than single step-ahead.
Crucially, the optimal training time horizons are not equivalent to the testing horizon, further suggesting that the optimal training horizons are inextricably linked to the system's dynamics ral her than the desired prediction horizon. This was found not only on our simple dynamical systems, but also on more complex benchmark datascts on climate modeling (an adapted version of the ClimSim Yu al. (2023), the sea surface temperature data National Oceanic and Atmospheric Administration Huang et al. 20021 and the Amazon stock prices Kouroupetroglou (0019))
10

---

Published in Transactions on Machine Learning Research (10/2025)
Food Web
Double Pendulum
Limit Cycle
Lorenz Attractor
relelve LOSS
relative Loss
relative Loss
0.50
rolativo
0.50
0.25
10
0.25
0.25
0:5
0.0
0.00
0.00
2.00
-1.0
-0.
0.5
1.0
-0.55-01
0010
3.05
00
0.05
0.30
0.25
-0.6
4.
0.
0.2
0.4
6.8
-2
f1 Tt
mi mi*
d- g*
Figure 6: Normalized values of (0, T) from eq. (3) when altering one dimension of for different values of T in several dynamical systems models. The loss is normalized for each T so that its maximum value is 1. For all the systems, the loss is generally flat with single minima for low T and becomes steeper and has more minima for large T.
As we show the existence of a non-trivial optimal T for both synthelic and realistic datasets, the next natural question is how to find it. We discuss this problem in the next section.
5.2 Optimal temporal horizons
Our theory suggests that the upper limit on T (theorem 4.7 is sct by the roughness of the loss landscape. While it is possible to find good minima in rough loss landscapes, but doing so requires significant computational resources. Thus, increasing the computational resources should lead to higher optimal temporal horizons. We tested this in appendix H for the ecology system, finding that the computing time used for training does indeed correlate with the optimal temporal horizon for training T.
This finding is problematic, because temporal horizon T is only optimal for given compute budget, making naive hyperparameter search methods very expensive. For example, grid search methods usually train models with different parameter values and a limited amount of computational resources, and based on the results select the best combination of hyperparameters for a longer training. However, this process becomes vory expensive if the hyperparameter depends on the computational resources, because the optimal hyperparametens would only be found each step in the hyperparameter search uses a similar amount of computational resources as in the final training after the hyperparameter search.
A potential solution would be to use an itcrative algorithm that slowly increases T while decreasing the learning rate. To lest this, we designed a simple scheme following this logic, (see appendix D and applied it to the synthetic datasets, (see algorithm D. The scheme did improve the performance over the one-step prediction, and it outperformed even a fixed optimal 7 for limit cycles. In more chaotic systems, the scheme performs worse than an optimal fixed temporal horizon, but it is important to notice that the optimal temporal horizon is not known, so the comparison not fair to the iterative algorithm.
5.3 Loss landscape geometry for mechanistic models
Even though we focused on neural networks, AR models include mechanistic models such as differential equations, where the processes driving the dynamics of the system under study are known and already encoded the model's structure, but where the parameters of the model are unknown. Here we extend the insights form our theory to this sctting
We used the same systems as in the theory section, but this time used the models described in appendix C both to generate time series data and as models to train, where the parameters are unknown. We study the geometry of the forecasting loss from eq. 3 in fig. 6 where we changed a single parameter per model and evaluated the loss function al multiple temporal horizons. Notice that the roughness on the loss only appears for very high forecasting horizons T with respect to the ones we saw in the previous sections. This happens because the models here are very similar to the models generating the data, thus it takes much longer for the divergence to become high enough to affect training. more complete view over all parameters is presented for the Lorenz system in appendix F
11

---

Published in Transactions on Machine Learning Research (10/2025)
Since the models that train are the same as the ones used to generate the data, all models achieve loss of wero when the parameters of the model are the same as those that generated the data. Thus, the generalization ratio r(Zi, Ii) in theorem 4.6 is undetermined because both long and short I provide a loss of zero, implying that a short T generalizes as well as a long T for the true optima. In contrast, we observe that the loss landscape becomes rougher as grows, in line with theorem 4.7 and for extremely large T the loss is similar across almost all parameters, as per lemma A.5
Thus, our theory would suggest that it is always better to train with the smallest possible T, when the system is known and perfectly deterministic. However, this statement should be contrasted with the practical limitations of mechanistic models, which are often not perfect, and real observations, which are contaminated with noise. Indeed, empirical studies considering noise and imperfect models suggest that there is a trade-off in the choice of T similar to the one we found with MLPs (Pisarenko & Sornette 2004 Boussange et al., 2024.
6 Conclusion and outlook
Our work establishes a fundamental connection between the temporal horizon T of an autoregresive model and the geometry of the loss landscape. From a practical standpoint, we show that common training paradigms such as single step prediction or matching the testing horizon-an are rarely optimal. Instead, the ideal T depends on the system's 's intrinsic dynamics (e.g., Lyapumov exponent A), but practical considerations like model capacity, mumber of epochs, lcarning rate and stochasticity also play a rolc. Although our thcory considers Markovian systems, the general principles laid out should hold for more complex systems with long-term dependencies and associated architectures such as RNNs and Transformers.
Our findings motivate principled approach hyperparameter scarch, where T is selected based the system's 's dynamical properties, or updated during training. Future works should investigate this further, for example by jointly optimizing T and the learning rate by leveraging loss landscape curvature, building on idcas from Roulet et al. (2024). Such an approach could extend beyond static horizons: many systems exhibit time- sane-varying predictability (e.g., intermittent chaos or periiiiirio Zhou et al. (2006)), where a dynamic T could sharpen the trade-off between learnability and generalization. For instance, shorter horizons might. allow training on unpredictable regions, while longer horizons can refine learning in more ordered regions. This could unlock more efficient training for complex, non-stationary systems.
References
Anna Åkesson, Alva Curtsdotter, Anna Eklöf, Bo Ebenman, Jon Norberg, and György Barabás. The importance of species interactions in eco-evolutionary community dynamics under climate change. Nature Communications, 12(1):475, 2021.
Nishanth Anand and Doina Precup. Preferential temporal difference learning. ariio preprind trXiv21066005506 2021.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pp- 1120-112 1128. PMLR, 2016.
Ozgur Aydogmus and Ali Hakan Tor. A modified multiple shooting algorithm for parameter estimation in odes using adjoint sensitivity analysis. Applied Mathemaliccs and Compulation, 390:125644,, 2021.
Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael Mahoney, Forecasting sequential data using consistent koopman autocncoders. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 475 485. PMLR, 13 18 Jul 2020. CRL httops:/pproceednngg mlr pttps//v11//eeenotttaaea
html
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent. is difficult. JEEE transactions on neural networks, 5(2):157-166, 1994.
Elisa Benincà, Klaus D Jöhnk, Reinhard Heerkloss, and Jef Huisman. Coupled predator precy oscillations in chaotic food web. Ecology letters, 13677311377-1338 2009.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Dawid Bernaciak and Jim E Griffin. A loss discounting framework for model averaging and selection in time series models. International Journal of Foreensting, 2024.
Hans Georg Bock. Numerical treatment of inverse problems in chemical reaction kinetics. In Modelling of Chemical Reaction Systems: Proceedings of an International Workshop, Heidelberg, Fed. Rep. of Germany, September 1-5, 1980, pp. 102-125. Springer, 1981.
Victor Boussange, Pau Vilimelis Aceituno, Frank Schäfer, and Loic Pellissier. Partitioning time series improve process-based based models with machine learning. bioRric, 2024. doi: 1111110022.0..5555 biorxiv. https://www wwwpxxxiwwiiixix..o/o/ooooo org/ coattent/eallyy/244/////////////
URL
07.25. 201366
Michael Brin and Garrett Stuck. Introduction to dynamical systems. Cambridge university press, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricrmn: A dynamical system view on recurrent neural networks. arXiv preprinl Xiv: 1et.199929.999, 2019.
Nathan Cormille, Marie-Francine Moens, and Florian Mai. Learning to plan for language modeling from unlabeled data. Xiv preprint arXi::244.006614 2024.
Gustavo Deco and Bernd Schürmann. Information dynamics: foundations and applications. Springer Science & Business Media, 2012.
Kenji Doya et al. Bifurcations in the learning of recurrent neural networks 3. learning (RTRL), 3:17, 1992.
Gerald A Edgar and Gerald A Edgar. Measure, topology, and fractal geometry, volume 2. Springer, 2008.
Roland England. Some Examples of Parameter Estimation by Multiple Shooting, pp. 122-136. Birkhäuser Boston, Boston, MA, 1983. ISBN 978-1-4684-7322-77 doi: 111100/977--14484-7322 //doi. wttpg/10..007////000 100.0000000700---1-000 10811777778---44473---------
URL https
N. Benjamin Erichson, Michael Muchlebach, and Michael W. Mahoney, Physics- informedd autoencoders for lyapumov-stable fluid flow prediction, 2019.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006. 12070, 2020.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint Xix:222404. 19737, 2024.
Ian Goodfellow. Deep learning, 2016.
Mukul K. Gupta, Kamal Bansal, and Arun K. Singh. Mass and length dependent chaotic behavior of a double pendulum. IFAC Proceedings Volumes, 47(1):297-301, 2014. ISSN 1474-6670. doi: attps://doi. org/10.3188 4031-01-333333333222 $1474667016326714 3rd International Conference on Advances in Control and Optimization of Dynamical
www.seeeienccccc..cccmmmm.... URL https: hittts://wwyy http:://wwwwwswwneeeeee.oo
Systems (2014).
Alan Hastings. Structured models of metapopulation dynamics. Biological Journal of the Linnean Society, 42 (1-2):57-71) 1991.
Frederik Heetmeyer, Marcin Paluch, Diego Bolliger, Florian Bolli, Xiang Deng, Ennio Filicicchia, and Tobi Delbruck. Rpgd: A small-batch parallel gradient descent optimizer with explorative resampling for nonlincar model predictive control. In 2028 IEEE International Conference on Robotics and Automation (ICRA), pp. 3218 1224.. IEF., 2023.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Lukas Hewing, Kim P Wabersich, Marcel Menner, and Mclanic N Zeilinger. Learning- based model predictive control: Toward safe learning in control. Annual Review of Control, Robolics, and Autonomous Systems, 3 (1):269 296, 2020.
S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
John Theodore Houghton, YDJG Ding, David J Griggs, Maria Noguer, Paul J van der Linden, Xiaosu Dai, Kathy Maskell, Colin A Johnson, et al. Climate change 2001: the scientific basis, volume 881. Cambridge university press, 2001.
Boyin Huang, Chumying Liu, Viva Banzon, Eric Freeman, Garrett Graham, Bill Hankins, Tom Smith, and Huai-Min Zhang. Improvements of the daily optimum interpolation sea surface temperature (doisst) version 2.1. Journal of Climate, 34(8):2923 2939, 2021. doi: (0-1177/JCLI-D-20-000 //journals ametsoc.
20-6166.1.. URL https:
2ttpa/////a/aa///////000-----0-0 --L--//--C--------------- xm1
Jef Huisman and Franz J. Weissing. Biodiversity of plankton by species oscillations and chaos. Nature, 402 (6760):407 410, 1999.
Sajid Iqbal, Xizhe Zang, Yanhe Zhu, and Jie Zhao. Bifurcations and chaos in passive dynamic walking: A review. Robotics and Autonomous Systems, 62(6):889-909, 2014.
Jürgen Jost. Dynamical systems: ezamples of complex behaviour. Springer Science & Business Media, 2005.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocodal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint NK09.1609.0488 2016.
Patrick Kidger. On Neural Differential Equations. PhD thesis, University of Oxford, 2021.
Aaron Klebanoff and Alan Hastings. Chaos in three species food chains. Journal of Mathematical Biology, 32: 427-451, 1994.
Dmitrii Kochkov, Janni Yuval, lan Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Klöwer, James Lottes, Stephan Rasp, Peter Düben, Sam Hatfield, Peter Battaglia, Alvaro Sanchez- Gonzalzz, Matthew Willson, Michael Brenner, and Stephan Hoyer. Neural general circulation models for weather and climate, 2024.
Nikolaus Kouroupetroolon. Amzn (amazon) stock prices 1997 to 2017, 2019. URL https://ww/ h/gg::///wwkgggeeee ctt/aatteett/ppaaaeeea///aaaaaaa
series-datasets
Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar. Fourcastnet: Accelerating global high- resolution weather forecasting using adaptive fourier noural operators. In Proceedings of the platform for advanced scientific computing conference, pp. 1-11, 2023.
Dihui Lai and Bingfeng Lu. Autoregressive model for time series as a deterministic dynamic system. Predictive Analytics and Futurism, 15:7-9, 99, 2017.
Remi Lam, Alvaro Sanchez-Gonzaler Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Faton-Rosen, Weihua. Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia. Graphcast: Learning skillful medium-range global weather forecasting, 2023.
RB Levien and SM Tan. Double pendulum: An experiment chaos. American Journal of Physics, 61((1): 1038-1044 1993.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koopman
operators for model-based control. In International Conference on Learning Regresentations, 2020. URL
https://openrevier httpp//rwwrit..eeeoooooooooooaaa
Phillip Lippe, Bas Veeling, Paris Perdikaris, Richard Turner, and Johannes Brandstetter. Pde-refiner: Achicving accurate long rollouts with ncural pde solvers. Advances in Neural Information Processing Systems, 36:67398 67433, 2023.
Edward N. Lorenz. Deterministic uonperiodic flow. Journal of Almospheric Sciences, 20(2):130 141, 1963. (1000099962220011.0D7
doi: https//doi.org///
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmakers. A tutorial on fisher information. Journal of Mathematical Psychology, 80:40-56 55, 2017.
B.B. Mandelbrot. The Fractol Geometry of Nature. Einaudi paperbacks. Henry Holt and Company, 1983. ISBN 9780716711865. URL http::///bboos.oooooooeee.bbbbaa
Noah Marti. URL ChaotieInference- - Julia library for parameter inference specialized for chaotic systems., 2024.
Kevin McCann and Peter Yodzis. Biological conditions for chaos in three-species food chain. Ecology, 75 (2):561-566 1644, 1994.
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns, 2022.
Jack W. Miller, Charles O'Neill, Navid C. Constantinou, and Omri Azencot. Eigenvalue initialisation and regularisation for koopman autoencoders, 2022.
Meenal V Narkhede, Prashant P Bartakke, and Mukul S Sutaone. A review weight initialization strategies for neural networks. Artificial intelligence review, 55(1):291 322, 2022.
Yuqi Nie, Nam Н Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-lerm forecasting with transformers. ICLR, 2022.
A Emin Orhan and Xaq Pitkow. Improved memory in recurrent neural networks with sequential non-normal dynamics. orXiv preprint arXin:1905.1991 2019.
R Pascanu. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.506 2013.
Fotios Petropoulos, Daniele Apiletti, Vassilios Assimakopoulos, Mohamed Zied Babai, Devon K Barrow, Souhaib Ben Taieb, Christoph Bergmeir, Ricardo J Bessa, Jakub Bijak, John F. Boylan, et al. Forecasting: theory and practice. International Journal of Forecasting, 38(3):705 871, 2022.
V. F. Pisarenko and D. Sornette. Statistical methods of parameter estimation for deterministically chaotic time series. Physical Review E, 69(3):03622, March 2004. ISSN 1539-3755. doi: 10.100//PiysReee..6..00033 09..2.1...
David M Post, Michael L Pace, and Nelson G Hairston Jr. Ecosystem size determines food-chain length in lakes. Nature, 10446799))0047-11444, 2000.
Iaa Price, Alvaro Sanchez- Gonzalzz, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Fwalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Probabilistic weather forecasting with machine learning. Nature, 637(8044):84 90, 2025.
Christopher Rackauckas and Qing Nie. DifferentiallEquatiinnjjj performant and feature-rich ecosystem for solving differential equations in Julia. Journal of Open Research Software, 5(1), 2017.
Vincent Roulet, Atish Agarwala, Jean-Bastien Grill, Grzegorz Swirszez, Mathieu Blondel, and Fabian Pedregosa. Stepping on the edge: Curvature aware learning rate tuners. arXiv preprint 0r11:2407.06185. 2024.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Ayumu Sasagawa, Sho Sakaino, and Toshiaki Tsuji. Motion generation using bilateral control-based imitation learning with autoregressive learning. TEEE Access, 9:20508-222 205200, 2021.
Dominík Schmidt, Georgia Koppe, Zahra Monfared, Max Beutelspacher, and Daniel Durstewitz. Identifying
nonlinear dynamical systems with multiple time scales and long-range dependencies, 2021. URL https
wwtpiivoorg/bbss. wwwp.r/////g...o..////.. 03471
Richard Sutton. Td models: Modeling the world at a mixture time scales, In Machine learning proceedings 1995, pp. 531-539. Elsevicr, 1995.
Floris Takens. Dynamical systems and turbulence. Worwick 1980, pp. 366 381, 1981a.
Fq Takens. Lecture notes in mathematics. by DA Rand and L.-S. Young Springer, Berlin, 898:366, 1981b.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29, 2016.
Norbert Wiener. Generalized harmonic analysis. Acta mathematica, 55(1):117 258, 1930.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model-based reinforcement learning, In 2017 IEEE international conference robotics and automation (ICRA), pp. 1714-1721. IEEE, 2017.
Alan Wolf, Jack B Swift, Harry L Swinney, and John A Vastano. Determining lyapunov exponents from a time series. Physicn D: nonlinear phenomena, 16(3):285 317, 1985.
Sungduk Yu, Walter Hannah, Liran Peng, Jerry Lin, Mohamed Aziz Bhouri, Ritwik Gupta, Björn Lütjens, Justus Christopher Will, Gunnar Behrens, Julius Busecke, Nora Loose, Charles I Stern, Tom Beucler, Bryce Harrop, Benjamin R Hillman, Andrea Jenney, Savannah Ferretti, Nana Liu, Anima Anandkumar, Noah D Brenowilz, Veronika Eyring, Nicholas Geneva, Pierre Gentine, Stephan Mandt, Jaideep Pathak, Akshay Subramaniam, Carl Vondrick, Rose Yu, Laure Zanna, Tian Zheng, Ryan Abernathey, Fiaz Ahmed, David C Bader, Pierre Baldi, Elizabeth Barnes, Christopher Bretherton, Peter Caldwell, Wayne Chuang, Yilun Han, YU HUANG, Fernando gglesias-Suaree, Sanket Jantre, Karthik Kashinath, Marat Khairoutdinov, Thorsten Kurth, Nicholas Lutsko, Po-Tun Ma, Griffin Mooers, J. David Neelin, David Randall, Sara Shamekh, Mark A Taylor, Nathan Urban, Janni Yuval, Guang Zhang, and Michael Pritchard. Climsim: A large multi-scale datasct for hybrid physics-ML climate emulation. In Thirty-seventh Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2023. URL htttps://openreview..r net/
dorum?1d==W11991xq)
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu, Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaic. Why gradient clipping accelerates training: A theoretical justification for adaptivity. ICLR, 2020.
Shangtong Zhang and Richard Sutton. A deeper look at experience replay. Xix preprint arxiv:1722.00 2017.
Bin Zhou, Li Gao, and Yu-Hong Dai. Gradient met with adaptive step-sizes. Computational optimization and applications, 35:69-80 86, 2006.
16

---

Published in Transactions on Machine Learning Research (10/2025)
A Proofs for propositions and theorems
In the following subsections, provide full proofs or sketches for all of the non-trivial mathematical claims made in the main text. Note that these subsections have been arranged so that the relevant theorems or propositions follow the same order as in the main text.
A.1 Existence of neural network for epsilon-regions
Lemma A.1. For any c > O and any real p E [0, 1], there erists number of observations M. and sufficienlly large newral network that has an e-bounded region with a probability p
Proof. By ergodicity, for sufficiently large M, the distribution of samples will converge to the stationary distribution. At some point there will be enough points in the (bounded) stationary distribution to guarantee that all points have a neighbour at a distance of e with a probability larger than With a sufficiently large neural network, we can set the parameters such that every point x in the neighborhood of x(t) gets sent to a(l + 1). By construction, this will always map f(x + <ř, 0) onto either f(x) or a close neighbour which will, by the compressive nature of the projection, fulfill the conditions of our definition.
Lemma A.2. In a multi-layer perceptron with at least one hidden variable, the training loss that can be achieved decreases with the size of the hidden layers.
Proof. Consider a multi-layer perceptron with at least one hidden layer and N neurons per hidden layer and trajectory length M in a system with D variables where the MLP achieves a training error per sample of e(m) = [=x(m) - f(x(m 1,0) Then add D D+1 neurons the first hidden layer and one neuron any subsequent layers. We can create bounding box with the extra D neurons for the sample Mmax corresponding to the maximum e(mmax) and have neuron in any subsequent layer whose only task is to transfer that signal to the output layer, correcting the error.
Lemma A.3. For stochastic dynamical system of the form i d(x) E, where
I1f( + (f () Jz(x)F()l <
(10)
and & is a vandom random vector such that Pr[|||| b) < a,
Pr + J < < ~² + bb <a Vx E Conv(x),
Proof. We are simply adding the noise term to the definition of e-bounded region, and note that since the noise is unrelated to the dynamics, the errors add up.
Remark A.4. If b is small with approaching one, then the system is effectively deterministic in the short term horizon.
Lemma A.5. Consider model that approzzimates dynamical system that is not stable and has a bounded state space in its stationary distribution. For an initial position z(0), the forecasting error as T + oo behaves as a random variable with erpected value.
Ex(o) (I((((<(0))))))))))))))(>))))( xx (T(a()) -11 {0 (Varx [x]) If the model is not strictly perfect
If the model perfectly predicts the system
(11)
where x (T))(0)) is the state of the system at time T for a starting position x0) and V [x] is the variance of the state space.
Proof. The gist of our argument is that as the model and the system are ergodic, their difference is also ergodic (if they are different) or non-cxistent (if the model perfect). Thus, for infinite time horizons the error in the forecasting follows a 0-1 law. If the model perfectly captures the dynamics, the error is clearly 0. If the model is not perfect at any region in the stationary distribution of the dynamical system, there will
17

---

Published in Transactions on Machine Learning Research (10/2025)
be perturbation of the trajectory. Such perturbations will not fade away since system is not stable. Because the system is bounded but the trajectory is infinitely long, any region will be visited infinite times. Thus the error in the trajectory will accumulate. Eventually, any correlation between the model and the dynamical system will disappear. By ergodicity, the model and the system will generate points that are effectively independent with a variance of order Varx [a]. The expected value of the difference belween two random variables the sum of their variances, and since the model is similar to the system, the variances are within the same order of magnitude.
A.2 The model captures the dynamics
Lemma A.6 (Approximate Jacobian). For + O, in an e-bounded region of the parameter space the Jacobian of the model converges to the Jacobian of the true dynomics
lim 2
- -
(12)
where Illor is the absolute value of the largest eigenvaluue of the matrix (also known the operator norm).
Proof- By using the equation in the definition of e-bounded region,
[] + Ver. (f(z, Jo(x) ve) - Jo((=)FVEll
and taking the Taylor expansion,
[I Vor, - f(z, [ - Jol))/c
which converges to an equality in the limit e 0, yielding
[J Fo(=)FV[ll ( Jo(x)Al <E
J)(())- <ve
A.3 Bounded loss
Lemma A.7 (Bounded loss). In an e-bounded region of the parameter space, the loss is bounded by
C(0,T) <
max (1..
|| - x2112 1021-2112++2 22
L.axx
VT
(13)
Proof. For any dynamical system with bounded state space, there is a maximum distance bet its two farthest points. Since the model itself is also at maximum distance from any point in the original system,
C(,,T) <
max |m - 22 <
max || Ex.
x2112 + = Lmax,
VT
(14)
#1 EX
Note that for 0, the upper bound is the variance of the ergodic distribution of the system.
7
A.4 Relating gradients and temporal horizons
Theorem A.8 (Unbounded loss gradient). Consider dynamical system which is either chaotic, contains locally unstable trajectories contains limit cycles, and corresponding model with an e-bounded region of the parameter space θ, with non-zero loss. When the forecasting horizon T is large, the expected gradient magnilude grows with T as
VoL T) (oL((8,1) 1) ((wT)) for limit cycles
(O(eT), for chaotic systems
(15)
where ||. is the Fuclidean norm.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. We start by analyzing the loss given in eq- (3), which is function of the parameters 6 given the true trajectory at any time & by s(t). To understand how learning operates under this loss function, we study how the landscape induced by [(,, T) varies under different choices of I', We can get the gradient of eq. (3), our loss function, by evaluating
1
M-K K Σ
(x,
V)L(O,T)
(x +) (z(k),
(16)
M
00
lm(k),
where we must still compute the partial derivatives of the function f For T = 1, we denote the derivatives of f = f(x, 0) as
Jo(æ, 0) = Vol(x,0).
= =(,,,))=V((,,00))
(17)
Using the chain rule, we obtain the expression
Σ (f))
(18)
where (x, = II,- (z), To analyze eq. 16 and thus eq. 3 we can use some knowledge we have about the properties of the state space and parameter space Jacobians involved in eq. (18). Specifically, the state space Jacobian, (x, 0), is directly dependent on the trajectories of the system and indirectly dependent the parameters. The notable implication of this is that any set of parameters where the model captures the dynamics reasonably well, the Jacobian in the state space of the model will be very closcly aligned with the Jacobian of the system dynamics, which is given by the data (see Lemma A.6). In contrast, the Jacobian of the parameters Jo(æ, depends on the model and its parametrization.
Vo T'll for a given by using [o,( 1)]l
We can therefore compute the ratio of gradients,
1 M-K M
Vo[(O,
(1) (f))), (z(m + k) -ht (z(m).
(19)
K
In chaotic systems, we know that the product of Jacobians scales as
lim In (ilu))())))) XT/K 1-00
(20)
where À is the Lyapunov exponent of the system, which is positive (Wlf al. 1985). and T/K is simply scaling factor to convert the forecasting steps into the timescale of the system. Thus, loss behaves
VeC(o, = O (e*T).
(21)
If we apply the same logic to the gradient with time horizon of one (or any other reference length),
Vo,(0, T)ll o(( [V, 1)ll
(22)
We also analyze limit cycles by focusing on the phase of the rotation. trajectory produced by fT 0) where the period is not exactly the same as the original dynamical system will slowly drift as the phases will change move at slightly different angular velocities. In more formal terms, if we parameterize the limit cycle by the phase &p and ignore its support obtain
Σ II ((()()))0)) Jo ((P()))0) - Jo (f(())
1b
(23)
Lm=i+1
19

---

Published in Transactions on Machine Learning Research (10/2025)
where the position is repeated for large (this is a limit cycle), with difference given by the quality of the approximation. To make this simple, we can consider the growth per cycle, and count the cycles,
2x
Jo (1, Tw L. Jo (a, 0) da
J7 ()p0))=c
(24)
where the integral gives the average Jacobian through the cylce, and I 2 Tw is the number of cycles (notice the change of variable the integral).
As such with chaotic behavior and locally unstable trajectories, the norm of the Jacobian grows exponentially with T. For limit cycles, we know that the Jacobian grows at least linearly with wT.
Corollary A.9. For the models considered in A.8 the hessian of the model around its minima grows os [H(0,T) T)Il [H 1)]l O(WT), for limit cycles
((e2)), for chaotic systems
(25)
where || is the Nuclear norm.
Proof. The gist of the proof is to show that as the gradient grows with T, the second derivative must also grow to keep up. We build this connection between the gradient and the Hessian through the divergence theorem.
Consider the gradient of the loss T)) vector field in the space of parameters. Pick minima of the loss opin, and a ball around it denoted by Bagin and its spherical boundary Spmin. Then, by the divergence theorem,
div
VL(,,
(26)
Bagin
Samin
where dš a vector normal to the sphere scaled by a differential surface element, and dV a differential element of volume. Since 8pin minima, for small enough ball the gradient of the loss always pointing towards the inside, hence the product with d$ will have constant sign. Thus,
J Bgmmn
(IV
IVC(0,
(27)
Now, note that the divergence of the gradient is the trace of the Hessian,
J Bamin
Tr (H
IVC(O,T)dS])
(28)
Sami
Now we simply need to remember that the ball was relatively small and the loss smooth, so that we can use the approximation
Tr (O T)] R Tr (Opin T)] vO Boping
(29)
and noticing that the trace is sum eigenvalues of a matrix, which is the muclear norm,
dV [0Tin T)1l" / dV
dv=/ = / Samin
Tr H(pii T)]
[V(()T))d
(30)
Bagin
Bagiu
and if we apply this for T = 1 and divide,
Sg gmmn [VC(0T)455)
H H(ppi T>
(((x)),, for chaotic systems
(31)
(Omin 1)Il*
J Smmin (V((0
O(wT), for limit cycles
where the last step comes from proposition
20

---

Published in Transactions on Machine Learning Research (10/2025)
A.5 Minima with longer forecasting horizons generalize better
Theorem A.10 (Minima with long forecasting horizons generalize to lower horizons). Consider model f(-.0) with 0 in an e-bounded region. We assume the eristence of two minima 0 and hh for the losses (x, Ti) and (x( respectively with Tn > Ti such that 8 would converge to 0h if trained on Th, and vice-versa. Then the difference in the change in losses follows
[O( for chaotic/unstable (ww(TT -T))) for limåt cycles
(T Ti) -
(32)
Proof. Consider the path p of the gradient between the minima @m for the temporal horizon T and another point @basin such that it will converge to Opin through training by gradient descent. On this path we compute the change in loss,
[( ,T) - L(Obasin T) = / (VL(
[V
(33)
JEEP
JAEP
= Jeep
VC 7) 1)
=
VC 1)
d8
(34)
where p( is the unit vector the direction of the path of the gradient, and since the path is defined by the gradient itself, (VC(0, T), ((0, (1, T)I.. By proposition 4.3 the ratio of loss norms in the integral either grows exponentially or linearly with T. By substituting T = T, and Ti then taking the fraction
in eq. 32 we complete the proof.
A.6 Loss landscape roughness
Lemma A.11. Consider a continuous differentiable function g(s) on an interval s E [Sarsp] bounded from above and below,
gmin < g()) < 9max
(35)
with a variation in value
2b
dg(#) ds > 2n(0max Imin); N
0g
(36)
dz
then g(s) has at least minêma and maxima in the interval considered.
Proof. Assume without loss of generality that g is initially increasing. That is, dg(sa) > 0. Further define, ds
(s(s) /-' dg(s) ds
ds
(37)
which we know to be semi-positive monotonically increasing function with range [0,U] where U 2n(Amax Iminn) Consider the first point 21 such that (g(s1) > Imax Imin Assume for contradiction that g does not have a maxima on the interval (Sa, 81]- We know that,
g(s1) = g(8a) f( dg(s')
ds' ds'.
(38)
By assumption, dg(s') ds'
is semi-positive and must not change sign on the interval. As such,
1"1
(")
dg((')
g(s1) = g((a) +
ds' g(8a) + Imax Jmin = g(s1) g(8a) Imax Imin-
(39)
ds'
8n
This implies that g(s1) - g(sa) > Imax Imin: which is contradiction. Therefore on the interval [sa $1] there must cxist a maximum s7 Applying the same logic the interval [st, s2] such that (,22)) > 2(9max Imin) proves there must be a minimum on that interval. Further applying these arguments n more times shows g must have at least minima and maxima [Sa, 86]-
21

---

Published in Transactions on Machine Learning Research (10/2025)
Theorem A.12 (Loss landscape roughness). For any two points 01, B2 in e-bounded region of the parameter space that are not in a connected region of zero los the number of minima and mazima along the line segment that connects them will grow as
z(T) O 110, 02ll).
for chaotic unstable systems O - 02ll), for limit cycles
(40)
Proof. Let us look at arbitrary elements of the parameter space 01,02 O. By lemma A.7 the difference in their loss is bounded,
1)) - [(02,T)) L.
(41)
We now consider the variation in the loss function that goes from 01 to 02 through a straight line,
(2) = _
T
dl,
(42)
1=81
where is the variable that represents points a line from 01 0z. We note that the derivative of the loss is projection of the gradient onto the line,
OC1, T) 01
(Vo[((,)), -00)
(43)
where ūo -0, is the unitary vector of the line going from 0 to 02. By proposition 4.3 the value of this projection grows exponentially a chaotic or locally unstable trajectory. a limit cycle, the growth is or linearly with T. Thus, we only need linear growth in eq. 43 for an unbounded value of er (0100 02). Taking the integral over the line,
OC(L. T)
[TAT 82ll.
for chaotic or unstable systems
dl =
(44)
1=0
[1]10 02l1. for limit cycles
and by lemma A.11 we then have that z(T) must grow with lower bound given by eq. thus completing the proof.
Before continuing need to introduce the notion of box counting dimension (also known Minkowski Minkowski-Bouligandd dimension), which is defined as
dimbox (L(0, = lim log N(e) = log(s-1)
(45)
where (E) is the number of balls of radius e needed to cover a surface (here the loss). smooth surfaces, the box-counting dimension is equal to the topological dimension (the number of dimensions within the manifold), and the number of balls would scale with the number of parameters. However, if the box counting dimension is higher than its topological dimension, the surface is fractal Mandaelbor (1983) Edgar & Edgar (2008). We will use this in the next corollary, but it is also relevant concept in dynamical systemss, for example in Taken's 's theorem which we will use later.
Corollary A.13 (The loss function is a fractal). For an e-bounded region of the parameter space, in the limit + 00, the surface of the loss is a fractal occupying volume that converges to
Var X M
Var xr E +
10.1v
< <10
(46)
M
where 10.) is the area of the e-bounded region and Var X] the variance of the invariant probability distribution in the dynamical system.
2This covered in the assumptions an proposition but if we in such a region there no learning
22

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. The topological dimension is the number of parameters in the loss landscape. For the box counting dimension, we consider cover of the parameter space by infinitesinally small open balls. For each one of those balls, however, the loss function has unbounded gradients and thus we can always have a gradient large enough so that a ball of a radius e would not cover the neighborhood of a point. Hence, we would need the number of parameters plus one dimensions to cover the loss landscape surface. From the dynamics perspective, any parameter setting where [(e, 1) > 0 implics that there is at least some cases where the model deviates from the system. For some high value of t", the deviations accumulate until the position of the model is uncorrelated with the position of the dynamical system observed as z(t*). If T > t", the observations are uncorrelated with the model, and thus we are sampling variable with variance Var [xe E X). We have M samples and a margin of error of e, which gives us the term in the square root. The multiplication by O is simply because we need to consider all the parameter sets. Note that even if some parameters result in a zero-loss, they form surfaces of lower dimensionality and thus have zero volume.
A.7 Effects of noise
Our theory builds on the assumption that the e in proposition 4.3 is small enough so that models with 0 E O will approximate the system dynamics well (lemma (. In stochastic systems the e is bounded away from zero, meaning that in large temporal horizons any model is effectively a random guess (A.5). However, there is often a temporal horizon in which the noise is still small enough so that predictions are possible (A.14)
Proposition A.14 (Temporal horizon limits on systems with noise). In a dynamical system whose dynamics are given by limit cycle chaos, and where there is an extra term inducing random noise with variance uniform in all dimensions and with value o² (per dimension), there is marimum temporal horizon Tmax after which forecasting is effectively random guess given by
In(S) - -In(o)) for chaotic or unstable systems
0
T Tuax
(47)
L
for limit cycles
where L is the length of the limit cycle and the marimum rudius of the state space in the chaolic system.
Proof. The gist is that in any dynamical system where the noise accumulates, at some point the contribution of the noise to the current state of the system is going to be as large as the contribution of the deterministic dynamics. For chaotic system a perturbation of size will scale exponentially the effect of that perturbation is large the state space. For a limit cycle we can apply the same logic, except that the noise grows as To and we use the length of the limit cycle.
ae and when ge^T > S,
]
A.8 Generalization to partially observable systems and recurrent networks
We develop our theory using mostly autoregressive neural networks in systems with full observability. However, we can extend it to the more general settings of non-fully observable models. However, this requires either recurrent networks or models with a context window. In the following, we prove that any neural network that forecasts with high precision needs to have access to a representation of the state of the dynamical system which can be mapped one to one to all the states of the system as it was fully observed. In the dynamical systems literature, this is usually done by using a higher-dimensional representation of the system, which is known as an embedding. Then we show that enhaneing beed-forward networks with hidden states or context. can provide that representation. We note that don't 't believe that those results are new, but we could not find a direct reference, so we add it here for completeness.
The first step is to prove that an arbitrarily small forecast requires a representation of the state of the system that captures it fully. This is effectively the converse AA.6]
Lemma A.15. Consider a smooth deterministic, ergodic, discrete dynamical system, defined by x(m) = (a(m 1)), an observation function h E TRd and a function f that forecasts it with a precision of 2 (If(h(=)) - p(x)]]- For sufficiently small e, d must be higher or equal than the topological dimension of the dynamical system, and in the limit of e = 0, h(x)) must be an embedding of x.
23

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. f maps the potential precursors of each point to prediction that is at a distance of c. We can thus define partition of the dynamical system, where each point is in a ball of radius that contains its prediction. As é goes to zero, this partition gives a unique value to each point in the system. To do so, the topological dimensions need to match. The fact that this converges to an embodding in the limit of small e is straightforward, since every point corresponds to one image of h.
This proves that such architectures need that representation. In the next steps we prove that context windows or recurrent connections in the hidden layer are sufficient to do that.
The gist of our proofs relies on having a neural network that reconstructs the original dynamical system from observations of the current. and past (partial) observations of the system, following Taken's theorem. This can then be replaced by recurrent neural network using a construction that is presented in Deco & Schürmann (2012). Note that for simplicity we will assume a single observable variable per system, but the results are valid for more.
Lemma A.16. Consider a determinislic dymamical system sumpled al fixed time intervals, that arm) = à (x(m- 1)) and an observation function o(x) E IR that is smooth and coupled to every dimension of x such that Bo(x(m)) (m-)mkk # 0 for some k. If there crists neural network 0) such that
I (x(()),) x(m K)11 <ep
(48)
then for every error Ef < Eg there erists a neural network g(. with parameters k. and input dimension d < 2b + 1 where b is the box counting dimension of the invariant manifold of x such that
1)), llg ([o(x(m))). o(x(m - 1)), o(x(m 2))...((x(m k))l xx) o(x(m + K))l
(49)
Proof. Note that the conditions on the dynamical system and the observation function are pre- cisely those of the Taken's theorem Takens (1981a). which states that there is an embedding of (o(x(m))) o( - o(x(( 2)) k)) that captures the full attractor of the original dynami- cal system. As consequence of the embedding, we can construct smooth function which takes lo(x(m)), o(( - 1)), o( 22) - k))l and maps it to x(m).
A function such as v can be approximated with arbitrary precision by a neural network M(31) with parameters 4 by the universal approximation theorem.
Thus, we can use to recover z from o(z), and then compose the results with the original neural network f. giving us g(:, x) (()((o(((m)) o(=(m 1)), o(x(m 2))_..((r(m k))) (), 0) which is composition of the
reconstruction and the original f. The error on each timestep will be composition of the errors induced by h and f.
J
Remark A.17. Note that the transformer architecture fulfills the conditions of the theorem, but the sclf- allention mechanism is known to disregard the temporal information that is crucial for forecasting [Zen et al.)
2023 The current solution is the PatchTST model, which effectively uses this embedding (Nie et al. 2022).
Lemma A.18. Consider discrete deterministic dynamical system defined by z(m) = (x(m 1)) and observation function o(z) E R that is smooth and coupled to every dimension of z. If there exists newral network ((,00) such that
(x(m), 0) x(m + K)112 < <6f,
(50)
then for all cg > Es there exist a recurrent neural network g(,, h, k)) where h is the hidden state of dimension d < 26 + 1 with b being the box counting dimension of the invoriant manifold of x such that if the inputs to g have been (o(x(m)), o(x(m 1)), o(x(m- -())_ k))] then
llg (o(x(m)), h, k) o(x(m + +))I <c
(51)
Proof. We use similar argument as presented in (Deco Schürmann 2012). We build a hidden layer with recurrent connections h; + h441, and the input o(x) > hy. Then the hidden states save the last d inputs, and we can apply lemma A.16
24

---

Published in Transactions on Machine Learning Research (10/2025)
Remark A.19. Note that the construction simply proves the existence of such a network, not that it needs to be build in this manner. Other architectures like vanilla recurrent neural networks or long-short term memory networks could implement this construction, but it does not need to be the best solution.
Now we put together lemma A.15 lemma A.16 and lemma A.18
Proposition A.20. Assume the existence of smooth neural network that forecasts deterministic dynamical system with precision e from a partial observation of the state o(x) and hidden state or contert window h(z). For sufficiently small e, [o(x), is on embedding of the dynamicol system, for which define the functions 9, such that
e(x(m +r) .... (....))(((((())).m. 0) g" (e(x(m))), 0),
(52)
where e(x) is an embedding of
Proof. Lemma lemma A.15 makes the embedding necessary, and lemmas A.16 and A.18 prove that RNNs
and transformers can have it (respectively). g is then simply a function equivalent to in eq. but applied to the embedding e.
Remark A.21. Rewriting proofs with the extended state space would be cumbersome, because the loss would only refer to part of the state space. We would have to then create surogate loss covering the extended space, and bound it to the loss on the observed space. This would include extra definitions and clarifications for each theorem, and compromise readability.
25

---

Published in Transactions on Machine Learning Research (10/2025)
B Necessary assumptions
Some of the assumptions in our theory are necessary to have any form of statistical learning. Here we explain why.
Stationarity: In dynamical systems, stationarity refers to the property where the statistical characteristics of system's 's behavior remain constant over time. This mcans that the system's 'bchavior' or patterns don't t change, regardless of when you observe them. It is impossible to make forecasts based on statistical learning if the statistics of the system are not known (or change). Thus, this assumption is unavoidable.
Ergodicity: In dynamical systems, ergodicity describes system's 's behavior where the time average of a property along a trajectory is equal to the space average of that property. Essentially, a system is ergodic if a single trajectory visits all parts of the system's phase space (or a significant portion of it) a statistically representative way over a long period. Since we need to be able to apply statistical learning, some form of ergodicity is necessary to guaranlee that the data seen during training is also representative of the data during testing.
Uniform state coverage: The last critical assumption is that the training data should capture all the relevant state space seen during lesting. It is in principle impossible to prove forecasting capacity in states that have never been observed before without prior knowledge on the dynamics of the system, because one could always imaginc some extremely convoluted dynamics happoning only on that unobserved region of the stale space. we have some prior information though, we might be able to leverage it. In the extreme, this is corresponds to mechanistic models where the physics of the system are known, but the specific parameters are not. Note that we cover this setting in section 5.3
Finally, note that it is possible to partially relax those assumptions, for example by having qtationtay systems or mesastability. However, this makes proofs significantly more cumbersome, and introduces subtle mathematical notions that are not easy to asses.
26

---

Published in Transactions on Machine Learning Research (10/2025)
C
Dynamical systems equations
Here we describe the differential equations governing the four dynamical systems used in the main text.
C.1 Lorenz attractor
The Lorenz attractor three dimensional dynamical system described by the following equations
aly z), dy x(p 2) dx dt
dz
-y. y,- ry Bz
(53)
where we take o 10.0, p 28.0 and 3= 3. The system was originally described by Edward Lorenz and is a simplification of hydrodynamic flows. We note that its trajectories are bounded and non-periodic (Lorenz 1963
C.2 Double pendulum
The double pendulum is simple mechanical system that exhibits chaotic behavior Gupta al. (2014). It consists of two masses rigidly linked to each other, both can rotate freely around their anchor point. The first mass is connected to center point and the second mass is linked to the first mass.
The dynamics are determined by applying the standard equations of motion from Newtonian mechanics, resulting in eq. ff.
T1 sin 01
(54)
2
= cos 01
(55)
2
+ 12 sin 0g
(56)
1/2 = y1 122 cos 02
(57)
C.3
Food web model
We consider gencralizod 7-specics food-web model inspired by Hastings (1911] 0: McCann & Yodzis
(1944);
Klebanoff & Hastings (1994); Post al. (2000): Akesson al. 2021 where species interact through trophic and competitive relationships.
Denoting the abundance of species i by N₁, and defining N = (N1,... N7). the generalized community model can be expressed as
1 d N; = rx(())
Σ N₁ Ois + -F(N))
(58)
N₂ di
eq. 558 the first two terms capture intrinsic growth rate and intra- and interspecific competition. The last two terms capture growth and loss due trophic interactions.
The per capita growth rate r; may depend timo-dependent environmental forcing u(t). The competition coelficient between species i and is denoted by oig. The feeding rate [F(N) of species i on j follows a functional response of type II
W.W 1 qz 2i, His El Wo
[(N),,
(59)
27

---

Published in Transactions on Machine Learning Research (10/2025)
where 4u and H. represent the attack rate and handling time of species i when feeding species j, respectively. We used the biologically realistic parameter values with r(u(()) being constant. For = 0.2, our system is defined by
1.0
0
0 0
0
0
0
0.15
0
0
0
0
0.08
w
1
0
1.0
1.0
W
0
0
-0.15
w
-0.11
0
0.005
0
0
1
0
0
2.89855
0
0
7.35294
0
7.35294
H
0
0
0
0
0
2.89855
0
0
0
0
8.0
0
0
0
0
0
0
12.0
0
0
0
1.38
0
0
0
0
0.272
0
0
0.272
0
0
0
0
0
0
0
0
0
0
1.38
0
0
0
0
0
0.1
0
0
0.05
The dynamics the system are chaotic for this set parameter values, but in short timescales they resemble limit cycle. All non-zero coefficients were set as free parameters, excluding the coefficients of the adjacency matrix W, where only w was considered a free parameter.
C.4 Limit cycle
The basic limit cycle system is defined below,
dæ dt
1 dy
pr y x(x² +2))
= = my =2(22 =2(2+
(60)
a dt
with parameters 1.0, μ 0.4.
C.5 Implementation Details
The exact way in which the dynamical systems solved depended the figure. For fig. 6 they were solved in Julia using blaoticInfeeencdlll 2024 whereas for the other plots they solved in Python using diffrax (Kidger 2021
C.5.1 Fitting dynamical systems
The four systems, Lorenz, double pendulum, ecological model and the limit cycle are implemented dynamical systems, where an ODE gives rise to trajectory.
As the dynamical systems are solved by numerical integration over some time and not autoregressive, an adjusted loss function (Equation 61 is used.
T
(x,,T)
Σ
|a(m +T) F (x(mT),0))
(61)
m=
r=1
28

---

Published in Transactions on Machine Learning Research (10/2025)
Where FT (....... denotes the numerical solution for z(7)-
Heavy lifting is done by the ChaoticInf erence library (Marti 2024), namely data generation, loss evaluation and plotting. Data generation was done with the fully implicit, fifth order RadauIIA5 RackkaBakaaskass & For more details regarding initial conditions, timespan, etc. see the respective implementations.
erennceee
Nie 2017
C.5.2 Fitting neural networks
The four systems were solved using the diffrax package developed by Kidger 2021 with 10 initial conditions sampled from a Gaussian distribution. For the dataset associated with the food web, we sample the system every 2 seconds, for those associated with the double pendulum they were sampled every 0.005 seconds, for the Lorenz they were either sampled every 0.005 or 0.04 depending on the experiment and for the limit cycle they were sampled every 0.5.
C.6 Timestep choice
We selected the sampling timestep in such a way that we could observe the growths the gradients within a reasonable range of T. Having a higher sampling rate would require computing a lot of values of and overburden our plots, while low T would prevent the model from learning.
29

---

Published in Transactions on Machine Learning Research (10/2025)
D Geo-spatial datasets
In the following section, we expand on the geo-spatial datasets used in the main text and how we processed them for training with our limited computational budget.
D.1 NOAA SST dataset
2021 provide daily sea surface lemperature data from Seplember 1981 to the present day a 1/4 global grid. We sub-sample this grid, converting it to a 4 resolution and also sub-sample the temporal horizon to get states every 10 days. We split the training and validation datascts using given years. 2000 to 2009 was used for training and 2011 to 2017 was used for validation. Notably, we deal with missing data over land by setting values to The data was also normalized for training and validation.
Huang et al.
D.2 ClimSim dataset
The ClimSim dataset was originally developed by Yu et al. (2023) to provide training data for models predicting complex climate variables from more simplistic variables. However, we noticed it could also serve as an excellent candidate for exploring the learnability trade-off as data is reported every 20 minutes of the simulation. As such, we converted the low-resolution version the dataset, labeled LEAP/CI LEAP/ClimSim _al-ree for use in training our autoregressive models. We took a subset of the variables, namely the lowest and v fields for prediction. The training and validation sets were taken from non-overlaping year-long periods. As with the SST data, we also normalised this dataset.
30

---

Published in Transactions on Machine Learning Research (10/2025)
E
Loss plots for Lorenz system
The Lorenz system is paramcterized by three parameters 01, 01,02,00 03 corresponding to p, in appendix In this section we expand the visualization from Fig, G to the other parameters for a given temporal horizon T. Note that systems with more parameters such as the food web model, the plots become too complex.
We evaluate this function for T [10, 500, 1000). See plots fig. and fig. In these figures we soc, that the prediction already verified in one dimensional loss in figure fig. holds also for higher dimensional losses in dynamical systems.
6,66g
5000
1000
2000
3300
5000
662
2006
1560
4000
202
2006
0,400
0,609
7066
1930
3500
1930
5000
1880
3230
4030
2065
1002
0,4:0,
0,5c0
1559
1000
4300
3523
1130
Figure 7: Surface plot of the piecewise loss function of the Lorenz system with = 10
31

---

Published in Transactions on Machine Learning Research (10/2025)
6,k6
1000
6300
2068
500
1002
4239
2006
1008
0,0
42
0,400
50CC
1300
1930
DUCC
2030
3300
AKC
200
3030
1002
0,4
0,809
3003
980
3509
3000
2003
1000
2000
Figure 8: Surface plot of the piecewise loss function of the Lorenz system with T 1000
32

---

Published in Transactions on Machine Learning Research (10/2025)
F Training neural networks
In the following section we provide additional details on the training sctup used for the four dynamical systems, as well as the networks used for the two climate-related datasets. We used both MLPs andresidual MLPs with the same architectures.
F.1 MLPs
We used two types of MLPss a plain MLP without residual connections and one with residual connections. Definition F.1 (MLP). We define multi-layer to consist of an embedding layer (Wembed), unembedding layer (W unembod) and series of blocks of the form:
block(x)) ReLU blockk((()) + botock)
(62)
where LN is layer normalization. The network makes a given prediction by first applying the embedding layer, series of blocks and then the unembed layer.
If let be the input size of the example, we generally take Wembed Raxx" Wolock E Raxxa and Wunembed E Rauxe with varying values of and the number of blocks depending the complexity of the dataset.
F.2 Computer resources
We use less than 34000 core hours, equivalent time of 1888 hours on single V100 GPU.
F.3
Hyperparameter choices
As general rule, we tried maintain some consistency between the hyperparameter choices for various datasets and architectures (for example we typically used batch size of 512). However, due the variability of both the data and the inductive biases, this was not always possible. For detailed set of these, one would need to refer to the source codc.
G Relationships between model complexity, learning rate, temporal horizon and noise on performance
the autoregressive setting studied this paper, there are many hyperparameters to set. One needs to decide, among other things, upon the complexity of the model, what learning rate use and which temporal horizon to chose. To provide guidance on this front, we explored how performance was related to temporal horizon, learning rate and model complexity under different levels of observational noise. Specifically, we repeated the setup of section 5 for the ecology system with increasing label noise for a small, medium and large MLP using a total wall-time cutoff. We believe the following conclusions are evident from fig. g
For (relatively) high forecasting horizons and learning rates the models fail to learn (red areas in all plots). This aligns with our theory showing that longer time horizons make the loss landscape harder to navigate, which can be alleviated with small learning rates.
For any training horizon, the learning rates have an optimal value which not at the extrema: high learning rates lead to a full faihure to learn (red), but decreasing the learning rate can alleviate this. However, having too low learning rate also decreases the performance of the model, likely because training has not been completed.
Those observations are both compatible with our theory and also fairly intuitive for machine learning practitioners. Taken together, we can conclude that with increasing time horizon and model complexity, the upper bound effective learning rates decreases.
33

---

Published in Transactions on Machine Learning Research (10/2025)
It is worth noticing that our theory was focused on deterministic systems with very small values, and both assumptions are not necessarily preserved in some subplots of Fig. g Thus, by comparing the different. subplots we can reach the following conclusions:
For small noise levels, the magnitude of the forecasting error for the optimal time horizon decreases with the model size (note that the error magnitudes decrease across rows). This can be understood as saying that the model can approximate the dynamics better (e reduces).
For high noise levels, all models achieve similar optimal performances, regardless of size. Our interpretation is that is bounded from below by the randomness of the dynamics, the model size does not contribute to make better predictions
Taken together, those observations suggest that our theory is valid for small noise lcvels, but is not informative for large noise levels. However, at such levels prediction is possible due to the inherently stochastic nature of the dynamics.
Noise Level 0,001
Noise Level 0.1
Noise Level 1.0
1.0e+00
Small Model
8.9c-02
7.8e-03
7.0e004
-2
0.2
1.0e044
1.0e+00
Medium Model
8.9e-02
0.6
7.8e-03
0.4
7.0e044
0.2
1.0e04
Large Model
8.9e-02
0.8
0.6
7.8e-03
-1
0.4
7.0e044
-2
-1
0.2
10e-04
10
10
10
Prediction Horizon
Prediction Horizon
Prediction Horizon
Figure 9: We trained our models at different levels of noise (columns) and with different parameter sizes (rows). For every combination, lested different forecasting horizons (x-axis) and learning rates (y-axis). Note that the loss is normalized per plot.
34

---

Published in Transactions on Machine Learning Research (10/2025)
H Increasing the temporal horizon increases performance of a sufficiently converged model
In this section, we provide empirical evidence for a down-stream effect that we should expect from theorem 4.6 For a sufficiently converged model, increasing the temporal horizon will increase performance, but a too large increase would make the gradients explode and disrupt the model. To do so, consider MLPs trained the ecology time series presented in the main text. We assume that we have a fixed training time B that we split equally among different choices of temporal horizon. Given this split, we train the model by starting it on prediction timestep k 1, then switching to k = and so on until we reach kmax- This process is visualized in fig. I0
88
10=2
1
k:
k=
==
k
Figure 10: Experimental setup demonstrating the utility of increasing the temporal horizon of converged model.
If our hypothesis truc, for sufficiently large values of up to a certain kmax we should get better performance from a greater splitting. fig. [ reports the results of the experiment. As can be seen there, do indeed get the expected trend, with greater total time favoring greater splitting with larger total time horizon.
35

---

Published in Transactions on Machine Learning Research (10/2025)
MSE for Different Max Horizons and Total Time
Max Horizon
0.0030
0.0025
0.0020
MSE
0.0015
0.0010
0.0005
0.0000
120.0
180.0
240.0
360.0
Total Time
Figure 11: MSE compared to total time for various maximum horizons where training is split equally among time horizons up to the maximum horizon. We used the ecology system and computed the MSF
Choosing time horizons and learning rates
Although we did not deem the results robust enough for inclusion in the main text, we did develop scheme consistent with our theory for automatically deciding upon the training temporal horizon and learning rate. We would tentatively recommend this scheme to practitioners first applying eq. (3) to a well-behaved time series problem as given an initial ro, it can sometimes out-performs the best but may also be able to uncover reasonable choice for T in cases where it does not perform as well.
1.1 Motivation and specification of the algorithm
The scheme, detailed in algorithm seeks to automatically choose T (or equivalently k, the number of forecasting steps) and y7 to overcome common difficulty in gradient descent: the presence of plateaus. As we saw in theorem 4.7 the loss landscape becomes less flat with increasing T, suggosting that should be possible increase T' order deal with the plateaus. There is, however, some subtlety, because plateaus come in two forms: minimum plateau and saddle-platean. A minimum plateau is region of low loss surrounded by higher losses, and saddle-plateau is connected to some region of lower loss. In the later case, we'd like to speed up optimization at the expense of skipping small improvements that could be made in the region. In contrast, for minimum plateau we would like to identify the best points within the plateau, and thus we would like to keep the size of the parameter update equal or smaller to explore a more informative minima. If we want increase the step size traverse a saddle-platean, suffices to increase T, and the gradient will increase either linearly or exponentially, depending on the underlying dynamics of the system per proposition 4.3 Translating this logic into practice lead to development of the algorithm presented
36

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm Iterative scheduling of k and 1
1: Initialize: Io +- 1, 1) t TJ, Prer <- 0b, y t- 1.5 10-1, look t- Truc, $ kmax/ maz +- Falsc
2: Set: lstart f- lime(), lours t- lime(),
while teurr Estart T maz do
if look then
E < 20
6:
<- Prev
7:
else
8:
E + None
9: time limit f- tmax (teurr tstart)
10: end if
11: Train MLP with T' for E' epochs while recording L, ||
12: if validation loss improves then
13: succeeded +- True
14:
look t False
15: else
16:
Adjust 7 using exponential fit of trend in past |VoCll
17:
< Prrev
18:
end if
19: if early stop due to gradient using || VoCl| < then
20: k +- k+1
21:
look + False
22:
succeeded <- False
23:
end
24: leurr f time()
25: end while
in algorithm 1 As can be read there, the algorithm works by waiting until we reach a flat region, looking ahead see if we can obtain lower loss by increasing T without decreasing 7 and otherwise decreasing v) by fitting exponential to the existing data (mimicking the scaling implied by proposition 4.3 Note that the algorithm is modified for the limit cycle system by replacing the exponential fit by linear one.
1.2 Results in comparison to normal training
In fig. 12 we compare algorithm | with networks trained using the same wall time with varying training temporal horizons and constant learning rate our four dynamical systems, changing the algorithm to fit a linear trend on the limit cycle. Note that some tuning to 7, foo and the wall time was needed to find examples where the iterative scheme showed promise. Nonetheless, as presented fig. I2 there are certain wall times, dynamical systems, learning rates and values of y one can choose which show that the iterative scheme can, in some cases, outperform or match the best choice of training temporal horizon without knowledge of that horizon beforchand. Some general observations from initial testing was that the scheme appeared perform better cases where 710 was relatively low and 7 also reasonably low. However, there non-robust sensitivity to especially in the case of the limit cycle.
37

---

Published in Transactions on Machine Learning Research (10/2025)
Food web
Limit cycle
Lorenz
Double Pendulum
x 104
Min. vol, Loss (MSE)
10-4
10-1
6
l01
xx9
10
36
109
10-2
10
103
Iter. 2
4
Iter. 1 2
ter.
2 3
Iter.
2 3
5
Training strategy (Tor iter.)
Training strategy or iter.)
Training strategy (f iter.)
Training Strategy (Tor iter.)
Figure 12: Comparison of iterative scheme to regular training on the four dynamical systems from the main text, as box plot of the performances taken to be L(A, T) where k is the maximum number of steps and the data is from the validation set. For each system and each time limit, we plot the temporal horizon for training (integers) or the iterative increase. The red line is the median of these scores for the iterative scheme. Note that no, wall time and 7 have been tuned in favor of the iterative scheme (although a full search was not applied).
38


Reviews:
[{"review_id": "9wbzC7JDxg", "reviewer": "Reviewer_4w7z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n(1) The paper is very well written and easy to understand\n\n(2) The theory, while simple, is very insightful, and directly answers the question of why it is often difficult to train a model to predict a far away horizon\n\n(3) This perspective is also novel, and I believe it could be extended to analyze other problems (such as LLM training)\n\n(4) The theory is quite comprehensive, covering the growth of gradient, Hessian and number of local minima, basically everything one should care about in learning these dynamical systems\n\n(5) A key message of the paper is that the longer the prediction horizon is, the more difficult it is to train the model -- a well observed (and almost universal phenomenon). Therefore, I believe that this is simple theory that partially tackles one of the most important problems in this field\n\nWeakness:\nNothing major. The figures are well plotted, the paper is well explain. The theory is clean and easy to understand\n\nInteresting and great work!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Sorry for writing my review late. \n\nThis work theoretically studies the implication for training when we train a model to predict a very long horizon.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Nothing really", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l8r7IEWInR", "reviewer": "Reviewer_GSFB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis is an interesting topic in general, and quite timely with the recent successes of state-space models like Mamba. \n\n## Weaknesses\n\nThe writing has several issues with unsubstantiated or outright incorrect claims (see below). The experiments involve only a few data points, and there is no consideration of the noise which may wash out the main results reported in this paper. For instance, noisy limit cycles or chaotic networks will likely have the same scaling of error in time, regardless of how far ahead they are trained to predict.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the trade-off between forecast horizon length and learnability in recurrent neural networks (RNNs, though the paper seems to make a new definition and call them auto-regressive neural networks...) trained on dynamical systems. The authors argue that longer training horizons provide better generalization, especially in chaotic systems, but make optimization harder due to a rougher loss landscape. They offer theoretical analyses of gradient and Hessian scaling with respect to the forecast horizon, and validate their results empirically on synthetic and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not believe the current version is publication level and requires substantial revisions to meet either of the two criteria. Here are my suggestions for a revised submission:\n\n1) Correct the definition of a dynamical system. The current formulation inverts time and misrepresents standard terminology. Clearly distinguish between continuous and discrete-time systems and define them properly.\n\n2) Revise the taxonomy of dynamical systems. The four-category classification (“stable, unstable, limit cycles, chaotic”) is incorrect and conflates local stability with global system types. I would avoid simplification of this form, which was quite unpleasant for someone with a physics background to read.\n\n3) Clarify model architecture terminology. The term “ARNN” is used in a non-standard way. Clearly differentiate between feedforward autoregressive models, RNNs, and Transformers, and avoid suggesting that only the former are autoregressive. The definition in Eq. (2) IS an RNN.\n\n\n4) Acknowledge that exponential forecast divergence occurs even with perfect models and address the role of noise more directly. The exponential gradient and loss growth described in the main theorems are not specific to learned models, they arise directly from chaotic dynamics and would be observed even with true parameters. A similar argument can be made for the limit cycle. This weakens the claim that long-horizon training yields fundamentally better generalization. Given that exponential divergence arises even in perfect models, the paper’s main results appear to restate known properties of chaotic dynamics rather than introduce new insights into training RNNs. Many real-world systems are stochastic. The theory should either explicitly incorporate noise into the core results (not just appendices) or clearly state that the current framework applies only to noise-free systems, in which case I am not sure if it satisfies the interest to the general TMLR audience criteria. \n\n5) Rewrite the manuscript for clarity. The core results should be presented in the main text, not appendices. The main text is very hard to read and digest in its current form without looking at the appendices regularly. The work has more space to work with, and potentially can be longer than 12 pages as TMLR allows it.\n\n6) Few data points is not enough to claim exponential/linear divergence: Plots include only a few data points and no rigorous evidence that the stated scaling takes place. The scaling claims (exponential vs. linear) cannot be substantiated with only 5–6 data points and should be supported by formal curve fitting or statistical analysis.\n\n7) Ignoring advantages of single-step paradigms: Several works have surfaced recently that use single-step paradigms to train RNNs with convex optimization within seconds, what would have taken days with BPTT. What the theoretical results are saying is that accounting for longer time windows does not prevent any further correctness than just training with a single-step prediction errors if we have a noisy model (which is the case in practice). So, the argument made by the paper for me is actually the opposite of what is claimed.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pf2fqUTkZH", "reviewer": "Reviewer_EERN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper addresses an important problem with clear motivation.\n* The paper combines theoretical analysis with empirical validation in a well-balanced manner.\n* The paper is well-written with a clear and logical structure across sections.\n\nWeaknesses\n* The theoretical analysis relies on strong assumptions such as ergodicity, stationarity, full observability, and complete state space coverage. These assumptions are rarely satisfied in real-world forecasting tasks, limiting the practical applicability of the paper contributions.\n* Experiments are primarily conducted on synthetic dynamical systems. The absence of evaluations on widely-used benchmarks (e.g., Electricity, Traffic, Weather, and ETT time series [1]) reduces the generalizability of the conclusions and connectivity with existing works.\n* While the authors present interesting findings based on ARNNs, it is important to note that ARNNs are not commonly regarded as a mainstream architecture in time-series forecasting, especially compared to RNNs or Transformers. This raises concerns about the general impact of the contribution.\nTo strengthen the significance of the proposed findings, it would be valuable to either (1) demonstrate that similar trends hold across other widely used architectures, or (2) provide more concrete evidence that ARNNs offer clear advantages over existing models in the context studied. This would clarify whether the observed effects are intrinsic to ARNNs or part of a broader phenomenon, thereby enhancing the relevance and impact of the paper.\n* Horizon selection is still heuristic. Despite theoretical insights into the training horizon $T$, the practical selection still relies on grid search. The adaptive algorithm in Appendix H shows limited success, especially in chaotic systems, and does not fully address this limitation.\n* It is difficult to find detailed descriptions of the values used in the experiments (e.g., the exact values of Tl and Th are unclear).\n* In Figure 2, the y-axis label $r(T_h - T_l)$ is ambiguous; it is unclear whether this denotes a specific metric or is simply a typo.\n* Figure 3 does not provide a clear explanation of z(T) in the caption. This forces readers to search the main text repeatedly to understand the notation, which is inconvenient.\n* In Figure 4, the y-axis label uses undefined abbreviations in the main text, potentially causing confusion.\n* Overall, the figures lack visual consistency (e.g., the placement of plot titles is not uniform across the figures). \n\n[1] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting, NeurIPS'21", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how the training horizon affects forecasting performance in autoregressive neural networks (ARNNs). The paper shows that longer horizons improve generalization but make optimization harder, due to increased loss landscape roughness. The authors support this with theorical evidence and empirical experiments on synthetic dynamical systems. They conclude that the optimal training horizon depends on system dynamics and should be selected accordingly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses mentioned above.", "broader_impact_concerns": "No significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)